Skip to contentSkip to conversation

infiniband-diags/libibnetdisc: Introduce a context object.

PATCH In review Sasha Khapyorsky 3 participants 14 messages Oct 23, 2009 – Jan 25, 2010 14 messages not in the archive InfiniBand

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 20090816110200.GS25501@me

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 20090823120609.GG9547@me

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 4 lines · Ira Weiny

The discussion on the list has digressed from this patch. I still think this patch is valid and adds a level of flexibility which is needed regardless of what is decided about libibmad. Do you agree?

Not really. I still think that needed level of flexibility is available for us without adding new mandatory APIs.

Quoted 4 lines

Also, the last patch in the series ([PATCH 5/5] infiniband-diags/libibnetdisc:
remove members of the fabric struct which are used in the scan only) cleans up some stuff from the external interface. If you really don't want to introduce a context object, then I can regenerate that final patch without the context.

Yes, since this patch is unrelated logically to a context stuff. It would be good to rebase it back and repost as independent change.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

Re: Multi-threaded diags (Was: Re: [PATCH 4/5] infiniband-diags/libibnetdisc: Introduce a context object.)

Quoted 31 lines · Ira Weiny

Of course! :-) But first I would like to mention some numbers from the prototype code I have. When running on a small fabric the additional overhead of thread creation actually slows down the scan. :-(

Current master: Threaded version:

real    0m0.101s         0m0.266s
user    0m0.000s         0m0.000s
sys     0m0.011s         0m0.014s

But, as expected, on a large system (1152 nodes) there is a decent speed up.

Current Master: Threaded version:

real    0m3.046s         0m1.748s
user    0m0.073s         0m0.331s
sys     0m0.158s         0m0.822s

However, the biggest speed up comes when there are errors on the fabric. This is the same 1152 node cluster with just 14 "bad" ports on the fabric. This is of course because the scan continues "around" the bad ports.

Current Master: Threaded version:

real    0m33.051s        0m5.609s
user    0m0.071s         0m0.353s
sys     0m0.156s         0m1.113s

Since you are usually running these tools when things are bad I think there is a big gain here. Even running with a faster timeout of 200ms results in a big difference.

Quoted 6 lines

Current Master: Threaded version:

real    0m9.149s        0m2.223s
user    0m0.016s        0m0.374s
sys     0m0.372s        0m1.056s

With that in mind...

Good. So what do you think due to which factor most of this performance gain was achieved? Due to using multiple threads or due to SMP queries parallelism? I would suspect that it is a parallelism.

Quoted 5 lines
Quoted 3 lines

implemented with libibnetdisc: goals (in particular is it support for multithreaded apps or just multithreaded discovery function), interaction with caller application, etc.?

My initial goal was to make the libibnetdisc safe for multithreaded apps

Ok.

Quoted 2 lines

and
make a multithreaded discovery function.

And this is not the same as above. I would really suggest before doing multithreaded implementation to check just single threaded but parallel querying first. Somehow I believe that an impressive numbers could be shown there and also without multithreaded overhead and complexity.

Quoted 11 lines

However, since libibmad itself is
not thread safe, and you expressed a desire to keep it that way[*], I reduced that goal to just making the discovery function multithreaded (using mad_[send|receive]_via).

Although I don't like this restriction I can see it as a valid design decision as long as it is documented that the discover function is not thread safe in regards to the ibmad_port object. This is because the ibnd_discover_fabric uses libibmad calls and would require a complicated API to allow the user app to synchronize with those calls.

Quoted 8 lines

In order to make things thread safe for the user apps as well as the library I can see 3 options.

1) make libibmad thread safe (which you were hesitant to do)

2) add a thread safe interface to libibmad. User apps will need to know to use this interface while using libibnetdisc and libibnetdisc will use this interface.

Why should it be related to libibmad? Make libibnetdisc by itself thread safe and that is all (see below).

Quoted 3 lines

3) Create a wrapper lib which is thread safe. In this case the apps and libibnetdisc would call into this wrapper lib and we would have to change the API to libibnetdisc.

4) Instead of bothering with slow (blocked) libibmad rpc use umad_send() and umad_recv() directly in parallel hop by hop fabric discovery.

(And following my personal experience it is ended in smaller and simpler code using umad directly instead of rpc for applications which slightly more complex than 'smpquery' style tools).

Quoted 5 lines

[*] http://lists.openfabrics.org/pipermail/general/2009-July/060677.html

"madrpc() is too primitive interface for such applications. There would be better to use umad_send/recv() directly or may be mad_send_via(). Example is mcast_storm.c distributed with ibsim."

Another option is adding a new thread safe API. But I don't really see how it is related to libibnetdisc discussion.

Quoted 8 lines

[$] It is my opinion that mad_rpc is _not_ primitive. In my mind it _is_
the wrapper around the primitive umad_send/recv calls. If you are interested perhaps I can try to explain what I wanted to do in the library to make it thread safe more clearly. The point I might not have made clear was that I don't think the library will have to do any threading on it's own, just some locks and storing of responses. Of course the down side to this is the libibmad code would be slightly slower. But I don't think by very much.

I don't like a lock introduction there - right, it is unneeded overhead (and dependencies) for single threaded tools. And again - I have nothing against thread safe rpc++ API, but would prefer instead of locking use reenrant states.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#10 lore raw

PATCH 1/1 tests/subnet_discover: discover test utility

'subnet_discover' is simple test utility which implements "non-blocking" discovery method where mads are sending "in parallel" (unlike the current implementation of 'ibnetdiscover' and similar to how OpenSM does). For this a recently discovered node id value is encoded as lower 16 bits of mad transaction id.

---

Hi Ira,

Quoted 11 lines · Sasha Khapyorsky
Quoted 7 lines

Current Master: Threaded version:

real    0m9.149s        0m2.223s
user    0m0.016s        0m0.374s
sys     0m0.372s        0m1.056s

With that in mind...

Good. So what do you think due to which factor most of this performance gain was achieved? Due to using multiple threads or due to SMP queries parallelism? I would suspect that it is a parallelism.

For some purposes in ibsim/tests I wrote a simple utility 'subnet_discover', this works as single thread and utilizes a "parallel" mad sending method and also uses libibumad for all mad sending/receiving stuff.

I think that similar implementation in libibnetdisc (I can do it if we are in agreement :)) would improve its performance.

Would you like to look at this?

Sasha

 tests/Makefile          |    2 +-
 tests/subnet_discover.c |  495 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 496 insertions(+), 1 deletions(-)
 create mode 100644 tests/subnet_discover.c
tests/Makefile +1 -1
index dd4cd55..bd415d8 100644
@@ -1,4 +1,4 @@
-progs:= mcast_storm
+progs:= subnet_discover mcast_storm
 
 -include ../defs.mk
 
tests/subnet_discover.c new file +495 -0

495 lines not on this page · raw message

Signature · 6 lines
1.6.6.rc3

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Hi Ira,

Quoted 2 lines · Ira Weiny

Yes, a similar mechanism would work in libibnetdisc. However, it looks like you are doing a depth first search

I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.

Quoted 1 line

which I fear might exceed the path count limit in a DR path?

Right, hops count should be limited. I will add this.

Quoted 1 line

Is this how OpenSM works?

I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and subnet_discover doesn't (and there could be a lot of MADs).

Quoted 7 lines

I was trying to do a breath first search like ibnetdiscover does.

Quoted 2 lines

Would you like to look at this?

No problem, I have enclosed the output from a run on Hyperion.
There appears to be a lot of errors. I am not sure what the issue is right off. I have included an ibnetdiscover output for comparision.

Thanks.

An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Quoted 9 lines · Sasha Khapyorsky

Hi Ira,

Quoted 2 lines · Ira Weiny

Yes, a similar mechanism would work in libibnetdisc.  However, it looks like you are doing a depth first search

I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.

Does anything limit the amount of parallelism ?

Quoted 10 lines
Quoted 2 lines

which I fear might exceed the path count limit in a DR path?

Right, hops count should be limited. I will add this.

Quoted 1 line

Is this how OpenSM works?

I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and subnet_discover doesn't (and there could be a lot of MADs).

Isn't that dangerous in a real subnet ?

-- Hal

Quoted 15 lines
Quoted 8 lines

I was trying to do a breath first search like ibnetdiscover does.

Quoted 2 lines

Would you like to look at this?

No problem, I have enclosed the output from a run on Hyperion.
There appears to be a lot of errors.  I am not sure what the issue is right off.  I have included an ibnetdiscover output for comparision.

Thanks.

An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.

Quoted 6 lines

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f230912210602i5e3f528h2b0630420346db82-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#16 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Quoted 5 lines · Hal Rosenstock
Quoted 3 lines

I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.

Does anything limit the amount of parallelism ?

Nothing in this version.

Quoted 5 lines
Quoted 3 lines

I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and subnet_discover doesn't (and there could be a lot of MADs).

Isn't that dangerous in a real subnet ?

This is test tool, so I don't think that it is "dangerous". And of course this can lead to an issues with test results.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#17 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

In reply to Sasha Khapyorsky

Hi Ira,

Quoted 4 lines · Sasha Khapyorsky

An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.

I did a couple of modifications in the code (exact log is listed below). In particular there are default limitation for number of outstanding MADs on the wire and proper tracking for failed (timedout) MADs. I tested this where possible. Could you re-run this? Thanks.

Sasha

commit 9e24853c30351f6ea65ffcccf184bdf7586dfe8e

Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Fri Dec 25 13:01:35 2009 +0200

tests/subnet_discover: limit possible number of hops

As pointed out by Ira there was no hops limitation. Adding this.

tests/subnet_discover.c +5 -2
index a577cc7..9857913 100644
@@ -17,6 +17,8 @@
 #include <infiniband/umad.h>
 #include <infiniband/mad.h>
 
+#define MAX_HOPS 63
+
 struct port {
 	struct node *node;
 	uint64_t guid;
@@ -217,8 +219,9 @@ static int process_port_info(void *umad, unsigned node_id, int fd, int agent,
 	if (port_num &&
 	    mad_get_field(port_info, 0, IB_PORT_PHYS_STATE_F) == 5 &&
 	    ((node->is_switch && port_num != local_port) ||
-	     (node_id == 0 && port_num == local_port))) {
-		path[++path_cnt] = port_num;
+	     (node_id == 0 && port_num == local_port)) &&
+	    path_cnt++ < MAX_HOPS) {
+		path[path_cnt] = port_num;
 		return query_node_info(fd, agent, umad, node_id, path,
 				       path_cnt);
 	}

commit 6e7817433c17bf2b8861639852dc0e70e8d0ec5f

Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Fri Dec 25 16:11:53 2009 +0200

tests/subnet_discover: add --help option

Add --help command line option. Also cosmetic improvements.

tests/subnet_discover.c +10 -7
index 9857913..7f8a85c 100644
@@ -70,7 +70,7 @@ DBG_DUMP_FUNC(nodedesc);
 DBG_DUMP_FUNC(portinfo);
 DBG_DUMP_FUNC(switchinfo);
 
-static void build_umad_req(void *umad, uint8_t * path, unsigned path_cnt,
+static void build_umad_req(void *umad, uint8_t path[], unsigned path_cnt,
 			   uint64_t trid, uint8_t method,
 			   uint16_t attr_id, uint32_t attr_mod, uint64_t mkey)
 {
@@ -94,7 +94,7 @@ static void build_umad_req(void *umad, uint8_t * path, unsigned path_cnt,
 }
 
 static int send_query(int fd, int agent, void *umad, unsigned node_id,
-		      uint8_t * path, size_t path_cnt, uint16_t attr_id,
+		      uint8_t path[], size_t path_cnt, uint16_t attr_id,
 		      uint32_t attr_mod)
 {
 	uint64_t trid;
@@ -138,28 +138,28 @@ static int recv_response(int fd, int agent, uint8_t * umad, size_t length)
 }
 
 static int query_node_info(int fd, int agent, void *umad, unsigned node_id,
-			   uint8_t * path, size_t path_cnt)
+			   uint8_t path[], size_t path_cnt)
 {
 	return send_query(fd, agent, umad, node_id, path, path_cnt,
 			  IB_ATTR_NODE_INFO, 0);
 }
 
 static int query_node_desc(int fd, int agent, void *umad, unsigned node_id,
-			   uint8_t * path, size_t path_cnt)
+			   uint8_t path[], size_t path_cnt)
 {
 	return send_query(fd, agent, umad, node_id, path, path_cnt,
 			  IB_ATTR_NODE_DESC, 0);
 }
 
 static int query_switch_info(int fd, int agent, void *umad, unsigned node_id,
-			     uint8_t * path, size_t path_cnt)
+			     uint8_t path[], size_t path_cnt)
 {
 	return send_query(fd, agent, umad, node_id, path, path_cnt,
 			  IB_ATTR_SWITCH_INFO, 0);
 }
 
 static int query_port_info(int fd, int agent, void *umad, unsigned node_id,
-			   uint8_t * path, size_t path_cnt, unsigned port_num)
+			   uint8_t path[], size_t path_cnt, unsigned port_num)
 {
 	return send_query(fd, agent, umad, node_id, path, path_cnt,
 			  IB_ATTR_PORT_INFO, port_num);
@@ -456,6 +456,8 @@ int main(int argc, char **argv)
 		{"Port", 1, 0, 'P'},
 		{"timeout", 1, 0, 't'},
 		{"retries", 1, 0, 'r'},
+		{"verbose", 0, 0, 'v'},
+		{"help", 0, 0, 'h'},
 		{}
 	};
 	char *card_name = NULL;
@@ -463,7 +465,7 @@ int main(int argc, char **argv)
 	int ch, ret;
 
 	while (1) {
-		ch = getopt_long(argc, argv, "C:P:t:r:v", long_opts, NULL);
+		ch = getopt_long(argc, argv, "C:P:t:r:vh", long_opts, NULL);
 		if (ch == -1)
 			break;
 		switch (ch) {
@@ -482,6 +484,7 @@ int main(int argc, char **argv)
 		case 'v':
 			verbose++;
 			break;
+		case 'h':
 		default:
 			printf("usage: %s [-C card_name] [-P port_num]"
 			       " [-t timeout] [-r retries] [-v[v]]\n", argv[0]);

commit da6aa19840cb2d37e8cd3daa3874b87657a76ddc

Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Fri Dec 25 16:24:13 2009 +0200

tests/subnet_discover: --maxsmps (-n) option

This implements the limitation of outstanding SMPs on a wire at any one time. --maxsmps=0 means - no limit.

tests/subnet_discover.c +105 -34

265 lines not on this page · raw message

commit a422ea90334441144f2a1212de40085bbe36cf7e

Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Sun Dec 27 18:55:35 2009 +0200

tests/subnet_discover.c: print useful information

Print additional useful information about a subnet and discovery
process: such as number of MADs used, number of hops reached, direct paths for nodes as it was discovered. Better error messages (in particular - don't print MAD content in error message when returned valid data from umad_recv() is only umad header).

tests/subnet_discover.c +23 -8
index 42e7aee..b6aada9 100644
@@ -30,6 +30,8 @@ struct node {
 	uint64_t guid;
 	unsigned num_ports;
 	unsigned is_switch;
+	size_t path_size;
+	uint8_t path[64];
 	uint8_t node_info[IB_SMP_DATA_SIZE];
 	uint8_t node_desc[IB_SMP_DATA_SIZE];
 	uint8_t switch_info[IB_SMP_DATA_SIZE];
@@ -45,6 +47,9 @@ static unsigned timeout = 100;
 static unsigned retries = 3;
 static unsigned verbose = 0;
 
+static unsigned total_mads = 0;
+static unsigned max_hops = 0;
+
 #define ERROR(fmt, ...) fprintf(stderr, "ERR: " fmt, ##__VA_ARGS__)
 #define VERBOSE(fmt, ...) if (verbose) fprintf(stderr, fmt, ##__VA_ARGS__)
 #define NOISE(fmt, ...) if (verbose > 1) fprintf(stderr, fmt, ##__VA_ARGS__)
@@ -142,6 +147,7 @@ static void run_request_queue(int fd, int agent)
 		q = q->next;
 		free(prev);
 		outstanding++;
+		total_mads++;
 	}
 	request_queue.next = q;
 	if (!q)
@@ -201,10 +207,10 @@ static int recv_response(int fd, int agent, uint8_t * umad, size_t length)
 	if (ret < 0 || umad_status(umad)) {
 		ERROR("umad_recv failed: umad status %x: %s\n",
 		      umad_status(umad), strerror(errno));
-		return -1;
+		return len > umad_size() ? 1 : -1;
 	}
 
-	return ret;
+	return 0;
 }
 
 static int query_node_info(int fd, int agent, unsigned node_id,
@@ -235,7 +241,7 @@ static int query_port_info(int fd, int agent, unsigned node_id,
 			  IB_ATTR_PORT_INFO, port_num);
 }
 
-static int add_node(uint8_t * node_info)
+static int add_node(uint8_t * node_info, uint8_t path[], size_t path_size)
 {
 	struct node *node;
 	unsigned i, num_ports = mad_get_field(node_info, 0, IB_NODE_NPORTS_F);
@@ -250,6 +256,8 @@ static int add_node(uint8_t * node_info)
 	node->guid = mad_get_field64(node_info, 0, IB_NODE_GUID_F);
 	node->is_switch = ((mad_get_field(node_info, 0, IB_NODE_TYPE_F)) ==
 			   IB_NODE_SWITCH);
+	memcpy(node->path, path, path_size + 1);
+	node->path_size = path_size;
 	memcpy(node->node_info, node_info, sizeof(node->node_info));
 	for (i = 0; i <= num_ports; i++)
 		node->ports[i].node = node;
@@ -291,6 +299,8 @@ static int process_port_info(void *umad, unsigned node_id, int fd, int agent,
 	    ((node->is_switch && port_num != local_port) ||
 	     (node_id == 0 && port_num == local_port)) &&
 	    path_cnt++ < MAX_HOPS) {
+		if (path_cnt > max_hops)
+			max_hops = path_cnt;
 		path[path_cnt] = port_num;
 		return query_node_info(fd, agent, node_id, path, path_cnt);
 	}
@@ -341,7 +351,7 @@ static int process_node(void *umad, unsigned remote_id, int fd, int agent,
 	dbg_dump_nodeinfo(node_info);
 
 	if ((id = find_node(node_info)) < 0) {
-		id = add_node(node_info);
+		id = add_node(node_info, path, path_cnt);
 		if (id < 0)
 			return -1;
 		node_is_new = 1;
@@ -398,7 +408,9 @@ static int recv_smp_resp(int fd, int agent, uint8_t * umad, uint8_t path[])
 	outstanding--;
 	run_request_queue(fd, agent);
 
-	if (ret < 0 || status) {
+	if (ret < 0)
+		return ret;
+	else if (ret || status) {
 		ERROR("error response 0x%016" PRIx64 ": attr_id %x"
 		      ", attr_mod %x from %s with status %x\n", trid,
 		      attr_id, attr_mod, print_path(path, path_cnt), status);
@@ -477,7 +489,7 @@ static int umad_discover(char *card_name, unsigned int port_num)
 
 	ret = discover(fd, agent);
 	if (ret)
-		ERROR("Failed to discover.\n");
+		fprintf(stderr, "\nThere are problems during discovery.\n");
 
 	umad_unregister(fd, agent);
 	umad_close_port(fd);
@@ -493,12 +505,15 @@ static void print_subnet()
 	struct port *local, *remote;
 	unsigned i, j;
 
+	printf("\n# The subnet discovered using %u mads, reaching %d hops\n\n",
+	       total_mads, max_hops);
+
 	for (i = 0; i < node_count; i++) {
 		node = node_array[i];
-		printf("%s %u \"%s-%016" PRIx64 "\" \t# %s\n",
+		printf("%s %u \"%s-%016" PRIx64 "\" \t# %s %s\n",
 		       node->is_switch ? "Switch" : "Ca", node->num_ports,
 		       node->is_switch ? "S" : "H", node->guid,
-		       node->node_desc);
+		       print_path(node->path, node->path_size), node->node_desc);
 		for (j = 1; j <= node->num_ports; j++) {
 			local = &node->ports[j];
 			remote = local->remote;

commit 4a23f9e7f339e93f2a77f213d4ce80e4bc7d7b9f

Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Sun Dec 27 21:19:30 2009 +0200

tests/subnet_discover: report unresponded transactions

Report unresponded transactions (requests) in case of MAD failures.

tests/subnet_discover.c +58 -3

94 lines not on this page · raw message

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#18 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Hi Sasha,

Quoted 15 lines · Sasha Khapyorsky

Hi Ira,

Quoted 4 lines · Sasha Khapyorsky

An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.

I did a couple of modifications in the code (exact log is listed below). In particular there are default limitation for number of outstanding MADs on the wire and proper tracking for failed (timedout) MADs. I tested this where possible. Could you re-run this? Thanks.

Sasha

[snip...]

Quoted 16 lines

commit da6aa19840cb2d37e8cd3daa3874b87657a76ddc
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Fri Dec 25 16:24:13 2009 +0200

   tests/subnet_discover: --maxsmps (-n) option

   This implements the limitation of outstanding SMPs on a wire at any one time. --maxsmps=0 means - no limit.

tests/subnet_discover.c +0 -0
index 7f8a85c..42e7aee 100644

tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43

Quoted 4 lines
tests/subnet_discover.c +1 -0
@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];
  static unsigned node_count = 0;
  static unsigned trid_cnt = 0;
  static unsigned outstanding = 0;
+static unsigned max_outstanding = 8;

Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).

-- Hal

tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 46

Quoted 3 lines
tests/subnet_discover.c +0 -0
  static unsigned timeout = 100;
  static unsigned retries = 3;
  static unsigned verbose = 0;

[snip...]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f231001110556y7c47cc54oa3cfd5859f9a4e76-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#20 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Hi Hal,

Quoted 13 lines · Hal Rosenstock

tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43

Quoted 10 lines
tests/subnet_discover.c +1 -0
index 7f8a85c..42e7aee 100644
@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];
  static unsigned node_count = 0;
  static unsigned trid_cnt = 0;
  static unsigned outstanding = 0;
+static unsigned max_outstanding = 8;

Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).

In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.

Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Hi Sasha,

Quoted 17 lines · Sasha Khapyorsky

Hi Hal,

Quoted 13 lines · Hal Rosenstock

tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43

Quoted 10 lines
tests/subnet_discover.c +1 -0
index 7f8a85c..42e7aee 100644
@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];
 static unsigned node_count = 0;
 static unsigned trid_cnt = 0;
 static unsigned outstanding = 0;
+static unsigned max_outstanding = 8;

Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).

Quoted 5 lines

In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.

Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.

This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.

There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.

Also, the OpenSM default should be at least as large as the diags for this.

-- Hal

Quoted 2 lines

Sasha

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f231001131211y64489a51nd2621cefdb27ad25-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#23 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Quoted 10 lines · Hal Rosenstock
Quoted 6 lines

In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.

Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.

This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.

Was it investigated? What was a conclusions?

Quoted 5 lines

There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.

Also, the OpenSM default should be at least as large as the diags for this.

I don't know where concurrent MADs are used in the current "diags", do you?

This utility is not 'diags' at all. It is placed in ibsim/tests and I wrote it for test purpose.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#24 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Quoted 5 lines · Sasha Khapyorsky
Quoted 4 lines · Hal Rosenstock
Quoted 4 lines

In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.

This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.

Was it investigated? What was a conclusions?

I was pointing out testing that needs to be done rather than testing already done. The use cases I mentioned are based on current experience. I'm sure there are more.

Quoted 11 lines
Quoted 5 lines

There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.

Also, the OpenSM default should be at least as large as the diags for this.

I don't know where concurrent MADs are used in the current "diags", do you?

This utility is not 'diags' at all. It is placed in ibsim/tests and I wrote it for test purpose.

In your original email on this, you proposed that this algorithm be incorporated in the diags (you wrote "I think that similar implementation in libibnetdisc (I can do it if we
are in agreement :)) would improve its performance") so this comment thread appears relevant to me.

-- Hal

Quoted 2 lines

Sasha

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#25 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

In reply to a message not in the archive

Hey Sasha,

I am finally getting back to this... Sorry.

Quoted 25 lines · On Wed, 13 Jan 2010 15:11:44 -0500 Hal Rosenstock…

Hi Sasha,

Quoted 22 lines · Sasha Khapyorsky

Hi Hal,

Quoted 13 lines · Hal Rosenstock

tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43

Quoted 10 lines
tests/subnet_discover.c +1 -0
index 7f8a85c..42e7aee 100644
@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];
 static unsigned node_count = 0;
 static unsigned trid_cnt = 0;
 static unsigned outstanding = 0;
+static unsigned max_outstanding = 8;

Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).

In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.

Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.

Here is some test data on a real cluster.

09:49:10 > ibhosts | wc -l
1158

09:49:28 > ibswitches | wc -l
281

09:44:45 > time ./subnet_discover -n 1 > /dev/null

real    0m1.414s
user    0m0.309s
sys     0m0.244s

09:44:55 > time ./subnet_discover -n 2 > /dev/null

real    0m1.025s
user    0m0.284s
sys     0m0.201s

09:45:00 > time ./subnet_discover -n 4 > /dev/null

real    0m0.644s
user    0m0.268s
sys     0m0.228s

09:45:04 > time ./subnet_discover -n 8 > /dev/null

real    0m0.550s
user    0m0.253s
sys     0m0.184s

09:45:08 > time ./subnet_discover -n 12 > /dev/null

real    0m0.524s
user    0m0.207s
sys     0m0.201s

09:45:14 > time ./subnet_discover -n 16 > /dev/null

real    0m0.432s
user    0m0.248s
sys     0m0.144s

09:45:18 > time ./subnet_discover -n 32 > /dev/null

real    0m0.484s
user    0m0.260s
sys     0m0.150s

09:45:57 > time ibnetdiscover > /dev/null

real    0m3.180s
user    0m0.068s
sys     0m0.672s

What I find most interesting is that your test utility runs nearly 2x faster even when there is only 1 outstanding MAD. :-/ ibnetdiscover (libibnetdisc) does do a lot more with the data but I would not have expected such a difference.

As a comparison I ran iblinkinfo it would seem that there is something in the library which takes a lot more time.

09:51:59 > time iblinkinfo > /dev/null

real    0m3.159s
user    0m0.063s
sys     0m0.526s

For further comparison I rebuilt the parallel version of libibnetdisc.

12:39:02 > time ./ibnetdiscover > /dev/null

real    0m2.552s
user    0m0.295s
sys     0m0.863s

This is with 8 threads (ie 8 outstanding SMP's).

I would appear that your algorithm is superior. I will look at converting libibnetdisc, test, and submit a patch. I still don't know why there would be so much difference when only using 1 outstanding MAD though? :-/

Quoted 8 lines

This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.

There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.

This is a problem. I have seen this issue with large systems which are having trouble. OpenSM is trying to discover and route. We are running diags trying to figure out what is going on. There is hardware going up and down; bad switches or nodes which are booting/rebooting.

I plan to go forward with this but having an option for outstanding MAD's is a good idea. I don't have an opinion on where it should default.

Quoted 2 lines

Also, the OpenSM default should be at least as large as the diags for this.

I agree. OpenSM should have some priority in this matter.

Ira

Quoted 5 lines

-- Hal

Quoted 2 lines

Sasha

Signature · 9 lines
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#27 lore raw

Re: [PATCH] tests/subnet_discover: discover test utility

Hi Ira,

Quoted 39 lines · Ira Weiny

Here is some test data on a real cluster.

09:49:10 > ibhosts | wc -l
1158

09:49:28 > ibswitches | wc -l
281

09:44:45 > time ./subnet_discover -n 1 > /dev/null

real    0m1.414s
user    0m0.309s
sys     0m0.244s

09:44:55 > time ./subnet_discover -n 2 > /dev/null

real    0m1.025s
user    0m0.284s
sys     0m0.201s

09:45:00 > time ./subnet_discover -n 4 > /dev/null

real    0m0.644s
user    0m0.268s
sys     0m0.228s

09:45:04 > time ./subnet_discover -n 8 > /dev/null

real    0m0.550s
user    0m0.253s
sys     0m0.184s

09:45:08 > time ./subnet_discover -n 12 > /dev/null

real    0m0.524s
user    0m0.207s
sys     0m0.201s
Quoted 5 lines

09:45:14 > time ./subnet_discover -n 16 > /dev/null

real    0m0.432s
user    0m0.248s
sys     0m0.144s

Looks like a very nice results. Thanks for probing this.

Quoted 5 lines

09:45:18 > time ./subnet_discover -n 32 > /dev/null

real    0m0.484s
user    0m0.260s
sys     0m0.150s

With '-n 32' the total time is increased. It is probably due to VL15 packets drops on switches. This can be ensured by checking VL15Drops counter value. Also in this case using different from default timeout values with ./subnet_discover (for example --timeout=1000 (default is 100ms)) will affect the total running time (this doesn't happen when packets were not dropped and ib_mad retrying mechanism was not activated).

Quoted 11 lines

09:45:57 > time ibnetdiscover > /dev/null

real    0m3.180s
user    0m0.068s
sys     0m0.672s

What I find most interesting is that your test utility runs nearly 2x faster even when there is only 1 outstanding MAD. :-/ ibnetdiscover (libibnetdisc) does do a lot more with the data but I would not have expected such a difference.

Your fix for subnet_discover explains the difference.

Quoted 2 lines

I would appear that your algorithm is superior. I will look at converting libibnetdisc, test, and submit a patch.

Thanks.

Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html