infiniband-diags/libibnetdisc: Introduce a context object.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 20090816110200.GS25501@me
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 20090823120609.GG9547@me
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Ira Weiny
The discussion on the list has digressed from this patch. I still think this patch is valid and adds a level of flexibility which is needed regardless of what is decided about libibmad. Do you agree?
Not really. I still think that needed level of flexibility is available for us without adding new mandatory APIs.
Quoted 4 lines
Also, the last patch in the series ([PATCH 5/5] infiniband-diags/libibnetdisc:
remove members of the fabric struct which are used in the scan only) cleans up some stuff from the external interface. If you really don't want to introduce a context object, then I can regenerate that final patch without the context.
Yes, since this patch is unrelated logically to a context stuff. It would be good to rebase it back and repost as independent change.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: Multi-threaded diags (Was: Re: [PATCH 4/5] infiniband-diags/libibnetdisc: Introduce a context object.)
Quoted 31 lines · Ira Weiny
Of course! :-) But first I would like to mention some numbers from the prototype code I have. When running on a small fabric the additional overhead of thread creation actually slows down the scan. :-(
Current master: Threaded version:
real 0m0.101s 0m0.266s user 0m0.000s 0m0.000s sys 0m0.011s 0m0.014sBut, as expected, on a large system (1152 nodes) there is a decent speed up.
Current Master: Threaded version:
real 0m3.046s 0m1.748s user 0m0.073s 0m0.331s sys 0m0.158s 0m0.822sHowever, the biggest speed up comes when there are errors on the fabric. This is the same 1152 node cluster with just 14 "bad" ports on the fabric. This is of course because the scan continues "around" the bad ports.
Current Master: Threaded version:
real 0m33.051s 0m5.609s user 0m0.071s 0m0.353s sys 0m0.156s 0m1.113sSince you are usually running these tools when things are bad I think there is a big gain here. Even running with a faster timeout of 200ms results in a big difference.
Quoted 6 lines
Current Master: Threaded version:
real 0m9.149s 0m2.223s user 0m0.016s 0m0.374s sys 0m0.372s 0m1.056sWith that in mind...
Good. So what do you think due to which factor most of this performance gain was achieved? Due to using multiple threads or due to SMP queries parallelism? I would suspect that it is a parallelism.
Quoted 5 lines
Quoted 3 lines
implemented with libibnetdisc: goals (in particular is it support for multithreaded apps or just multithreaded discovery function), interaction with caller application, etc.?
My initial goal was to make the libibnetdisc safe for multithreaded apps
Ok.
Quoted 2 lines
and
make a multithreaded discovery function.
And this is not the same as above. I would really suggest before doing multithreaded implementation to check just single threaded but parallel querying first. Somehow I believe that an impressive numbers could be shown there and also without multithreaded overhead and complexity.
Quoted 11 lines
However, since libibmad itself is
not thread safe, and you expressed a desire to keep it that way[*], I reduced that goal to just making the discovery function multithreaded (using mad_[send|receive]_via).Although I don't like this restriction I can see it as a valid design decision as long as it is documented that the discover function is not thread safe in regards to the
ibmad_portobject. This is because theibnd_discover_fabricuses libibmad calls and would require a complicated API to allow the user app to synchronize with those calls.
Quoted 8 lines
In order to make things thread safe for the user apps as well as the library I can see 3 options.
1) make libibmad thread safe (which you were hesitant to do)
2) add a thread safe interface to libibmad. User apps will need to know to use this interface while using libibnetdisc and libibnetdisc will use this interface.
Why should it be related to libibmad? Make libibnetdisc by itself thread safe and that is all (see below).
Quoted 3 lines
3) Create a wrapper lib which is thread safe. In this case the apps and libibnetdisc would call into this wrapper lib and we would have to change the API to libibnetdisc.
4) Instead of bothering with slow (blocked) libibmad rpc use umad_send() and umad_recv() directly in parallel hop by hop fabric discovery.
(And following my personal experience it is ended in smaller and simpler code using umad directly instead of rpc for applications which slightly more complex than 'smpquery' style tools).
Quoted 5 lines
[*] http://lists.openfabrics.org/pipermail/general/2009-July/060677.html
"
madrpc()is too primitive interface for such applications. There would be better to use umad_send/recv() directly or may bemad_send_via(). Example ismcast_storm.cdistributed with ibsim."
Another option is adding a new thread safe API. But I don't really see how it is related to libibnetdisc discussion.
Quoted 8 lines
[$] It is my opinion that
mad_rpcis _not_ primitive. In my mind it _is_
the wrapper around the primitive umad_send/recv calls. If you are interested perhaps I can try to explain what I wanted to do in the library to make it thread safe more clearly. The point I might not have made clear was that I don't think the library will have to do any threading on it's own, just some locks and storing of responses. Of course the down side to this is the libibmad code would be slightly slower. But I don't think by very much.
I don't like a lock introduction there - right, it is unneeded overhead (and dependencies) for single threaded tools. And again - I have nothing against thread safe rpc++ API, but would prefer instead of locking use reenrant states.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 1/1 tests/subnet_discover: discover test utility
'subnet_discover' is simple test utility which implements "non-blocking" discovery method where mads are sending "in parallel" (unlike the current implementation of 'ibnetdiscover' and similar to how OpenSM does). For this a recently discovered node id value is encoded as lower 16 bits of mad transaction id.
---
Hi Ira,
Quoted 11 lines · Sasha Khapyorsky
Quoted 7 lines
Current Master: Threaded version:
real 0m9.149s 0m2.223s user 0m0.016s 0m0.374s sys 0m0.372s 0m1.056sWith that in mind...
Good. So what do you think due to which factor most of this performance gain was achieved? Due to using multiple threads or due to SMP queries parallelism? I would suspect that it is a parallelism.
For some purposes in ibsim/tests I wrote a simple utility 'subnet_discover', this works as single thread and utilizes a "parallel" mad sending method and also uses libibumad for all mad sending/receiving stuff.
I think that similar implementation in libibnetdisc (I can do it if we are in agreement :)) would improve its performance.
Would you like to look at this?
Sasha
tests/Makefile | 2 +- tests/subnet_discover.c | 495 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 496 insertions(+), 1 deletions(-) create mode 100644 tests/subnet_discover.c
tests/Makefile +1 -1
@@ -1,4 +1,4 @@ -progs:= mcast_storm +progs:= subnet_discover mcast_storm -include ../defs.mk
tests/subnet_discover.c new file +495 -0
495 lines not on this page · raw message
Signature · 6 lines
1.6.6.rc3 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] tests/subnet_discover: discover test utility
Hi Ira,
Quoted 2 lines · Ira Weiny
Yes, a similar mechanism would work in libibnetdisc. However, it looks like you are doing a depth first search
I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.
Quoted 1 line
which I fear might exceed the path count limit in a DR path?
Right, hops count should be limited. I will add this.
Quoted 1 line
Is this how OpenSM works?
I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and subnet_discover doesn't (and there could be a lot of MADs).
Quoted 7 lines
I was trying to do a breath first search like ibnetdiscover does.
Quoted 2 lines
Would you like to look at this?
No problem, I have enclosed the output from a run on Hyperion.
There appears to be a lot of errors. I am not sure what the issue is right off. I have included an ibnetdiscover output for comparision.
Thanks.
An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
Quoted 9 lines · Sasha Khapyorsky
Hi Ira,
Quoted 2 lines · Ira Weiny
Yes, a similar mechanism would work in libibnetdisc. However, it looks like you are doing a depth first search
I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.
Does anything limit the amount of parallelism ?
Quoted 10 lines
Quoted 2 lines
which I fear might exceed the path count limit in a DR path?
Right, hops count should be limited. I will add this.
Quoted 1 line
Is this how OpenSM works?
I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and
subnet_discoverdoesn't (and there could be a lot of MADs).
Isn't that dangerous in a real subnet ?
-- Hal
Quoted 15 lines
Quoted 8 lines
I was trying to do a breath first search like ibnetdiscover does.
Quoted 2 lines
Would you like to look at this?
No problem, I have enclosed the output from a run on Hyperion.
There appears to be a lot of errors. I am not sure what the issue is right off. I have included an ibnetdiscover output for comparision.Thanks.
An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.
Quoted 6 lines
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): f0e08f230912210602i5e3f528h2b0630420346db82-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] tests/subnet_discover: discover test utility
Quoted 5 lines · Hal Rosenstock
Quoted 3 lines
I wouldn't call it so, it is rather "parallel" than "first" depth or breath - discovery continues at first responding node doesn't matter how was it queried in depth or in breath.
Does anything limit the amount of parallelism ?
Nothing in this version.
Quoted 5 lines
Quoted 3 lines
I think so. The difference is that OpenSM has a limit of outstanding MADs on the wire and
subnet_discoverdoesn't (and there could be a lot of MADs).Isn't that dangerous in a real subnet ?
This is test tool, so I don't think that it is "dangerous". And of course this can lead to an issues with test results.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
In reply to Sasha KhapyorskyHi Ira,
Quoted 4 lines · Sasha Khapyorsky
An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.
I did a couple of modifications in the code (exact log is listed below). In particular there are default limitation for number of outstanding MADs on the wire and proper tracking for failed (timedout) MADs. I tested this where possible. Could you re-run this? Thanks.
Sasha
commit 9e24853c30351f6ea65ffcccf184bdf7586dfe8e
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> Date: Fri Dec 25 13:01:35 2009 +0200
tests/subnet_discover: limit possible number of hops
As pointed out by Ira there was no hops limitation. Adding this.
tests/subnet_discover.c +5 -2
@@ -17,6 +17,8 @@#include <infiniband/umad.h> #include <infiniband/mad.h> +#define MAX_HOPS 63 +struct port {struct node *node;uint64_t guid;
@@ -217,8 +219,9 @@ static int process_port_info(void *umad, unsigned node_id, int fd, int agent,if (port_num &&mad_get_field (port_info,0 ,IB_PORT_PHYS_STATE_F ) ==5 && ((node->is_switch && port_num != local_port) || - (node_id ==0 && port_num == local_port))) { - path[++path_cnt] = port_num; + (node_id ==0 && port_num == local_port)) && + path_cnt++ <MAX_HOPS ) { + path[path_cnt] = port_num;return query_node_info (fd, agent, umad, node_id, path, path_cnt); }
commit 6e7817433c17bf2b8861639852dc0e70e8d0ec5f
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> Date: Fri Dec 25 16:11:53 2009 +0200
tests/subnet_discover: add --help option
Add --help command line option. Also cosmetic improvements.
tests/subnet_discover.c +10 -7
@@ -70,7 +70,7 @@ DBG_DUMP_FUNC(nodedesc);DBG_DUMP_FUNC (portinfo);DBG_DUMP_FUNC (switchinfo); -static void build_umad_req (void *umad,uint8_t * path,unsigned path_cnt, +static void build_umad_req (void *umad,uint8_t path[],unsigned path_cnt,uint64_t trid,uint8_t method,uint16_t attr_id,uint32_t attr_mod,uint64_t mkey) {
@@ -94,7 +94,7 @@ static void build_umad_req(void *umad, uint8_t * path, unsigned path_cnt, }static int send_query (int fd,int agent,void *umad,unsigned node_id, -uint8_t * path,size_t path_cnt,uint16_t attr_id, +uint8_t path[],size_t path_cnt,uint16_t attr_id,uint32_t attr_mod) {uint64_t trid;
@@ -138,28 +138,28 @@ static int recv_response(int fd, int agent, uint8_t * umad, size_t length) }static int query_node_info (int fd,int agent,void *umad,unsigned node_id, -uint8_t * path,size_t path_cnt) +uint8_t path[],size_t path_cnt) {return send_query (fd, agent, umad, node_id, path, path_cnt,IB_ATTR_NODE_INFO ,0 ); }static int query_node_desc (int fd,int agent,void *umad,unsigned node_id, -uint8_t * path,size_t path_cnt) +uint8_t path[],size_t path_cnt) {return send_query (fd, agent, umad, node_id, path, path_cnt,IB_ATTR_NODE_DESC ,0 ); }static int query_switch_info (int fd,int agent,void *umad,unsigned node_id, -uint8_t * path,size_t path_cnt) +uint8_t path[],size_t path_cnt) {return send_query (fd, agent, umad, node_id, path, path_cnt,IB_ATTR_SWITCH_INFO ,0 ); }static int query_port_info (int fd,int agent,void *umad,unsigned node_id, -uint8_t * path,size_t path_cnt,unsigned port_num) +uint8_t path[],size_t path_cnt,unsigned port_num) {return send_query (fd, agent, umad, node_id, path, path_cnt,IB_ATTR_PORT_INFO , port_num);
@@ -456,6 +456,8 @@ int main(int argc, char **argv) {"Port", 1, 0, 'P'}, {"timeout", 1, 0, 't'}, {"retries", 1, 0, 'r'}, + {"verbose", 0, 0, 'v'}, + {"help", 0, 0, 'h'}, {} };char *card_name =NULL ;
@@ -463,7 +465,7 @@ int main(int argc, char **argv)int ch, ret;while (1 ) { - ch =getopt_long (argc, argv,"C:P:t:r:v" , long_opts,NULL ); + ch =getopt_long (argc, argv,"C:P:t:r:vh" , long_opts,NULL );if (ch ==-1 )break ;switch (ch) {
@@ -482,6 +484,7 @@ int main(int argc, char **argv)case 'v' : verbose++;break ; +case 'h' :default :printf ("usage: %s [-C card_name] [-P port_num]" " [-t timeout] [-r retries] [-v[v]]\n" , argv[0 ]);
commit da6aa19840cb2d37e8cd3daa3874b87657a76ddc
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> Date: Fri Dec 25 16:24:13 2009 +0200
tests/subnet_discover: --maxsmps (-n) option
This implements the limitation of outstanding SMPs on a wire at any one time. --maxsmps=0 means - no limit.
tests/subnet_discover.c +105 -34
265 lines not on this page · raw message
commit a422ea90334441144f2a1212de40085bbe36cf7e
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> Date: Sun Dec 27 18:55:35 2009 +0200
tests/subnet_discover.c: print useful information
Print additional useful information about a subnet and discovery
process: such as number of MADs used, number of hops reached, direct paths for nodes as it was discovered. Better error messages (in particular - don't print MAD content in error message when returned valid data from umad_recv() is only umad header).
tests/subnet_discover.c +23 -8
@@ -30,6 +30,8 @@ struct node {uint64_t guid;unsigned num_ports;unsigned is_switch; +size_t path_size; +uint8_t path[64 ];uint8_t node_info[IB_SMP_DATA_SIZE ];uint8_t node_desc[IB_SMP_DATA_SIZE ];uint8_t switch_info[IB_SMP_DATA_SIZE ];
@@ -45,6 +47,9 @@ static unsigned timeout = 100;static unsigned retries =3 ;static unsigned verbose =0 ; +static unsigned total_mads =0 ; +static unsigned max_hops =0 ; +#define ERROR (fmt, ...) fprintf(stderr, "ERR: " fmt, ##__VA_ARGS__)#define VERBOSE (fmt, ...) if (verbose) fprintf(stderr, fmt, ##__VA_ARGS__)#define NOISE (fmt, ...) if (verbose > 1) fprintf(stderr, fmt, ##__VA_ARGS__)
@@ -142,6 +147,7 @@ static void run_request_queue(int fd, int agent) q = q->next;free (prev); outstanding++; + total_mads++; } request_queue.next = q;if (!q)
@@ -201,10 +207,10 @@ static int recv_response(int fd, int agent, uint8_t * umad, size_t length)if (ret <0 ||umad_status (umad)) {ERROR ("umad_recv failed: umad status %x: %s\n" ,umad_status (umad),strerror (errno)); -return -1 ; +return len >umad_size () ?1 :-1 ; } -return ret; +return 0 ; } static int query_node_info(int fd, int agent, unsigned node_id,
@@ -235,7 +241,7 @@ static int query_port_info(int fd, int agent, unsigned node_id, IB_ATTR_PORT_INFO, port_num); } -static int add_node (uint8_t * node_info) +static int add_node (uint8_t * node_info,uint8_t path[],size_t path_size) {struct node *node;unsigned i, num_ports =mad_get_field (node_info,0 ,IB_NODE_NPORTS_F );
@@ -250,6 +256,8 @@ static int add_node(uint8_t * node_info) node->guid =mad_get_field64 (node_info,0 ,IB_NODE_GUID_F ); node->is_switch = ((mad_get_field (node_info,0 ,IB_NODE_TYPE_F )) ==IB_NODE_SWITCH ); +memcpy (node->path, path, path_size +1 ); + node->path_size = path_size;memcpy (node->node_info, node_info,sizeof (node->node_info));for (i =0 ; i <= num_ports; i++) node->ports[i].node = node;
@@ -291,6 +299,8 @@ static int process_port_info(void *umad, unsigned node_id, int fd, int agent, ((node->is_switch && port_num != local_port) || (node_id == 0 && port_num == local_port)) && path_cnt++ < MAX_HOPS) { +if (path_cnt > max_hops) + max_hops = path_cnt; path[path_cnt] = port_num;return query_node_info (fd, agent, node_id, path, path_cnt); }
@@ -341,7 +351,7 @@ static int process_node(void *umad, unsigned remote_id, int fd, int agent,dbg_dump_nodeinfo (node_info);if ((id =find_node (node_info)) <0 ) { - id =add_node (node_info); + id =add_node (node_info, path, path_cnt);if (id <0 )return -1 ; node_is_new =1 ;
@@ -398,7 +408,9 @@ static int recv_smp_resp(int fd, int agent, uint8_t * umad, uint8_t path[]) outstanding--;run_request_queue (fd, agent); -if (ret <0 || status) { +if (ret <0 ) +return ret; +else if (ret || status) {ERROR ("error response 0x%016" PRIx64": attr_id %x" ", attr_mod %x from %s with status %x\n" , trid, attr_id, attr_mod,print_path (path, path_cnt), status);
@@ -477,7 +489,7 @@ static int umad_discover(char *card_name, unsigned int port_num) ret =discover (fd, agent);if (ret) -ERROR ("Failed to discover.\n" ); +fprintf (stderr,"\nThere are problems during discovery.\n" );umad_unregister (fd, agent);umad_close_port (fd);
@@ -493,12 +505,15 @@ static void print_subnet()struct port *local, *remote;unsigned i, j; +printf ("\n# The subnet discovered using %u mads, reaching %d hops\n\n" , + total_mads, max_hops); +for (i =0 ; i < node_count; i++) { node = node_array[i]; -printf ("%s %u \"%s-%016" PRIx64"\" \t# %s\n" , +printf ("%s %u \"%s-%016" PRIx64"\" \t# %s %s\n" , node->is_switch ?"Switch" :"Ca" , node->num_ports, node->is_switch ?"S" :"H" , node->guid, - node->node_desc); +print_path (node->path, node->path_size), node->node_desc);for (j =1 ; j <= node->num_ports; j++) { local = &node->ports[j]; remote = local->remote;
commit 4a23f9e7f339e93f2a77f213d4ce80e4bc7d7b9f
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]> Date: Sun Dec 27 21:19:30 2009 +0200
tests/subnet_discover: report unresponded transactions
Report unresponded transactions (requests) in case of MAD failures.
tests/subnet_discover.c +58 -3
94 lines not on this page · raw message
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
Hi Sasha,
Quoted 15 lines · Sasha Khapyorsky
Hi Ira,
Quoted 4 lines · Sasha Khapyorsky
An errors are response timeouts. I guess that most of them are due to switches' VL15 overflow (could be verified by VL15Dropped counter evaluation). Will look at this deeply.
I did a couple of modifications in the code (exact log is listed below). In particular there are default limitation for number of outstanding MADs on the wire and proper tracking for failed (timedout) MADs. I tested this where possible. Could you re-run this? Thanks.
Sasha
[snip...]
Quoted 16 lines
commit
da6aa19840cb2d37e8cd3daa3874b87657a76ddc
Author: Sasha Khapyorsky <sashak-smomgflXvOZWk0Htik3J/[email protected]>
Date: Fri Dec 25 16:24:13 2009 +0200tests/subnet_discover: --maxsmps (-n) option
This implements the limitation of outstanding SMPs on a wire at any one time. --maxsmps=0 means - no limit.
tests/subnet_discover.c +0
-0
tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43
Quoted 4 lines
tests/subnet_discover.c +1
-0@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024]; static unsigned node_count = 0; static unsigned trid_cnt = 0; static unsigned outstanding = 0; +static unsigned max_outstanding =8 ;
Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).
-- Hal
tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 46
Quoted 3 lines
tests/subnet_discover.c +0
-0static unsigned timeout = 100; static unsigned retries = 3; static unsigned verbose = 0;
[snip...]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): f0e08f231001110556y7c47cc54oa3cfd5859f9a4e76-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] tests/subnet_discover: discover test utility
Hi Hal,
Quoted 13 lines · Hal Rosenstock
tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43
Quoted 10 lines
tests/subnet_discover.c +1
-0@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024]; static unsigned node_count = 0; static unsigned trid_cnt = 0; static unsigned outstanding = 0; +static unsigned max_outstanding =8 ;Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).
In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.
Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
Hi Sasha,
Quoted 17 lines · Sasha Khapyorsky
Hi Hal,
Quoted 13 lines · Hal Rosenstock
tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43
Quoted 10 lines
tests/subnet_discover.c +1
-0@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];static unsigned node_count =0 ;static unsigned trid_cnt =0 ;static unsigned outstanding =0 ; +static unsigned max_outstanding =8 ;Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).
Quoted 5 lines
In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.
Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.
This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.
There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.
Also, the OpenSM default should be at least as large as the diags for this.
-- Hal
Quoted 2 lines
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): f0e08f231001131211y64489a51nd2621cefdb27ad25-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] tests/subnet_discover: discover test utility
Quoted 10 lines · Hal Rosenstock
Quoted 6 lines
In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.
Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.
This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.
Was it investigated? What was a conclusions?
Quoted 5 lines
There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.
Also, the OpenSM default should be at least as large as the diags for this.
I don't know where concurrent MADs are used in the current "diags", do you?
This utility is not 'diags' at all. It is placed in ibsim/tests and I wrote it for test purpose.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
Quoted 5 lines · Sasha Khapyorsky
Quoted 4 lines · Hal Rosenstock
Quoted 4 lines
In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.
This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.
Was it investigated? What was a conclusions?
I was pointing out testing that needs to be done rather than testing already done. The use cases I mentioned are based on current experience. I'm sure there are more.
Quoted 11 lines
Quoted 5 lines
There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.
Also, the OpenSM default should be at least as large as the diags for this.
I don't know where concurrent MADs are used in the current "diags", do you?
This utility is not 'diags' at all. It is placed in ibsim/tests and I wrote it for test purpose.
In your original email on this, you proposed that this algorithm be incorporated in the diags (you wrote "I think that similar implementation in libibnetdisc (I can do it if we
are in agreement :)) would improve its performance") so this comment thread appears relevant to me.
-- Hal
Quoted 2 lines
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] tests/subnet_discover: discover test utility
In reply to a message not in the archiveHey Sasha,
I am finally getting back to this... Sorry.
Quoted 25 lines · On Wed, 13 Jan 2010 15:11:44 -0500 Hal Rosenstock…
Hi Sasha,
Quoted 22 lines · Sasha Khapyorsky
Hi Hal,
Quoted 13 lines · Hal Rosenstock
tests/subnet_discover.c static struct node *node_array[32 * 1024]; · line 43
Quoted 10 lines
tests/subnet_discover.c +1
-0@@ -40,6 +40,7 @@ static struct node *node_array[32 * 1024];static unsigned node_count =0 ;static unsigned trid_cnt =0 ;static unsigned outstanding =0 ; +static unsigned max_outstanding =8 ;Any reason why this default is different from the one which OpenSM uses ? Seems to me it should be the same (or less).
In my tests I found that '8' is more optimal number (the tool works faster and without drops) than '4' used in OpenSM.
Of course it would be helpful to run this over bigger cluster than what I have to see that the results are consistent.
Here is some test data on a real cluster.
09:49:10 > ibhosts | wc -l
1158
09:49:28 > ibswitches | wc -l
281
09:44:45 > time ./subnet_discover -n 1 > /dev/null
real 0m1.414s user 0m0.309s sys 0m0.244s
09:44:55 > time ./subnet_discover -n 2 > /dev/null
real 0m1.025s user 0m0.284s sys 0m0.201s
09:45:00 > time ./subnet_discover -n 4 > /dev/null
real 0m0.644s user 0m0.268s sys 0m0.228s
09:45:04 > time ./subnet_discover -n 8 > /dev/null
real 0m0.550s user 0m0.253s sys 0m0.184s
09:45:08 > time ./subnet_discover -n 12 > /dev/null
real 0m0.524s user 0m0.207s sys 0m0.201s
09:45:14 > time ./subnet_discover -n 16 > /dev/null
real 0m0.432s user 0m0.248s sys 0m0.144s
09:45:18 > time ./subnet_discover -n 32 > /dev/null
real 0m0.484s user 0m0.260s sys 0m0.150s
09:45:57 > time ibnetdiscover > /dev/null
real 0m3.180s user 0m0.068s sys 0m0.672s
What I find most interesting is that your test utility runs nearly 2x faster even when there is only 1 outstanding MAD. :-/ ibnetdiscover (libibnetdisc) does do a lot more with the data but I would not have expected such a difference.
As a comparison I ran iblinkinfo it would seem that there is something in the library which takes a lot more time.
09:51:59 > time iblinkinfo > /dev/null
real 0m3.159s user 0m0.063s sys 0m0.526s
For further comparison I rebuilt the parallel version of libibnetdisc.
12:39:02 > time ./ibnetdiscover > /dev/null
real 0m2.552s user 0m0.295s sys 0m0.863s
This is with 8 threads (ie 8 outstanding SMP's).
I would appear that your algorithm is superior. I will look at converting libibnetdisc, test, and submit a patch. I still don't know why there would be so much difference when only using 1 outstanding MAD though? :-/
Quoted 8 lines
This is exactly my concern. Not only cluster size but use cases including concurrent diag discover and SM operation where SMPs are heavily in use.
There already have been a number of reports of dropped SMPs on this list with the current diags and this change will only make things worse IMO.
This is a problem. I have seen this issue with large systems which are having trouble. OpenSM is trying to discover and route. We are running diags trying to figure out what is going on. There is hardware going up and down; bad switches or nodes which are booting/rebooting.
I plan to go forward with this but having an option for outstanding MAD's is a good idea. I don't have an opinion on where it should default.
Quoted 2 lines
Also, the OpenSM default should be at least as large as the diags for this.
I agree. OpenSM should have some priority in this matter.
Ira
Quoted 5 lines
-- Hal
Quoted 2 lines
Sasha
Signature · 9 lines
Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 [email protected] -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH] tests/subnet_discover: discover test utility
Hi Ira,
Quoted 39 lines · Ira Weiny
Here is some test data on a real cluster.
09:49:10 > ibhosts | wc -l
115809:49:28 > ibswitches | wc -l
28109:44:45 > time ./subnet_discover -n 1 >
/dev/nullreal 0m1.414s user 0m0.309s sys 0m0.244s09:44:55 > time ./subnet_discover -n 2 >
/dev/nullreal 0m1.025s user 0m0.284s sys 0m0.201s09:45:00 > time ./subnet_discover -n 4 >
/dev/nullreal 0m0.644s user 0m0.268s sys 0m0.228s09:45:04 > time ./subnet_discover -n 8 >
/dev/nullreal 0m0.550s user 0m0.253s sys 0m0.184s09:45:08 > time ./subnet_discover -n 12 >
/dev/nullreal 0m0.524s user 0m0.207s sys 0m0.201s
Quoted 5 lines
09:45:14 > time ./subnet_discover -n 16 >
/dev/nullreal 0m0.432s user 0m0.248s sys 0m0.144s
Looks like a very nice results. Thanks for probing this.
Quoted 5 lines
09:45:18 > time ./subnet_discover -n 32 >
/dev/nullreal 0m0.484s user 0m0.260s sys 0m0.150s
With '-n 32' the total time is increased. It is probably due to VL15 packets drops on switches. This can be ensured by checking VL15Drops counter value. Also in this case using different from default timeout values with ./subnet_discover (for example --timeout=1000 (default is 100ms)) will affect the total running time (this doesn't happen when packets were not dropped and ib_mad retrying mechanism was not activated).
Quoted 11 lines
09:45:57 > time ibnetdiscover >
/dev/nullreal 0m3.180s user 0m0.068s sys 0m0.672sWhat I find most interesting is that your test utility runs nearly 2x faster even when there is only 1 outstanding MAD. :-/ ibnetdiscover (libibnetdisc) does do a lot more with the data but I would not have expected such a difference.
Your fix for subnet_discover explains the difference.
Quoted 2 lines
I would appear that your algorithm is superior. I will look at converting libibnetdisc, test, and submit a patch.
Thanks.
Sasha
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html