Using multi-smps on the wire in libibnetdisc

PATCH Superseded Ira Weiny 2 participants 9 messages Feb 3, 2010 – Feb 6, 2010 7 messages not in the archive InfiniBand

lore

Share & tools

lore

Ira Weiny

author

2010-02-03 00:45 UTC #0 lore raw

cover Using multi-smps on the wire in libibnetdisc

Sasha,

Following up on our thread regarding having multiple outstanding SMP's in libibnetdisc.

These 2 patches implement that as well as add a function to set the max outstanding the lib will use.

I left the default here to be 4. On a large cluster there seems to be some variance with using 8 or 12. Sometimes I get a speed up over 4 and other times I don't see any. I think it has to do with the traffic on the fabric at any particular time.

For example here are some runs I just did on Hyperion.

14:31:55 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 14562048] [RcvData == 14563872] [XmtPkts == 202255] [RcvPkts == 202276]

       Link info:    139   9[  ] ==( 4X 5.0 Gbps Active/  LinkUp)==>  0x0002c9030001d736    864    1[  ] "hyperion1" ( )

14:32:02 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new

real    0m2.210s
user    0m1.251s
sys     0m0.869s

14:40:36 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new

real    0m3.385s
user    0m1.888s
sys     0m1.448s

14:40:46 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new

real    0m2.211s
user    0m1.165s
sys     0m0.951s

14:40:51 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new

real    0m2.249s
user    0m1.244s
sys     0m0.936s

14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new

real    0m2.170s
user    0m1.160s
sys     0m0.933s

14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954]

       Link info:    139   9[  ] ==( 4X 5.0 Gbps Active/  LinkUp)==>  0x0002c9030001d736    864    1[  ] "hyperion1" ( )

Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)

The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.

Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.

Ira

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hal Rosenstock

2010-02-04 14:19 UTC #2 lore raw

Quoted 46 lines · Ira Weiny

Sasha,
Following up on our thread regarding having multiple outstanding SMP's in libibnetdisc.
These 2 patches implement that as well as add a function to set the max outstanding the lib will use.
I left the default here to be 4. On a large cluster there seems to be some variance with using 8 or 12. Sometimes I get a speed up over 4 and other times I don't see any. I think it has to do with the traffic on the fabric at any particular time.
For example here are some runs I just did on Hyperion.
14:31:55 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 14562048] [RcvData == 14563872] [XmtPkts == 202255] [RcvPkts == 202276] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )
14:32:02 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.210s
user 0m1.251s
sys 0m0.869s
14:40:36 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m3.385s
user 0m1.888s
sys 0m1.448s
14:40:46 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.211s
user 0m1.165s
sys 0m0.951s
14:40:51 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.249s
user 0m1.244s
sys 0m0.936s
14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.170s
user 0m1.160s
sys 0m0.933s

Quoted 7 lines

14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )
Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)

Is this just with the SM doing light sweeping ?

Is there a speedup with 4 rather than 2 ?

-- Hal

Quoted 7 lines

The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira

Quoted 11 lines

--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f231002040619y34784bauec6a25d31e7d229b-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Ira Weiny

author

2010-02-04 18:00 UTC #4 lore raw

Quoted 53 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…

Quoted 52 lines · Ira Weiny
Sasha,
Following up on our thread regarding having multiple outstanding SMP's in libibnetdisc.
These 2 patches implement that as well as add a function to set the max outstanding the lib will use.
I left the default here to be 4. On a large cluster there seems to be some variance with using 8 or 12. Sometimes I get a speed up over 4 and other times I don't see any. I think it has to do with the traffic on the fabric at any particular time.
For example here are some runs I just did on Hyperion.
14:31:55 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 14562048] [RcvData == 14563872] [XmtPkts == 202255] [RcvPkts == 202276] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )
14:32:02 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.210s
user 0m1.251s
sys 0m0.869s
14:40:36 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m3.385s
user 0m1.888s
sys 0m1.448s
14:40:46 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.211s
user 0m1.165s
sys 0m0.951s
14:40:51 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.249s
user 0m1.244s
sys 0m0.936s
14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.170s
user 0m1.160s
sys 0m0.933s
14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )

Quoted 3 lines

Quoted 1 line
Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)
Is this just with the SM doing light sweeping ?

Yes.

Quoted 2 lines

Is there a speedup with 4 rather than 2 ?

There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.

Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.

Ira

Quoted 26 lines

-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://*vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://*vger.kernel.org/majordomo-info.html

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hal Rosenstock

2010-02-04 20:01 UTC #6 lore raw

Quoted 56 lines · Ira Weiny

Quoted 53 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…
Quoted 52 lines · Ira Weiny
Sasha,
Following up on our thread regarding having multiple outstanding SMP's in libibnetdisc.
These 2 patches implement that as well as add a function to set the max outstanding the lib will use.
I left the default here to be 4. On a large cluster there seems to be some variance with using 8 or 12. Sometimes I get a speed up over 4 and other times I don't see any. I think it has to do with the traffic on the fabric at any particular time.
For example here are some runs I just did on Hyperion.
14:31:55 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 14562048] [RcvData == 14563872] [XmtPkts == 202255] [RcvPkts == 202276] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )
14:32:02 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.210s
user 0m1.251s
sys 0m0.869s
14:40:36 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m3.385s
user 0m1.888s
sys 0m1.448s
14:40:46 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.211s
user 0m1.165s
sys 0m0.951s
14:40:51 > time ./ibnetdiscover -o 8 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.249s
user 0m1.244s
sys 0m0.936s
14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.170s
user 0m1.160s
sys 0m0.933s
14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )

Quoted 5 lines

Quoted 3 lines
Quoted 1 line
Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)
Is this just with the SM doing light sweeping ?
Yes.

That's not a lot of SMP stress from the SM side. SMP consumers are SM, diags, and the unsolicited traps.

Quoted 12 lines

Quoted 3 lines
Is there a speedup with 4 rather than 2 ?
There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.

Quoted 3 lines

Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.

I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.

-- Hal

Quoted 30 lines

Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://*vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://*vger.kernel.org/majordomo-info.html

Quoted 7 lines

--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f231002041201y2b48aa30nc33d2816ed11b1bf-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Ira Weiny

author

2010-02-05 00:13 UTC #8 lore raw

Quoted 7 lines · On Thu, 4 Feb 2010 15:01:32 -0500 Hal Rosenstock…

Quoted 6 lines · Ira Weiny
Quoted 3 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…
Quoted 2 lines · Ira Weiny
Sasha,

[snip]

Quoted 17 lines

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
real 0m2.249s
user 0m1.244s
sys 0m0.936s
14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.170s
user 0m1.160s
sys 0m0.933s
14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )

Quoted 8 lines

Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)
Is this just with the SM doing light sweeping ?
Yes.
That's not a lot of SMP stress from the SM side. SMP consumers are SM, diags, and the unsolicited traps.

Agreed. I hope to test this more next week.

Quoted 13 lines

Quoted 13 lines
Quoted 4 lines
Is there a speedup with 4 rather than 2 ?
There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.

Quoted 7 lines

Quoted 3 lines
Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.
I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.

So OpenSM is seeing dropped packets? With 4 SMP's on the wire? I do see some VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an issue. What kind of rate are you seeing?

The other question is; do people regularly run the tools which are using libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)? We do. If others are not then I would say this change would have less impact as they would want the diags to have some priority for debugging. The other option is to change the patch to be a default of 2 and allow user to change it depending on what they are trying to do. If you think that is best I will change the patch.

Ira

Quoted 41 lines

-- Hal
Quoted 37 lines
Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Ira Weiny

author

2010-02-05 02:18 UTC #10 lore raw

Quoted 12 lines · On Thu, 4 Feb 2010 16:13:25 -0800 Ira Weiny…

Quoted 7 lines · On Thu, 4 Feb 2010 15:01:32 -0500 Hal Rosenstock…
Quoted 6 lines · Ira Weiny
Quoted 3 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…
Quoted 2 lines · Ira Weiny
Sasha,
[snip]

[snip]

Quoted 10 lines

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Is there a speedup with 4 rather than 2 ?
There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.

Ok, I found a fabric with just 2 nodes which were unresponsive... A quick test shows...

Original ibnetdiscover:

18:12:29 > time ./ibnetdiscover > foo
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port

real    0m9.073s
user    0m0.137s
sys     0m0.172s

18:12:43 > time ./ibnetdiscover > foo
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port

real    0m9.103s
user    0m0.046s
sys     0m0.046s

*New* ibnetdiscover with different outstanding SMP's.

18:12:14 > time ./ibnetdiscover -o 2 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m9.746s
user    0m6.559s
sys     0m3.156s

18:13:00 > time ./ibnetdiscover -o 4 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m4.668s
user    0m3.043s
sys     0m1.601s

18:13:10 > time ./ibnetdiscover -o 8 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

real    0m4.360s
user    0m2.891s
sys     0m1.451s

Note that 2 does not give much speed up, where 4 does. Obviously this could have to do with the fact there were 2 nodes which were bad (so if you had 100's of nodes unresponsive a higher value might be worth using) but as a default compromise I think 4 is good.

Ira

Quoted 71 lines

Quoted 8 lines
Quoted 4 lines
Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.
I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.
So OpenSM is seeing dropped packets? With 4 SMP's on the wire? I do see some VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an issue. What kind of rate are you seeing?
The other question is; do people regularly run the tools which are using libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)? We do. If others are not then I would say this change would have less impact as they would want the diags to have some priority for debugging. The other option is to change the patch to be a default of 2 and allow user to change it depending on what they are trying to do. If you think that is best I will change the patch.
Ira
Quoted 41 lines
-- Hal
Quoted 37 lines
Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
Signature · 5 lines
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hal Rosenstock

2010-02-05 12:27 UTC #12 lore raw

Quoted 77 lines · Ira Weiny

Quoted 12 lines · On Thu, 4 Feb 2010 16:13:25 -0800 Ira Weiny…
Quoted 7 lines · On Thu, 4 Feb 2010 15:01:32 -0500 Hal Rosenstock…
Quoted 6 lines · Ira Weiny
Quoted 3 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…
Quoted 2 lines · Ira Weiny
Sasha,
[snip]
[snip]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Is there a speedup with 4 rather than 2 ?
There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.
Ok, I found a fabric with just 2 nodes which were unresponsive... A quick test shows...
Original ibnetdiscover:
18:12:29 > time ./ibnetdiscover > foo
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [26993] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port
real 0m9.073s
user 0m0.137s
sys 0m0.172s
18:12:43 > time ./ibnetdiscover > foo
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,11,9) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,11,9) failed, skipping port
ibwarn: [31111] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) src/ibnetdisc.c:457; Query remote node (DR path slid 0; dlid 0; 0,1,24,24,18,7,6) failed, skipping port
real 0m9.103s
user 0m0.046s
sys 0m0.046s
*New* ibnetdiscover with different outstanding SMP's.
18:12:14 > time ./ibnetdiscover -o 2 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out
real 0m9.746s
user 0m6.559s
sys 0m3.156s
18:13:00 > time ./ibnetdiscover -o 4 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out
real 0m4.668s
user 0m3.043s
sys 0m1.601s
18:13:10 > time ./ibnetdiscover -o 8 > foo
src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,11,9 Attr 0x11:0) bad status 110; Connection timed out src/query_smp.c:185; umad (DR path slid 0; dlid 0; 0,1,13,13,7,7,6 Attr 0x11:0) bad status 110; Connection timed out

Quoted 8 lines

real 0m4.360s
user 0m2.891s
sys 0m1.451s
Note that 2 does not give much speed up, where 4 does. Obviously this could have to do with the fact there were 2 nodes which were bad (so if you had 100's of nodes unresponsive a higher value might be worth using)

It depends on the number of unresponsive nodes being same or higher than number of outstanding/parallel SMPs. In a sense, the number of outstanding SMPs is a measure of how many unresponsive nodes one is willing to tolerate before slowing down/waiting for timeouts. In some environments, unresponsive nodes are a normal case.

-- Hal

Quoted 78 lines

but as a
default compromise I think 4 is good.
Ira
Quoted 65 lines
Quoted 8 lines
Quoted 4 lines
Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.
I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.
So OpenSM is seeing dropped packets? With 4 SMP's on the wire? I do see some VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an issue. What kind of rate are you seeing?
The other question is; do people regularly run the tools which are using libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)? We do. If others are not then I would say this change would have less impact as they would want the diags to have some priority for debugging. The other option is to change the patch to be a default of 2 and allow user to change it depending on what they are trying to do. If you think that is best I will change the patch.
Ira
Quoted 41 lines
-- Hal
Quoted 37 lines
Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
Quoted 6 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

Quoted 7 lines

--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): f0e08f231002050427r37d326b4w4c188e30992579ac-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Ira Weiny

author

2010-02-06 01:03 UTC #14 lore raw

Quoted 11 lines · On Fri, 5 Feb 2010 07:27:05 -0500 Hal Rosenstock…

Quoted 5 lines
Note that 2 does not give much speed up, where 4 does. Obviously this could have to do with the fact there were 2 nodes which were bad (so if you had 100's of nodes unresponsive a higher value might be worth using)
It depends on the number of unresponsive nodes being same or higher than number of outstanding/parallel SMPs. In a sense, the number of outstanding SMPs is a measure of how many unresponsive nodes one is willing to tolerate before slowing down/waiting for timeouts. In some environments, unresponsive nodes are a normal case.

Agreed but where should we set the default? I don't think 4 is a bad default. I don't think it makes the diags overly aggressive, compared with OpenSM. Sasha I guess this is your call.

Just tell me where to set it and I will make the patch. Basically with the user option it can always be changed on a run by run basis.

Ira

Quoted 89 lines

-- Hal
Quoted 85 lines
but as a
default compromise I think 4 is good.
Ira
Quoted 65 lines
Quoted 8 lines
Quoted 4 lines
Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.
I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.
So OpenSM is seeing dropped packets? With 4 SMP's on the wire? I do see some VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an issue. What kind of rate are you seeing?
The other question is; do people regularly run the tools which are using libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)? We do. If others are not then I would say this change would have less impact as they would want the diags to have some priority for debugging. The other option is to change the patch to be a default of 2 and allow user to change it depending on what they are trying to do. If you think that is best I will change the patch.
Ira
Quoted 41 lines
-- Hal
Quoted 37 lines
Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://***vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://***vger.kernel.org/majordomo-info.html
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
Quoted 6 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hal Rosenstock

2010-02-05 12:24 UTC #15 lore raw

In reply to a message not in the archive

Quoted 57 lines · Ira Weiny

Quoted 7 lines · On Thu, 4 Feb 2010 15:01:32 -0500 Hal Rosenstock…
Quoted 6 lines · Ira Weiny
Quoted 3 lines · On Thu, 4 Feb 2010 09:19:39 -0500 Hal Rosenstock…
Quoted 2 lines · Ira Weiny
Sasha,
[snip]
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
real 0m2.249s
user 0m1.244s
sys 0m0.936s
14:40:59 > time ./ibnetdiscover -o 4 --node-name-map /etc/opensm/ib-node-name-map -g > new
real 0m2.170s
user 0m1.160s
sys 0m0.933s
14:41:10 > /usr/sbin/ibqueryerrors -s RcvErrors,SymbolErrors,RcvSwRelayErrors,XmtWait -r --data
Suppressing: RcvErrors SymbolErrors RcvSwRelayErrors XmtWait
Errors for 0x66a00d90006fb "SW19"
GUID 0x66a00d90006fb port 9: [VL15Dropped == 3] [XmtData == 25187379] [RcvData == 25196688] [XmtPkts == 349861] [RcvPkts == 349954] Link info: 139 9[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 0x0002c9030001d736 864 1[ ] "hyperion1" ( )
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Note that there were no additional VL15Dropped packets on the fabric. I think 4 seems to be a good compromise. I have not tested when there are errors on the fabric. (Right now things seem to be good!)
Is this just with the SM doing light sweeping ?
Yes.
That's not a lot of SMP stress from the SM side. SMP consumers are SM, diags, and the unsolicited traps.
Agreed. I hope to test this more next week.
Quoted 17 lines
Quoted 16 lines
Quoted 4 lines
Is there a speedup with 4 rather than 2 ?
There is a bit of a speed up (~0.5 to 1.0 sec). But my main reason to want to go to 4 is that if there are issues on the fabric, unresponsive nodes etc.; 4 will give us better parallelism to get around these issues. I have not had the chance to test this condition with the new algorithm but the original ibnetdiscover would slow way down when there are nodes which have unresponsive SMA's. If there are only 2 outstanding this will not give us much speed up. This was the main motivation I had for improving the library in this way.
Also, I think you are correct that we should increase OpenSM's default from 4 to 8. For the same reason as above. Some of our clusters have worked better with 8 when we are having issues. But right now we are still running with 4.

Quoted 5 lines

Quoted 3 lines
I'm concerned about just increasing ibnetdiscover to 4 rather than 2. I've seen a number of clusters with SMP dropping with the current lower defaults.
So OpenSM is seeing dropped packets?

OpenSM is seeing timeouts and there are VL15 drops in the subnet.

Quoted 1 line

With 4 SMP's on the wire?

Yes.

Quoted 6 lines

I do see some
VL15Dropped errors (maybe 2-3 a day) but I did not think that would be an issue. What kind of rate are you seeing?
The other question is; do people regularly run the tools which are using libibnetdisc (ibqueryerrors, iblinkinfo, ibnetdiscover)?

These tools are being used (at least ibnetdiscover and ibqueryerrors).

Quoted 5 lines

We do. If others
are not then I would say this change would have less impact as they would want the diags to have some priority for debugging. The other option is to change the patch to be a default of 2 and allow user to change it depending on what they are trying to do. If you think that is best I will change the patch.

FWIW I think 2 is better until we have more exhaustive experience with
4. The other alternative would be to make it 4 and then see if people start noticing (more) VL15 drops and possibly other issues.

-- Hal

Quoted 45 lines

Ira
Quoted 41 lines
-- Hal
Quoted 37 lines
Ira
Quoted 21 lines
-- Hal
Quoted 7 lines
The first patch converts the algorithm and the second adds the ibnd_set_max_smps_on_wire call.
Let me know what you think. Because the algorithm changed so much testing this is a bit difficult because the order of the node discovery is different. However, I have done some extensive diffing of the output of ibnetdiscover and things look good.
Ira
Quoted 11 lines
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://**vger.kernel.org/majordomo-info.html
--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

Quoted 7 lines

--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html