Make multicast and path record queue flexible.
PATCH 1/1 Make multicast and path record queue flexible.
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
--- drivers/infiniband/ulp/ipoib/ipoib.h | 2 + drivers/infiniband/ulp/ipoib/ipoib_main.c | 91 +++++++++++++++++++++++- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 2 +- 3 files changed, 91 insertions(+), 4 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +2 -0
@@ -698,6 +698,8 @@ static inline void ipoib_unregister_debugfs(void) { }extern int ipoib_sendq_size;extern int ipoib_recvq_size; +extern unsigned int ipoib_prec_qlen; +extern unsigned int ipoib_mcast_qlen;extern struct ib_sa_client ipoib_sa_client;
drivers/infiniband/ulp/ipoib/ipoib_main.c +88 -3
127 lines not on this page · raw message
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +1 -1
@@ -693,7 +693,7 @@ void ipoib_mcast_send(struct net_device *dev, void *mgid, struct sk_buff *skb) }if (!mcast->ah) { -if (skb_queue_len (&mcast->pkt_queue) <IPOIB_MAX_MCAST_QUEUE ) +if (skb_queue_len (&mcast->pkt_queue) < ipoib_mcast_qlen)skb_queue_tail (&mcast->pkt_queue, skb);else { ++dev->stats.tx_dropped;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4CAB4D49.9090107-smomgflXvOZWk0Htik3J/[email protected]
Quoted 4 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
IMHO, I think it is better to send multicasts to the broadcast MLID than to queue them.. More like ethernet that way.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20101005162833.GC5967-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
IMHO, I think it is better to send multicasts to the broadcast MLID than to queue them.. More like ethernet that way.
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010051410370.7065-sBS69tsa9Uj/[email protected]
Quoted 8 lines · Christoph Lameter
Quoted 6 lines · Jason Gunthorpe
Quoted 4 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
Quoted 7 lines
Quoted 2 lines
IMHO, I think it is better to send multicasts to the broadcast MLID than to queue them.. More like ethernet that way.
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
Sure, but Aleksey's patch is aimed at the case when the SM has not yet replied, not for your problem with IGMPv2. If their is no MLID then sending to the broadcast MLID is a better choice than hanging onto the packets. I wonder if you could even send unicasts to the broadcast?
I still think the problem you have with IGMPv2 is best solved by leaning on the gateway vendors to support IGMPv3 - which *does* send all reports to 244.0.0.22
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20101005195428.GB24268-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
Sure, but Aleksey's patch is aimed at the case when the SM has not yet replied, not for your problem with IGMPv2. If their is no MLID then sending to the broadcast MLID is a better choice than hanging onto the packets. I wonder if you could even send unicasts to the broadcast?
The problem that the SM has not yet replied is no different between the IGMP versions. If you get a confirmation but the MC group is not functional then packets go nowhere.
Quoted 3 lines
I still think the problem you have with IGMPv2 is best solved by leaning on the gateway vendors to support IGMPv3 - which *does* send all reports to 244.0.0.22
s/22/2
Certainly a solution for the igmp messages themselves but not for initial traffic or traffic send via sendonly join.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010051500330.8786-sBS69tsa9Uj/[email protected]
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Jason Gunthorpe
Quoted 4 lines
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
Quoted 8 lines
Quoted 4 lines
Sure, but Aleksey's patch is aimed at the case when the SM has not yet replied, not for your problem with IGMPv2. If their is no MLID then sending to the broadcast MLID is a better choice than hanging onto the packets. I wonder if you could even send unicasts to the broadcast?
The problem that the SM has not yet replied is no different between the IGMP versions. If you get a confirmation but the MC group is not functional then packets go nowhere.
Getting a MLID that is not 'functional' is a different problem. Aleksey is looking at the case when there is no MLID at all, and I think queuing is the wrong approach.
Quoted 5 lines
Quoted 3 lines
I still think the problem you have with IGMPv2 is best solved by leaning on the gateway vendors to support IGMPv3 - which *does* send all reports to 244.0.0.22
s/22/2
No, 22. RFC 3376:
4.2.14. IP Destination Addresses for Reports
Version 3 Reports are sent with an IP destination address of 224.0.0.22, to which all IGMPv3-capable multicast routers listen. A system that is operating in version 1 or version 2 compatibility modes sends version 1 or version 2 Reports to the multicast group specified in the Group Address field of the Report. In addition, a system MUST accept and process any version 1 or version 2 Report whose IP Destination Address field contains *any* of the addresses (unicast or multicast) assigned to the interface on which the Report arrives.
Quoted 2 lines
Certainly a solution for the igmp messages themselves but not for initial traffic or traffic send via sendonly join.
Using .22 will generally solve the problems with sychronizing the IPoIB gateway to the state of the IGMPv3 clients. Yes, there will still be some unknown lag in building the IB side of the network and for the router(s) to get ready to handle the group - but at least it is no longer dependent on any timeouts.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20101005202121.GC24268-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 10 lines · Jason Gunthorpe
Quoted 5 lines
Quoted 3 lines
I still think the problem you have with IGMPv2 is best solved by leaning on the gateway vendors to support IGMPv3 - which *does* send all reports to 244.0.0.22
s/22/2
No, 22. RFC 3376:
4.2.14. IP Destination Addresses for Reports
Quoted 9 lines
Version 3 Reports are sent with an IP destination address of 224.0.0.22, to which all IGMPv3-capable multicast routers listen. A system that is operating in version 1 or version 2 compatibility modes sends version 1 or version 2 Reports to the multicast group specified in the Group Address field of the Report. In addition, a system MUST accept and process any version 1 or version 2 Report whose IP Destination Address field contains *any* of the addresses (unicast or multicast) assigned to the interface on which the Report arrives.
Argh. Another MC group. And the ib layer does need to do IB level joins for those. So the initial messages will be lost for the first join(s)?
Quoted 8 lines
Quoted 2 lines
Certainly a solution for the igmp messages themselves but not for initial traffic or traffic send via sendonly join.
Using .22 will generally solve the problems with sychronizing the IPoIB gateway to the state of the IGMPv3 clients. Yes, there will still be some unknown lag in building the IB side of the network and for the router(s) to get ready to handle the group - but at least it is no longer dependent on any timeouts.
How do you propose to handle the IB level join to 224.0.0.22 to avoid packet loss there? IGMP messages will still get lost because of that.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010051535480.10603-sBS69tsa9Uj/[email protected]
Quoted 8 lines · Christoph Lameter
Quoted 5 lines
Using .22 will generally solve the problems with sychronizing the IPoIB gateway to the state of the IGMPv3 clients. Yes, there will still be some unknown lag in building the IB side of the network and for the router(s) to get ready to handle the group - but at least it is no longer dependent on any timeouts.
How do you propose to handle the IB level join to 224.0.0.22 to avoid packet loss there? IGMP messages will still get lost because of that.
First, the routers all join the group at startup and stay joined forever. This avoids the race in the route joining a new MGID after the client creates it, but before the IGMPv2 report is sent. I expect this is a major source of delay and uncertainty
Second, since all clients join this group as send-only it becomes possible for the SM to do reasonable things - for instance the MLID can be pre-provisioned as send-only from any end-port and thus after the SM replies with a MLID the MLID is guaranteed good for send-only use immediately.
Third, once the client etners IGMPv3 mode and joins the group (maybe at system boot?) it stays joined forever.
Finally, by sending multicast packets to the broadcast during the time the MLID is unknown we can pretty much guarantee that the first IGMPv3 packet that is sent to .22 will reach all routers in a timely fashion. (Hence my objection to Aleksey's approach)
Basically, this completely solves the IGMP client to IPoIB router communication problem. Yes, there will still be an unknown time until the IB network, router, and whatever is beyond the router is ready to actually process packets on a new group - BUT that is normal for IP multicast! The main point is that without lost IGMP packets things can proceed without relying on timeouts.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20101005205545.GF24268-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 7 lines · Jason Gunthorpe
Quoted 2 lines
How do you propose to handle the IB level join to 224.0.0.22 to avoid packet loss there? IGMP messages will still get lost because of that.
First, the routers all join the group at startup and stay joined forever. This avoids the race in the route joining a new MGID after the client creates it, but before the IGMPv2 report is sent. I expect this is a major source of delay and uncertainty
I think the current routers join 224.0.0.2 already. Adding another MC group should come with IGMPv3 support.
Quoted 5 lines
Second, since all clients join this group as send-only it becomes possible for the SM to do reasonable things - for instance the MLID can be pre-provisioned as send-only from any end-port and thus after the SM replies with a MLID the MLID is guaranteed good for send-only use immediately.
The problem is that the client join on 224.0.0.22 will be delayed due to fabric reconfig. The group is joined on demand. It is not automatically joined.
Quoted 2 lines
Third, once the client etners IGMPv3 mode and joins the group (maybe at system boot?) it stays joined forever.
IGMP does not explicitly join 224.0.0.X groups. Looks like messages to 224.0.0.X will not be send unless there is no other responder on the subnet. So the initial messages for the first join getting lost may still be a problem.
Quoted 4 lines
Finally, by sending multicast packets to the broadcast during the time the MLID is unknown we can pretty much guarantee that the first IGMPv3 packet that is sent to .22 will reach all routers in a timely fashion. (Hence my objection to Aleksey's approach)
Right. So the multicast traffic will flow to the broadcast address until the SM sends the response. The multicast traffic will then get lost until the fabric reconfig is complete.
Quoted 6 lines
Basically, this completely solves the IGMP client to IPoIB router communication problem. Yes, there will still be an unknown time until the IB network, router, and whatever is beyond the router is ready to actually process packets on a new group - BUT that is normal for IP multicast! The main point is that without lost IGMP packets things can proceed without relying on timeouts.
Sure this sounds to be a much better approach (we have thought through such approaches here repeatedly) but I do not know of any IB gateway that supports IGMPv3.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010051600190.10603-sBS69tsa9Uj/[email protected]
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Jason Gunthorpe
Quoted 2 lines
How do you propose to handle the IB level join to 224.0.0.22 to avoid packet loss there? IGMP messages will still get lost because of that.
Quoted 7 lines
Quoted 4 lines
First, the routers all join the group at startup and stay joined forever. This avoids the race in the route joining a new MGID after the client creates it, but before the IGMPv2 report is sent. I expect this is a major source of delay and uncertainty
I think the current routers join 224.0.0.2 already. Adding another MC group should come with IGMPv3 support.
Sure, .22 is definately something routers need to have with IGMPv3.
Quoted 9 lines
Quoted 5 lines
Second, since all clients join this group as send-only it becomes possible for the SM to do reasonable things - for instance the MLID can be pre-provisioned as send-only from any end-port and thus after the SM replies with a MLID the MLID is guaranteed good for send-only use immediately.
The problem is that the client join on 224.0.0.22 will be delayed due to fabric reconfig. The group is joined on demand. It is not automatically joined.
I was trying to explain that it is possible for the SM to provide a MLID that is fully functional for .22 - there is no behind the scenes network reconfiguring delay. This is doable with IGMPv3 because the client join is send-only and all the listeners have been joined for a long time.
Basically, the SM pushes out an all end-ports send-only configuration for the MLID when the listeners join. So there *is no reconfiguration* for a new send-only join to complete. No reconfiguration means no lost packets.
Not sure if any SMs work this way already but they already have special support for things like the IPv4 broadcast so it is completely reasonable to have special support for IGMPv3 all routers as well. A 'fast send-only join' configurable for MGIDs would do the job.
There is virtually no cost with preconfiguring switches for send only traffic.
Quoted 8 lines
Quoted 4 lines
Finally, by sending multicast packets to the broadcast during the time the MLID is unknown we can pretty much guarantee that the first IGMPv3 packet that is sent to .22 will reach all routers in a timely fashion. (Hence my objection to Aleksey's approach)
Right. So the multicast traffic will flow to the broadcast address until the SM sends the response. The multicast traffic will then get lost until the fabric reconfig is complete.
See above, that is avoidable with some SM help too.
Quoted 3 lines
Sure this sounds to be a much better approach (we have thought through such approaches here repeatedly) but I do not know of any IB gateway that supports IGMPv3.
Lean on the vendors :( Seems crazy to not implement v3 when v2 is so unworkable on IB.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20101005212826.GG24268-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines
The problem is that the client join on 224.0.0.22 will be delayed due to fabric reconfig. The group is joined on demand. It is not automatically joined.
I was trying to explain that it is possible for the SM to provide a MLID that is fully functional for .22 - there is no behind the scenes network reconfiguring delay. This is doable with IGMPv3 because the client join is send-only and all the listeners have been joined for a long time.
Ahh.. Good idea.
Quoted 2 lines
There is virtually no cost with preconfiguring switches for send only traffic.
True.
Quoted 6 lines
Quoted 3 lines
Sure this sounds to be a much better approach (we have thought through such approaches here repeatedly) but I do not know of any IB gateway that supports IGMPv3.
Lean on the vendors :( Seems crazy to not implement v3 when v2 is so unworkable on IB.
Oh we will.... Do obsidian routers support IGMPv3 for IB?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Christoph Lameter
Quoted 6 lines · Jason Gunthorpe
Quoted 4 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
Quoted 9 lines
Quoted 2 lines
IMHO, I think it is better to send multicasts to the broadcast MLID than to queue them.. More like ethernet that way.
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
But what if it will not be available from some reason? How long should we wait? Do we need implement another queue/counter/timeout?
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
A message that is not in the archive (sent off-list or not yet received): 1286900993.31931.4.camel-uOVkuFIEnOODI2cvxHXf6UEOCMrvLtNR@public.gmane.org
Quoted 13 lines · Alekseys Senin
On Tue, 2010-10-05 at 14:12 -0500, Christoph Lameter wrote:
On Tue, 5 Oct 2010, Jason Gunthorpe wrote:
> On Tue, Oct 05, 2010 at 06:07:37PM +0200, Aleksey Senin wrote:
> > When using slow SM allow more packets to be buffered before answer > > comming back. This patch based on idea of Christoph Lameter. > >
> > http://lists.openfabrics.org/pipermail/general/2009-June/059853.html
>
> IMHO, I think it is better to send multicasts to the broadcast MLID than to > queue them.. More like ethernet that way.
Quoted 7 lines
I agree. We had similar ideas. However, the kernel does send igmp reports to the MC address not to 244.0.0.2. We would have to redirect at the IB layer until multicast via MLID becomes functional. We cannot tell when that will be the case.
But what if it will not be available from some reason? How long should we wait? Do we need implement another queue/counter/timeout?
If you follow the scheme I outlined - where traffic to a MGID that doesn't yet have a MLID is routed to the broadcast MLID then you do it until you get a MLID, with periodic retries/refreshes of the SA operation.
This is similar to how ethernet works, and is generally harmless. Better to have a working, but suboptimal network, than one that is busted.
Jason
Christoph Lameter
Reviewed-byQuoted 2 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
I agree, I think we need to have those things configurable.
How do you handle the situation of the SM responding before the fabric has been reconfigured? I do not see any delay on join. So they will be dropped if the fabric was not reconfigured fast enough? Or does the SM somehow delay the response?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010051415100.7065-sBS69tsa9Uj/[email protected]
Quoted 5 lines · Christoph Lameter
Quoted 2 lines · Aleksey Senin
When using slow SM allow more packets to be buffered before answer comming back. This patch based on idea of Christoph Lameter.
Quoted 8 lines
I agree, I think we need to have those things configurable.
Reviewed-by Christoph Lameter <[email protected]>How do you handle the situation of the SM responding before the fabric has been reconfigured? I do not see any delay on join. So they will be dropped if the fabric was not reconfigured fast enough? Or does the SM somehow delay the response?
I think this issue, that should solve the problem of sending or delaying packets after obtaining MLID should be solved in another patch. Proposed code improve today situation, when you can't change it at all. Relating to broadcast, I don't think that this is a good solution it will bring unwarranted load, specially in the case if no MLID received. The way of adding delay before we start to send packages, seems to me better.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1286380676.31487.1.camel-uOVkuFIEnOODI2cvxHXf6UEOCMrvLtNR@public.gmane.org
Quoted 8 lines · Alekseys Senin
Quoted 4 lines
How do you handle the situation of the SM responding before the fabric has been reconfigured? I do not see any delay on join. So they will be dropped if the fabric was not reconfigured fast enough? Or does the SM somehow delay the response?
I think this issue, that should solve the problem of sending or delaying packets after obtaining MLID should be solved in another patch. Proposed code improve today situation, when you can't change it at all.
I agree with that.
Quoted 4 lines
Relating to broadcast, I don't think that this is a good solution it will bring unwarranted load, specially in the case if no MLID received. The way of adding delay before we start to send packages, seems to me better.
Broadcast is a temporary solution and it only occurs with in an IB partition. It would be a problem if one host suddenly starts sending at full line speed. Maybe have that configurable?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.00.1010061114590.31538-sBS69tsa9Uj/[email protected]
Quoted 8 lines · Christoph Lameter
Quoted 4 lines
Relating to broadcast, I don't think that this is a good solution it will bring unwarranted load, specially in the case if no MLID received. The way of adding delay before we start to send packages, seems to me better.
Broadcast is a temporary solution and it only occurs with in an IB partition. It would be a problem if one host suddenly starts sending at full line speed. Maybe have that configurable?
Right, and keep in mind that sending a packet with the broadcast MLID and group-specific MGID will still result in the HCA's filtering based on MGID. Personally, I'm not concerned about flooding - this is how things work in ethernet - any sane app will need some way to cope with that.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html