RDMA/CMA: fix iWARP adapter TCP port space usage

This patch resolves the issue by allocating a kernel socket consuming the TCP port for the iWARP connection upon connection creation and freeing the socket once the iWARP connection is destroyed.

Quoted 3 lines

This patch resolves the issue by allocating a kernel socket consuming the TCP port for the iWARP connection upon connection creation and freeing the socket once the iWARP connection is destroyed.

Hasn't this exact approach been explicitly NAK'ed by David Miller?

Quoted 6 lines

Quoted 3 lines

This patch resolves the issue by allocating a kernel socket consuming the TCP port for the iWARP connection upon connection creation and freeing the socket once the iWARP connection is destroyed.

Hasn't this exact approach been explicitly NAK'ed by David Miller?

Yes but this approach still is the simplest and cleanest way to solve this problem that I am aware of.

I do want to make clear on the network resource usage. For example, if you run an MPI job over host TCP stack and the job creates 4 connections thus consuming 4 sockets. With this patch, the same job over iWARP will consume 4 sockets. This patch has zero impact on non-iWARP applications and only consumes equal amount of network resources.

I would like this patch to be reconsidered for inclusion into the kernel.

Chien

You can peruse this thread from 2007:

http://lkml.org/lkml/2007/8/15/174

In that thread David Miller said he would NAK this solution and even NAK a solution where we expose the low level port allocation services and allow RDMA/iWARP and TCP to share the port space. Thus I see no way to resolve this issue with Dave's ACK, and maybe the RDMA contributors and maintainers should move forward without it?

If Dave Miller says this is an unacceptable thing to merge, then I'm not going to merge it over his explicit and unequivocal NAK. Trying sneak this in by hiding this patch on linux-rdma without cc-ing netdev is the wrong approach... the right approach is to get the hordes of users being hurt by this issue (if there are such hordes), make a big stink, and force a solution that everyone agrees on.

What exactly is the end-user harm anyhow? Everything works 100% correctly without this patch, and your arguments all revolved around diagnostic capability..

It is a known weakness that the RDMA stack has crappy visibility. That has been talked about before. If you want better visibility then fix that by adding netlink stuff to dump out all the QP states, PDs, etc, similar to what TCP does.

Quoted 3 lines

What exactly is the end-user harm anyhow? Everything works 100% correctly without this patch, and your arguments all revolved around diagnostic capability..

OFED includes this patch, which was picked up by the distros. I'm not sure iWarp customers are running without it.

Quoted 4 lines

It is a known weakness that the RDMA stack has crappy visibility. That has been talked about before. If you want better visibility then fix that by adding netlink stuff to dump out all the QP states, PDs, etc, similar to what TCP does.

This is a separate topic from what the patch is trying to address.

Quoted 7 lines · Roland Dreier

If Dave Miller says this is an unacceptable thing to merge, then I'm not going to merge it over his explicit and unequivocal NAK. Trying sneak this in by hiding this patch on linux-rdma without cc-ing netdev is the wrong approach... the right approach is to get the hordes of users being hurt by this issue (if there are such hordes), make a big stink, and force a solution that everyone agrees on.

Right..

What exactly is the end-user harm anyhow? Everything works 100% correctly without this patch, and your arguments all revolved around diagnostic capability..

It is a known weakness that the RDMA stack has crappy visibility. That has been talked about before. If you want better visibility then fix that by adding netlink stuff to dump out all the QP states, PDs, etc, similar to what TCP does.

Jason

No, everything does not work without this patch. Go read the thread
from 2007 (I posted the URL to it on this thread). I describe exactly
how an application like MPI will get incorrect behavior due to port space collisions on iWARP HW.

Quoted 2 lines · Steve Wise

No, everything does not work without this patch. Go read the thread from 2007 (I posted the URL to it on this thread). I describe exactly how an application like MPI will get incorrect behavior due to port space collisions on iWARP HW.

I guess I missed the fact that OFED included this patch, rather than relying on the other administratively configured schemes you originally outlined. I was opining that nobody was complaining about the administrative stuff.. but of course nobody complains, they don't have to do it :(

A seperate IP for iWarp always seemed to me to be the way forward on this. And I do mean seperate, as in TCP packets for this IP are never sent into the net stack seperate :|

Further, its not even needed for IB, just iWARP. That's an unnecessary admin pain IMO.

iWARP should and can easily co-exist with the host TCP by sharing the port space. But, as Roland stated already, maybe the only way forward it to get end-user pressure applied at the appropriate places! :)

Quoted 3 lines · Steve Wise

Further, its not even needed for IB, just iWARP. That's an unnecessary admin pain IMO.

IB already completely seperates from the host stack, that is why it isn't effected by this problem. It has both a seperate port numbering space (in rdma cm) and a separate addressing space (GID).

The entire problem with iWARP is that it is trying to not be seperate, unlike IB. So.. simple answer: use a seperate IP, or use a seperate port space (ie don't use the TCP protocol number).

ROCEE won't have this problem either..

Quoted 5 lines

iWARP should and can easily co-exist with the host TCP by sharing the port space. But, as Roland stated already, maybe the only way forward it to get end-user pressure applied at the appropriate places! :)

*shrug* This isn't going to happen until netdev decides to design-in statefull offload. I doubt that is going to happen any time soon. I've already seen Linux max out 40GE on benchmarks, so it is hard to see what the driver would be.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 12 lines · Jason Gunthorpe

Quoted 3 lines · Steve Wise

Further, its not even needed for IB, just iWARP. That's an unnecessary admin pain IMO.

IB already completely seperates from the host stack, that is why it isn't effected by this problem. It has both a seperate port numbering space (in rdma cm) and a separate addressing space (GID).

By virtue of being a different L4 transport. iWARP uses TCP as the L4
and thus has these issues.

Also, from an application perspective, IB has IP addresses that are shared with the TCP stack and the RDMA stack. So it still appears as integrated. At least with librdmacm applications...

Quoted 4 lines

The entire problem with iWARP is that it is trying to not be seperate, unlike IB. So.. simple answer: use a seperate IP, or use a seperate port space (ie don't use the TCP protocol number).

Change the iWARP specification? I don't think that's a simple answer. :)

Quoted 1 line

ROCEE won't have this problem either..

Quoted 10 lines

Quoted 5 lines

iWARP should and can easily co-exist with the host TCP by sharing the port space. But, as Roland stated already, maybe the only way forward it to get end-user pressure applied at the appropriate places! :)

*shrug* This isn't going to happen until netdev decides to design-in statefull offload. I doubt that is going to happen any time soon. I've already seen Linux max out 40GE on benchmarks, so it is hard to see what the driver would be.

Other protocols are also running over networking today, such as iSCSI and FCoE. These happily co-exist with other L2->L4 protocols in the stack. This iWARP patch allows iWARP to happily co-exist on a TCP connection, and does *not* negatively affect the networking stack at all.

Quoted 5 lines

Other protocols are also running over networking today, such as iSCSI and FCoE. These happily co-exist with other L2->L4 protocols in the stack. This iWARP patch allows iWARP to happily co-exist on a TCP connection, and does *not* negatively affect the networking stack at all.

How do iSCSI offload HBAs coexist? As I understand it, they typically just choose a separate IP address.

In any case I'm not going to slip in a patch that another maintainer has explicitly NAKed. Maybe one way to force things forward would be to write up an exhaustive explanation of the underlying problem and the impact on end users, include this patch, explain that it touches only RDMA code, and point out that most end users are already using this patch since it's shipped in OFED. Then send the whole thing to Linus and Andrew Morton, making sure to cc Dave Miller, netdev, and linux-rdma.

Quoted 5 lines

Other protocols are also running over networking today, such as iSCSI and FCoE. These happily co-exist with other L2->L4 protocols in the stack. This iWARP patch allows iWARP to happily co-exist on a TCP connection, and does *not* negatively affect the networking stack at all.

How do iSCSI offload HBAs coexist? As I understand it, they typically just choose a separate IP address.

In any case I'm not going to slip in a patch that another maintainer has explicitly NAKed. Maybe one way to force things forward would be to write up an exhaustive explanation of the underlying problem and the impact on end users, include this patch, explain that it touches only RDMA code, and point out that most end users are already using this patch since it's shipped in OFED. Then send the whole thing to Linus and Andrew Morton, making sure to cc Dave Miller, netdev, and linux-rdma.

- R.

Quoted 1 line · Roland Dreier

Quoted 1 line

Other protocols are also running over networking today, such as

Quoted 9 lines

Quoted 4 lines

and FCoE. These happily co-exist with other L2->L4 protocols in the stack. This iWARP patch allows iWARP to happily co-exist on a TCP connection, and does *not* negatively affect the networking stack at all.

How do iSCSI offload HBAs coexist? As I understand it, they typically just choose a separate IP address.

In any case I'm not going to slip in a patch that another maintainer

Quoted 10 lines

explicitly NAKed. Maybe one way to force things forward would be to write up an exhaustive explanation of the underlying problem and the impact on end users, include this patch, explain that it touches only RDMA code, and point out that most end users are already using this patch since it's shipped in OFED. Then send the whole thing to Linus and Andrew Morton, making sure to cc Dave Miller, netdev, and linux-rdma.

- R.

My 2007 thread does this basically, but posted it to lkml and David Miller. But the rationale for why we need it as well as other possible solutions is included in that thread. We could re-package it and send it on as you suggest. It might carry more weight coming from the linux rdma maintainer though. :)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I agree that the issue must get solved and its good that it has been brought up again. I agree with Chien that the solution should respect and interface to a single in kernel instance maintaining host global TCP port space. iWARP is just another protocol on top of TCP - like iSCSI. There is no good reason to invent another TCP port maintainer per TCP user type trying to synchonize with the kernel if the resource is host global and already maintained by the kernel.

Since we are developing and already open sourced a full software implementation (SoftiWARP) of RDMA, our view on the optimal solution must be different. Like kernel iSCSI, we are running on top of regular kernel sockets. With that, there is no point having a connection manager blocking just the port we wanted to use for communication - SoftiWARP uses kernel sockets for data communication.

Therefore, I propose pushing back responsibility to the RDMA device driver, where the actual connection setup is initiated (RNIC) or takes place (software RMDA stack). I think, it is not the job of the RDMA connection manager to maintain TCP port space at all. It should be up to the driver

to do the appropriate steps. Due to the lack of another interface, an RNIC driver would create and bind a kernel socket to get hold of the TCP port it is intending to use for offloaded communication, while a software RDMA stack just goes forward doing communication on that socket. For the future it might be a good idea to approach the netdev folks kindly asking for a neat interface for just TCP port maintainance without the need to create and bind an otherwise useless socket.

Of course, the RNIC driver must restrict its activities to local IP adresses on its cards (or, for SoftiWARP, to IP adresses of interfaces it is bound to). For example, a wildcard listen must get translated into a listen restricted to the interface(s) under local control.

Quoted 9 lines · Bernard Metzler

I agree that the issue must get solved and its good that it has been brought up again. I agree with Chien that the solution should respect and interface to a single in kernel instance maintaining host global TCP port space. iWARP is just another protocol on top of TCP - like iSCSI. There is no good reason to invent another TCP port maintainer per TCP user type trying to synchonize with the kernel if the resource is host global and already maintained by the kernel.

Quoted 4 lines

Since we are developing and already open sourced a full software implementation (SoftiWARP) of RDMA, our view on the optimal solution must be different. Like kernel iSCSI, we are running on top of regular kernel sockets. With that, there is no point having a connection

Quoted 4 lines

blocking just the port we wanted to use for communication - SoftiWARP uses kernel sockets for data communication.

Hey Bernard,

Has SoftiWARP been submitted upstream yet?

Quoted 3 lines

Therefore, I propose pushing back responsibility to the RDMA device

Quoted 2 lines

where the actual connection setup is initiated (RNIC) or takes place (software RMDA stack). I think, it is not the job of the RDMA

Quoted 1 line

manager to maintain TCP port space at all. It should be up to the

Quoted 9 lines

to do the appropriate steps. Due to the lack of another interface, an RNIC driver would create and bind a kernel socket to get hold of the TCP port it is intending to use for offloaded communication, while a software RDMA stack just goes forward doing communication on that socket. For the future it might be a good idea to approach the netdev folks kindly asking for a neat interface for just TCP port maintainance without the need to create and bind an otherwise useless socket.

I proposed this design in 2007. It was NAK'd. Read the tail end of this email where I describe such a solution and indicate that Miller already NAK'd it. Now we could try again with this solution, but unless we have end users backing us and showing how much demand there is for this, it won't fly IMO.

http://lkml.org/lkml/2007/8/15/174

Quoted 2 lines

Of course, the RNIC driver must restrict its activities to local IP adresses on its cards (or, for SoftiWARP, to IP adresses of

Quoted 4 lines

it is bound to). For example, a wildcard listen must get translated into a listen restricted to the interface(s) under local control.

I implemented and submitted this type of solution for cxgb3 in 2007 as well.

http://lkml.org/lkml/2007/9/13/268

Roland didn't like it, I think, because it used well known tokens in the interface name to designate iwarp ip addresses via ifconfig. Like "eth0:iw1". So the solution really required the admin to setup these iwarp-only subnets/interfaces. There was nothing that prevented non iwarp traffic to arrive on these ip addresses other than admin policy. I think that was another reason Roland didn't like this solution. Anyway, you can peruse that thread and maybe its a starting point for some "separate iwarp ipaddresses" solution....

iWARP is just another protocol on top of TCP - like iSCSI. There is no good reason to invent another TCP port maintainer per TCP user type trying to synchonize with the kernel if the resource is host global and already maintained by the kernel.

Since we are developing and already open sourced a full software implementation (SoftiWARP) of RDMA, our view on the optimal solution must be different. Like kernel iSCSI, we are running on top of regular kernel sockets. With that, there is no point having a connection manager blocking just the port we wanted to use for communication - SoftiWARP uses kernel sockets for data communication.

Quoted 6 lines

Since we are developing and already open sourced a full software implementation (SoftiWARP) of RDMA, our view on the optimal solution must be different. Like kernel iSCSI, we are running on top of regular kernel sockets. With that, there is no point having a connection manager blocking just the port we wanted to use for communication - SoftiWARP uses kernel sockets for data communication.

I think this is an extremely strong argument against the patch that started the thread. Breaking soft iWARP seems a fatal flaw.

- R.

Quoted 5 lines

I think this is an extremely strong argument against the patch that started the thread. Breaking soft iWARP seems a fatal flaw.

- R.

I agree.

Quoted 8 lines

Quoted 6 lines

I think this is an extremely strong argument against the patch that started the thread. Breaking soft iWARP seems a fatal flaw.

- R.

I agree.

The patch or SoftiWARP can be reworked to allow the whole iWARP family to coexist. It is a matter of agreeing on which path to take.

Quoted 4 lines

iWARP is just another protocol on top of TCP - like iSCSI. There is no good reason to invent another TCP port maintainer per TCP user type trying to synchonize with the kernel if the resource is host global and already maintained by the kernel.

I think the counter-argument to this is than an iWARP offload NIC is an independent TCP stack and hence should not be tied into the host stack. It's interesting that you bring up iSCSI -- as I understand things, iSCSI offload HBAs are typically configured with their own IP, through a separate mechanism. (The port collision problem is not likely to be hit with iSCSI, since the HBA is an initiator and hence does only active connections, and a 4-tuple collision between connections to the iSCSI target is not likely and other host stack traffic is extremely unlikely)

Roland, do you think the iSCSI approach is a "good design" for iWARP devices?

Quoted 2 lines

Roland, do you think the iSCSI approach is a "good design" for iWARP devices?

Well, it's a different problem since as I said the port collision problem is a non-issue for iSCSI anyway. But yes having a separate interface to assign an iWARP IP address to an RNIC does seem to avoid the immediate problem.

I actually don't know what the right answer is -- having a separate IP address for iWARP does seem to lead to having to duplicate everything for configuring it. (And this is the approach for the cxgb[34] iSCSI drivers, right?)

On the other hand trying to hook offloaded iWARP into the normal stack does seem to lead to a mess. I see DaveM's point: TCP port space is just the beginning -- filtering, queueing, etc also have config that ultimately an offload device would want to hook too.

Maybe the sanest out of a bad set of options would be to come up with a standard way to configure independent TCP/IP stacks that share a link.

really, dunno.

- R.

I just think the customer looses when we add iwarp-specific tools, ipaddrs, subnets, etc etc. And what about software iwarp? Will it use the host stack tools and not these new tools? So then we end up with 2 sets of tools for iwarp devices. :(

Yes. Perusing the drivers/scsi/cxgb3i code I see the iscsi ipaddr is actually stored in the port_info struct which is hung of the netdev_priv of the cxgb3 device. It is set by cxgb3i_host_set_param() which is part of the iscsi transport interface.

I just think the customer looses when we add iwarp-specific tools, ipaddrs, subnets, etc etc. And what about software iwarp? Will it use the host stack tools and not these new tools? So then we end up with 2 sets of tools for iwarp devices. :(

Quoted 2 lines · Steve Wise

Yes. Perusing the drivers/scsi/cxgb3i code I see the iscsi ipaddr is actually stored in the port_info struct which is hung of the netdev_priv of the cxgb3 device. It is set by cxgb3i_host_set_param() which is part of the iscsi transport interface.

I wonder how does neighbor discovery, routing, etc work with iscsi?

Quoted 2 lines

I just think the customer looses when we add iwarp-specific tools, ipaddrs, subnets, etc etc. And what about software iwarp? Will it use the host stack tools and not these new tools? So then we end up with 2 sets of tools for iwarp devices. :(

Well, maybe you can get netdev to agree on some way to create an interface that has all the IP services, but no TCP protocol binding? Then the configuration could be largely the same. If you could share that with the iscsi world then maybe it isn't so bad?

Quoted 4 lines

I wonder how does neighbor discovery, routing, etc work with iscsi?

For cxgb3i:

ND is handled by initiating ND via exported kernel services (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net events to get updated neigh entries.

The host routing table is consulted via ip_route_output_flow() to map a destination ip address to a local netdev, and then if that device is T3, it will do the iscsi offload.

Quoted 9 lines

Quoted 4 lines

I wonder how does neighbor discovery, routing, etc work with iscsi?

For cxgb3i:

ND is handled by initiating ND via exported kernel services (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net events to get updated neigh entries.

Quoted 3 lines

The host routing table is consulted via ip_route_output_flow() to map a destination ip address to a local netdev, and then if that device is T3, it will do the iscsi offload.

By the way, this is how iWARP works too. The ND stuff is done by the IWCM during RESOLVE_ADDR. The routing lookups are done by the iWARP devices themselves typically.

Quoted 1 line

I wonder how does neighbor discovery, routing, etc work with iscsi?

For cxgb3i:

ND is handled by initiating ND via exported kernel services (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net events to get updated neigh entries.

The host routing table is consulted via ip_route_output_flow() to map a destination ip address to a local netdev, and then if that device is T3, it will do the iscsi offload.

Quoted 4 lines

Well, maybe you can get netdev to agree on some way to create an interface that has all the IP services, but no TCP protocol binding? Then the configuration could be largely the same. If you could share that with the iscsi world then maybe it isn't so bad?

Maybe. I fear this will meet the same resistance from the netdev folks.

Doesn't that mean the port collision problem still exits [for iSCSI], although probably less likely?

Quoted 2 lines

Doesn't that mean the port collision problem still exits [for iSCSI], although probably less likely?

Yes, it's there, but almost impossible to hit: first of all, iSCSI HBAs never listen on a port, so that can never collide. Second, iSCSI HBAs only establish connections to iSCSI targets on the iSCSI port -- so really your only chance of a problem is if you ran an offloaded and non-offloaded iSCSI initiator on the same IP to the same target, _and_ you got unlucky on the local ports that you chose. So in practice no one will hit this.

Quoted 4 lines · Steve Wise

Quoted 2 lines

I wonder how does neighbor discovery, routing, etc work with iscsi?

For cxgb3i:

Quoted 4 lines

ND is handled by initiating ND via exported kernel services (neigh_event_send()) and registering for NETEVENT_NEIGH_UPDATE net events to get updated neigh entries.

The host routing table is consulted via ip_route_output_flow() to map a destination ip address to a local netdev, and then if that device is T3, it will do the iscsi offload.

That is what RDMA does.. So that means that the IP used for iscsi is actually an IP assigned to the interface?

The IP address assigned for the cxgb3i iscsi device is _not_ assigned to a netdev interface via ifconfig, as far as I understand it (by looking at the cxgb3i code). So the host stack doesn't know about this address. There is an administrative requirement, I assume, that the secret iscsi ipaddr is within a subnet that is bound to the T3 ethX interface. Otherwise the routing lookup wouldn't work.

Quoted 2 lines · Steve Wise

The IP address assigned for the cxgb3i iscsi device is _not_ assigned to a netdev interface via ifconfig, as far as I understand it (by looking at the cxgb3i code). So the host stack doesn't know about this address. There is an administrative requirement, I assume, that the secret iscsi ipaddr is within a subnet that is bound to the T3 ethX interface. Otherwise the routing lookup wouldn't work.

So who responds to neighbor queries, and how do outgoing queries get sent with the right IP? Sounds odd...

Quoted 2 lines

So who responds to neighbor queries, and how do outgoing queries get sent with the right IP? Sounds odd...

The iscsi hba is only an initiator, so it doesn't need to respond to arp queries.

Quoted 5 lines · Steve Wise

Quoted 3 lines

So who responds to neighbor queries, and how do outgoing queries get sent with the right IP? Sounds odd...

The iscsi hba is only an initiator, so it doesn't need to respond to arp queries.

Hmm.. The other side could arp you at any time, and if you don't answer stuff can go bad, so something must be generating the replies.

But I guess that is seperate, sounds like iSCSI is in a similar boat and they were not able to reconcile either? :(

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines · Jason Gunthorpe

Quoted 2 lines · Steve Wise

The IP address assigned for the cxgb3i iscsi device is _not_ assigned to a netdev interface via ifconfig, as far as I understand it (by looking at the cxgb3i code). So the host stack doesn't know about this address. There is an administrative requirement, I assume, that the secret iscsi ipaddr is within a subnet that is bound to the T3 ethX interface. Otherwise the routing lookup wouldn't work.

Quoted 4 lines

So who responds to neighbor queries, and how do outgoing queries get sent with the right IP? Sounds odd...

The iscsi hba is only an initiator, so it doesn't need to respond to arp queries. I guess the Source Protocol Address in the outgoing ARP request will be the ipaddr of the outgoing interface. Its ok though because what is needed is the next-hop peer's hwaddr. So the ARP reply comes in, updates the host neigh entry, and a NEIGH_EVENT callout is performed to the offload device drivers. It is a little hackish, but that's the only way the netdev maintainers would allow iscsi offload in. They originally tried to use the src address from the ethX interface for the offload iscsi connections and that was rejected.

On the other hand trying to hook offloaded iWARP into the normal stack does seem to lead to a mess. I see DaveM's point: TCP port space is just the beginning -- filtering, queueing, etc also have config that ultimately an offload device would want to hook too.

Quoted 4 lines

On the other hand trying to hook offloaded iWARP into the normal stack does seem to lead to a mess. I see DaveM's point: TCP port space is just the beginning -- filtering, queueing, etc also have config that ultimately an offload device would want to hook too.

TCP port space is just the beginning but then these features didn't show up all at once in the kernel either. Instead of evolving iWARP implementation, we can't even take a baby step and fix a flaw that exists in the current kernel. Why are we "replicating" everything offered by the host stack instead of hooking in? It does not sound like good engineering to me.

Quoted 11 lines

Quoted 4 lines

On the other hand trying to hook offloaded iWARP into the normal stack does seem to lead to a mess. I see DaveM's point: TCP port space is just the beginning -- filtering, queueing, etc also have config that ultimately an offload device would want to hook too.

TCP port space is just the beginning but then these features didn't show up all at once in the kernel either. Instead of evolving iWARP implementation, we can't even take a baby step and fix a flaw that exists in the current kernel. Why are we "replicating" everything offered by the host stack instead of hooking in? It does not sound like good engineering to me.

Well as I said I don't particularly see a clean solution. But the point I was making was that the net stack is already very complex with many places where interface configs are controlled -- having to add hooks to pass that config on to offload devices is going to add even more complexity and also add constraints to the format of that config information. Which is not good.

Quoted 3 lines

Quoted 3 lines

On the other hand trying to hook offloaded iWARP into the normal

Quoted 10 lines

Quoted 3 lines

does seem to lead to a mess. I see DaveM's point: TCP port space is just the beginning -- filtering, queueing, etc also have config that ultimately an offload device would want to hook too.

TCP port space is just the beginning but then these features didn't show up all at once in the kernel either. Instead of evolving iWARP implementation, we can't even take a baby step and fix a flaw that exists in the current kernel. Why are we "replicating" everything offered by the host stack instead of hooking in? It does not sound like good engineering to me.

Well as I said I don't particularly see a clean solution. But the point I was making was that the net stack is already very complex with many places where interface configs are controlled -- having to add hooks to pass that config on to offload devices is going to add even more complexity and also add constraints to the format of that config information. Which is not good.

To my understanding, our discussion touches two topics. One is to solve the TCP port space issue, the other is more general, its about proper integration of offloaded TCP within Linux. So, the second topic is a generalization of the first.

Regarding the first topic, what I was about to propose is that the iWARP kernel driver (software iWARP or RNIC) itself should take care of port space allocations. Port space maintenance functionality should be minimized at iWARP CM level. It looks straightforward to me if during the rdma_connect() call the driver picks a free port using a socket/bind sequence for its local interface. The same would be possible for
the passive connection setup, which always involves an rdma_bind_addr()
- we would have to pass the rdma_bind_addr() call down to the driver and EADDRINUSE would be a reasonable return value.
Here things are getting a little more complicated, if it comes to

INADDR_ANY and port 0 bindings. In private email, Bob Sharp already suggested it - the iWARP CM would have to pick a port and try it on all interfaces....maybe by starting with port 0 binding on one interface and trying to extend with the returned port on all remaining interfaces. That introduces an unbind() call if things fail, too. In any case, the rdma_bind_addr() call would create additional state
at driver level.

For softiwarp, during bind() or connect(), a TCP socket would be created and bound, for an RNIC driver (currently) the same would happen. While with softiwarp this socket would be used for communication later, the RNIC driver
would simply have to keep it around until the connection endpoint gets destroyed
or the port gets unbound.

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Tung, Chien Tin
Sent: Friday, June 25, 2010 3:15 PM
To: Bernard Metzler; Roland Dreier
Cc: Jason Gunthorpe; [email protected]; [email protected]; Waskiewicz Jr, Peter P; Steve Wise
Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

Quoted 16 lines

To my understanding, our discussion touches two topics. One is to solve the TCP port space issue, the other is more general, its about proper integration of offloaded TCP within Linux. So, the second topic is a generalization of the first.

Regarding the first topic, what I was about to propose is that the iWARP kernel driver (software iWARP or RNIC) itself should take care of port space allocations. Port space maintenance functionality should be minimized at iWARP CM level. It looks straightforward to me if during the rdma_connect() call the driver picks a free port using a socket/bind sequence for its local interface. The same would be possible for
the passive connection setup, which always involves an rdma_bind_addr()
- we would have to pass the rdma_bind_addr() call down to the driver and EADDRINUSE would be a reasonable return value.
Here things are getting a little more complicated, if it comes to

Quoted 8 lines

INADDR_ANY and port 0 bindings. In private email, Bob Sharp already suggested it - the iWARP CM would have to pick a port and try it on all interfaces....maybe by starting with port 0 binding on one interface and trying to extend with the returned port on all remaining interfaces. That introduces an unbind() call if things fail, too. In any case, the rdma_bind_addr() call would create additional state
at driver level.

I am okay with adding rdma_bind_addr and rdma_unbind_addr calls. I won't speak for Sean and the work that needs to go into the CM. But this will allow all known iWARP implementations to work together.

Quoted 7 lines

For softiwarp, during bind() or connect(), a TCP socket would be created and bound, for an RNIC driver (currently) the same would happen. While with softiwarp this socket would be used for communication later, the RNIC driver
would simply have to keep it around until the connection endpoint gets destroyed
or the port gets unbound.

We want to be careful and make sure there is only one iWARP provider per IP address. If softiWARP binds and surfaces another verbs interface on an existing one, this scheme will not work.

Chien

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Steve,

Do you see any issues with Bernard's proposal? Is this something we can agree on?

Chien

Quoted 52 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Tung, Chien Tin
Sent: Friday, June 25, 2010 3:15 PM
To: Bernard Metzler; Roland Dreier
Cc: Jason Gunthorpe; [email protected]; [email protected]; Waskiewicz Jr, Peter P; Steve Wise
Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space usage

Quoted 16 lines

To my understanding, our discussion touches two topics. One is to solve the TCP port space issue, the other is more general, its about proper integration of offloaded TCP within Linux. So, the second topic is a generalization of the first.

Regarding the first topic, what I was about to propose is that the iWARP kernel driver (software iWARP or RNIC) itself should take care of port space allocations. Port space maintenance functionality should be minimized at iWARP CM level. It looks straightforward to me if during the rdma_connect() call the driver picks a free port using a socket/bind sequence for its local interface. The same would be possible for
the passive connection setup, which always involves an rdma_bind_addr()
- we would have to pass the rdma_bind_addr() call down to the driver and EADDRINUSE would be a reasonable return value.
Here things are getting a little more complicated, if it comes to

Quoted 9 lines

INADDR_ANY and port 0 bindings. In private email, Bob Sharp already suggested it - the iWARP CM would have to pick a port and try it on all interfaces....maybe by starting with port 0 binding on one interface and trying to extend with the returned port on all remaining interfaces. That introduces an unbind() call if things fail, too. In any case, the rdma_bind_addr() call would create additional state
at driver level.

I am okay with adding rdma_bind_addr and rdma_unbind_addr calls. I won't speak for Sean and the work that needs to go into the CM. But this will allow all known iWARP implementations to work together.

Quoted 8 lines

For softiwarp, during bind() or connect(), a TCP socket would be created and bound, for an RNIC driver (currently) the same would happen. While with softiwarp this socket would be used for communication later, the RNIC driver
would simply have to keep it around until the connection endpoint gets destroyed
or the port gets unbound.

We want to be careful and make sure there is only one iWARP provider per IP address. If softiWARP binds and surfaces another verbs interface on an existing one, this scheme will not work.

Quoted 8 lines

Chien

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

I haven't thought through all the details, but in principal this should work. But this isn't just and iWARP issue. Currently all RDMA-CM users
share the same port space. I think we need to maintain this, so a
transport-independent RDMA app can run over both IB and IW. This goes for server side wrt listen/accept as well.

Steve.

Quoted 3 lines · Chien Tin

Steve,

Do you see any issues with Bernard's proposal? Is this something

we can agree on?

Quoted 4 lines

Chien

Quoted 2 lines

Quoted 2 lines

-----Original Message-----
From: [email protected] [mailto:linux-rdma-

[email protected]] On Behalf Of Tung,

Quoted 4 lines

Quoted 4 lines

Chien Tin
Sent: Friday, June 25, 2010 3:15 PM
To: Bernard Metzler; Roland Dreier
Cc: Jason Gunthorpe; [email protected]; linux-rdma-

[email protected]; Waskiewicz Jr,

Quoted 2 lines

Quoted 2 lines

Peter P; Steve Wise
Subject: RE: [PATCH v2] RDMA/CMA: fix iWARP adapter TCP port space

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

To my understanding, our discussion touches two topics. One is to solve the TCP port space issue, the other is more general, its

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

proper integration of offloaded TCP within Linux. So, the second topic is a generalization of the first.

Regarding the first topic, what I was about to propose is that the iWARP kernel driver (software iWARP or RNIC) itself should take care

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

port space allocations. Port space maintenance functionality should be minimized at iWARP CM level. It looks straightforward to me if during the rdma_connect() call the driver picks a free port using a socket/bind sequence for its local interface. The same would be

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

for
the passive connection setup, which always involves an rdma_bind_addr

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

- we would have to pass the rdma_bind_addr() call down to the driver and EADDRINUSE would be a reasonable return value.
Here things are getting a little more complicated, if it comes to INADDR_ANY and port 0 bindings. In private email, Bob Sharp already suggested it - the iWARP CM would have to pick a port and try it on all interfaces....maybe by starting with port 0 binding on one interface and trying to extend with the returned port on all remaining interfaces. That introduces an unbind() call if things fail, too. In any case, the rdma_bind_addr() call would create

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

state
at driver level.

I am okay with adding rdma_bind_addr and rdma_unbind_addr calls. I

Quoted 1 line

Quoted 1 line

speak for Sean and the work that needs to go into the CM. But

this will allow

Quoted 4 lines

Quoted 4 lines

all known iWARP implementations to work together.

Quoted 1 line

For softiwarp, during bind() or connect(), a TCP socket would be

Quoted 1 line

Quoted 1 line

Quoted 1 line

and bound, for an RNIC driver (currently) the same would happen.While

Quoted 3 lines

Quoted 3 lines

Quoted 3 lines

softiwarp this socket would be used for communication later, the RNIC driver
would simply have to keep it around until the connection endpoint

Quoted 4 lines

Quoted 4 lines

Quoted 3 lines

destroyed
or the port gets unbound.

We want to be careful and make sure there is only one iWARP

provider per IP address.

Quoted 1 line

Quoted 1 line

If softiWARP binds and surfaces another verbs interface on an existing

Quoted 7 lines

Quoted 7 lines

this scheme will not work.

Chien

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma"

Quoted 5 lines

Quoted 3 lines

the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma"

Quoted 3 lines

the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Steve,

merging the port space of different transports is very particular and it would be good if we could avoid it w/o losing on transport independence.

Could we prove a strong need for that short-cut?
If its really needed, one global RDMA_CM instance should be able to achive that requirement. But wouldn't that go together with a waste of unused transports resources (i.e., IB binds would create and bind also TCP sockets..?).

But the IB and IW port space is _already_ merged by the RDMA_CM. That's my point. So a solution that breaks this impacts transport-independent applications that use the rdma cm...

Quoted 6 lines

Could we prove a strong need for that short-cut?
If its really needed, one global RDMA_CM instance should be able to achive that requirement. But wouldn't that go together with a waste of unused transports resources (i.e., IB binds would create and bind also TCP sockets..?).

My comment has nothing to do with TCP ports but rather RDMA ports as abstracted by the RDMA_CM. The RDMA_PS_TCP port space is shared among all the RDMA transports. So if you rdma_bind() to port space RDMA_PS_TCP, addr INADDR_ANY, port 9999, then it will bind the cm_id to all IB and IW providers for that address/port (note the providers don't really get called until rdma_listen()). Thus if you propose moving the bind operation down into the device providers, then the IB providers will also have to add this logic. Or some solution must be defined to keep the IB and IW port spaces common. Otherwise the RDMA_CM cannot support transport independent applications...

Quoted 12 lines

But the IB and IW port space is _already_ merged by the RDMA_CM. That's my point. So a solution that breaks this impacts transport-independent applications that use the rdma cm...

Quoted 7 lines

Could we prove a strong need for that short-cut?
If its really needed, one global RDMA_CM instance should be able to achive that requirement. But wouldn't that go together with a waste of unused transports resources (i.e., IB binds would create and bind also TCP sockets..?).

Quoted 11 lines

My comment has nothing to do with TCP ports but rather RDMA ports as abstracted by the RDMA_CM. The RDMA_PS_TCP port space is shared among all the RDMA transports. So if you rdma_bind() to port space RDMA_PS_TCP, addr INADDR_ANY, port 9999, then it will bind the cm_id to all IB and IW providers for that address/port (note the providers don't really get called until rdma_listen()). Thus if you propose moving the bind operation down into the device providers, then the IB providers will also have to add this logic. Or some solution must be defined to keep the IB and IW port spaces common. Otherwise the RDMA_CM cannot support transport independent applications...

I didn't think about this before, a bit too focused on solving the iWARP side. Bernard, what about letting RDMA CM allocate the socket and exposing it to softiWARP?

The base argument is Dave Miller doesn't like RDMA or iWARP. I completely respect Dave's opinions, but in this case, I don't think it's his call, since this patch is completely isolated to the IB tree and the RDMA stack. Not including this will directly affect anyone using iWARP with Linux. Hopefully those people will chime in at some point, but I'm not sure that will help because the arguments so far against this have been somewhat irrational, and nothing technical-oriented.

Quoted 7 lines

The base argument is Dave Miller doesn't like RDMA or iWARP. I completely respect Dave's opinions, but in this case, I don't think it's his call, since this patch is completely isolated to the IB tree and the RDMA stack. Not including this will directly affect anyone using iWARP with Linux. Hopefully those people will chime in at some point, but I'm not sure that will help because the arguments so far against this have been somewhat irrational, and nothing technical-oriented.

My impression of the general netdev argument is that if it touches sockets it has to have 100% of the behavior and functionality of the Linux stack. iwarp fails that test.. There is absolutely no way to fix that - that is rational, technical and not at all 'solution oriented' :)

The only reason iwarp has got as far as it has in the offical tree is because it piggy backed on IB's 'sandbox' approach to the RDMA layer. So, if you want to play in that area you have to play by those rules, and that means staying in the sandbox ..

Quoted 9 lines

You can peruse this thread from 2007:

http://lkml.org/lkml/2007/8/15/174

In that thread David Miller said he would NAK this solution and even NAK a solution where we expose the low level port allocation services and allow RDMA/iWARP and TCP to share the port space. Thus I see no way to resolve this issue with Dave's ACK, and maybe the RDMA contributors and maintainers should move forward without it?

If Dave Miller says this is an unacceptable thing to merge, then I'm not going to merge it over his explicit and unequivocal NAK. Trying sneak this in by hiding this patch on linux-rdma without cc-ing netdev is the wrong approach... the right approach is to get the hordes of users being hurt by this issue (if there are such hordes), make a big stink, and force a solution that everyone agrees on.

V2 changes:

Quoted 1 line

V2 changes:

What you consider to be V1, this thread from 2007?