rsockets and other performance

Traditional sockets based applications wanting high throughput could use rsockets Since it is layered on top of uverbs we expected to see good throughput numbers. So, we started to run netperf and iperf. We observed that it tops off at about 20Gb/s with QDR adapters. A quick "perf top" revealed a lot of cycles spent in memcpy(). We had hoped these numbers would be somewhat higher since we did not expect the memcpy() to have such a large overhead.

Given the copy overhead, we wanted to revisit the IPoIB and SDP performance. Hence we installed to OFED-1.5.4.1 on RHEL 6.2. We found that for small packets SDP starts with low throughputs, but seems to catch up with rsockets at about 16 KB packets. On the other hand IPoIB CM tops off at about 10 Gb/s.

Since SDP does in kernel RDMA we expected IPoIB CM and SDP numbers to be much closer. Again "perf top" revealed that IPoIB was spending a large number of cycles in checksum computation. Out of curiosity Sridhar made the following changes:

@@ -670,6 +670,7 @@ copied:
     skb->dev = dev;
     /* XXX get correct PACKET_ type here */
     skb->pkt_type = PACKET_HOST;
+    skb->ip_summed = CHECKSUM_UNNECESSARY;
     netif_receive_skb(skb);
 

@@ -1464,7 +1464,8 @@ static ssize_t set_mode(struct device *d
                "will cause multicast packet drops\n");
 
         rtnl_lock();
-        dev->features &= ~(NETIF_F_IP_CSUM | NETIF_F_SG | NETIF_F_TSO);
+        dev->features &= ~(NETIF_F_SG | NETIF_F_TSO);
         priv->tx_wr.send_flags &= ~IB_SEND_IP_CSUM;
 
         if (ipoib_cm_max_mtu(dev) > priv->mcast_mtu)

With these minimal changes IPoIB throughput reached between 19-20Gb/s with just 2 threads. This was really unexpected. Given that, we wanted to revisit the usage of checksums in IPoIB. So, it looks worthwhile to allow for 'checksum-less' IPoIB-CM within a cluster on a single subnet. From a checksum perspective, this would be no different from RDMA. What are your thoughts?

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Someone more familiar with ipoib needs to respond regarding those changes. For rsockets, please make sure that you've pulled the latest code. You can improve the performance by adjusting the QP size and send/receive buffers, which can be done through config files. I started an rsocket man page that describes the files that I just pushed out to my git tree.

- Sean
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Sridhar,

If you **only** enable NETIF_F_SG for CM, does this yields any gain? did you had to patch the code for that end?

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Or, We did not try only NETIF_F_SG for IPoIB CM. We tried it along with NETIF_F_IP_CSUM. We did run
into issues with netperf i.e. netperf errors out along with NETIF_F_SG. However, iperf worked fine. Did not investigate that further.

Thanks
Pradeep

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

There have been discussions around a 'checksum-less' IPoIB operation for a little while.

The basic notion was to enable the checksum offload mechanism, pass the information from Linux for offload straight through to the other side (eg via an extra header or something), have the other side reconstruct the offload indication on RX and inject back to into the net stack.

This would be similar to the way checksum bypass works in virtualization (Xen/KVM) where the virtualized net TX just packages the offload data and sends it to the hyperviser kernel which then RX's it and restores the very same checksum offload information.

During the CM process this feature would be negotiated.

I don't think anyone ever made patches for this, but considering the performance delta you see it really seems worthwhile.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

How about something like below? Basically the the 'checksum-less' operation is only between hosts that both support it by extending the existing IB connection setup mechanism. The following also keeps the
changes confined to ipoib-cm module.

- add a sysctl variable csum_simulate
- In ipoinb-cm module
if (csum_simulate)
advertize hardware checksum offload capabilities

- when a QP is created to a remote host it checks csum_simulate. if (csum_simulate)
include CSUM_SIMULATE command in RC private data when setting up the connection
Note: the RFC 4755 utilizes this private data to exchange the receive MTU and UD QP. We just add another parameter to it.

If accepted by the other end during connection negotiation, then set csum_simulate_on = 1 (For the QP)

- when a QP connection request is received
if (csum_simulate)
look for CSUM_SIMULATE field in the private data if present respond with CSUM_SIMULATE else zero it in response set csum_simulate_on flag = 1 for the QP

In the above two steps one would also want to check that the peer is a directly connected host.

- when sending data
if (csum_simulate_on)
send the data over ipoib-cm link normally (no data checksum added)
else /* sending over a QP not enabled for checksum offload */
calculate the overall checksum
send data

- when receiving data
if (csum_simulate_on)
set CSUM_UNNECESSARY indicating csum has been validated

thanks
Vivek

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

It is much better to do things properly (as I described) so you don't have a gotcha when a packet gets routed.

We don't want to fake that we are doing csum, we want to forward the csum offload data to the other side, which might discard it or might use it to forward the packet out another interface. This is not hard, just fiddly work.

Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>        (780)4406067x832
Chief Technology Officer, Obsidian Research Corp         Edmonton, Canada
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

We certainly need to ensure that the routed packet is not left without a
checksum. One either does not enable 'checksum less link' on routers, or else the router must be able to take care of it when it forwards the packet.

By 'csum offload data' do you mean something akin to csum_start/csum_offset? How would one transmit this information when using IPoIB-CM across from the sending host to the router? If we can do that it will certainly be useful.

Otherwise what I was proposing was that a router is either configured to not
accept checksum less configuration on IPoIb-CM, or if checksum less links are supported then, when forwarding to a 'checksum less' interface it does nothing; but when forwarding to any other interface the checksum is added.

This addition is really done in the ipoib-cm driver to insulate from the rest of the stack. The 'faking' of checksum offload is only to the local IP stack so that it does not do the calculation but leaves it to the IPoIB-CM driver. That is what we essentially do with virtio or hardware offload mechanisms.

We are working on a test patch. That should make it easier to discuss the gaps or if the solution is complete.

thanks
Vivek

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

The latter is vastly preferred.

That plus a ip_summed needed flag seems to match what virtio_net is doing. Review virtio_net.c:receive_buf and xmit_skb to see how it handles the packet.

Also, I am just reminded, there has been some interest in also forwarding GSO through the CM connection. This would speed things up as well and avoid the need for the wonky 64k MTU.

ie rework the CM mode to follow more closely how virtio-net works.

Like virtio_net you'd have to prefix a small header, or if you are lucky, maybe encoding things in the 32 bits of immediate data.

This sort of solution is something I'd like to avoid, it is not necessary to make something so fragile.

I mean 'faking' in the sense you tell the stack you will compute the checksum, throw away the information needed to do that calculation, and the send a packet on the wire that can never be check-summed properly.

I don't see the point of keeping things contained to the CM part of the ipoib driver, if the ipoib packet handling needs to be changed then change it..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html