rxe dev tree available (soft RDMAoE)
Hello,
The development tree for a soft RDMA transport over Ethernet driver (rxe) is available in the OFA git repository. This is a work in progress but has enough functionality for people interested in looking at it to be worth sharing. Going forward changes to the driver will be maintained here.
To get a copy of the driver type
git clone git://git.openfabrics.org/~fzago/rxe
This work is being supported by Mellanox and System Fabric Works.
Regards,
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · frank zago
Hello,
The development tree for a soft RDMA transport over Ethernet driver (rxe) is available in the OFA git repository. This is a work in progress but has enough functionality for people interested in looking at it to be worth sharing. Going forward changes to the driver will be maintained here.
Is this going to be turned into a proper kernel patch submission?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Jason,
Yes, but later. For now it is still being actively developed and is changing a lot. Once it settles down we plan to convert to patches and push up stream.
Bob Pearson
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Wednesday, December 16, 2009 3:38 PM
To: frank zago
Cc: [email protected]; Bob Pearson; John Groves
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 7 lines · frank zago
Hello,
The development tree for a soft RDMA transport over Ethernet driver (rxe) is available in the OFA git repository. This is a work in progress but has enough functionality for people interested in looking at it to be worth sharing. Going forward changes to the driver will be maintained here.
Is this going to be turned into a proper kernel patch submission?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines
Yes, but later. For now it is still being actively developed and is changing a lot. Once it settles down we plan to convert to patches and push up stream.
Is there any commonality that can be shared between rxe and the qlogic driver?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Sean,
Probably someday. We looked at the ipath driver before writing rxe but the qib driver came out much later and we haven't really spent any time looking at it. Currently rxe is written to be standalone and doesn't have an obvious place to split into pieces. Some of the header files could maybe go into core. (For example rxe_hdr.h and rxe_opcode.[ch] overlap include/rdma/ib_pack.h).
The hard parts of the driver are the state machines and they are fairly specific.
Bob
-----Original Message-----
From: Sean Hefty [mailto:[email protected]]
Sent: Wednesday, December 16, 2009 4:20 PM
To: 'Robert Pearson'; 'Jason Gunthorpe'; 'frank zago'
Cc: [email protected]; 'John Groves'
Subject: RE: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 1 line
Yes, but later. For now it is still being actively developed and is
changing
Quoted 2 lines
a lot. Once it settles down we plan to convert to patches and push up stream.
Is there any commonality that can be shared between rxe and the qlogic driver?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines
Is there any commonality that can be shared between rxe and the qlogic driver?
Interesting to note that according to ohcount, rxe is 9591 non-blank non-comment lines of kernel code, while qib is 36957 lines. So the complexity of IB transport is a small part of the complexity of qib.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
What is soft RDMA over ethernet, does this mean it is RDMAoE that will run over any ethernet interface that is installed in the system ?
woody
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of frank zago
Sent: Wednesday, December 16, 2009 1:07 PM
To: [email protected]
Cc: Bob Pearson; John Groves
Subject: [Announce] rxe dev tree available (soft RDMAoE)
Hello,
The development tree for a soft RDMA transport over Ethernet driver (rxe) is available in the OFA git repository. This is a work in progress but has enough functionality for people interested in looking at it to be worth sharing. Going forward changes to the driver will be maintained here.
To get a copy of the driver type
git clone git://git.openfabrics.org/~fzago/rxe
This work is being supported by Mellanox and System Fabric Works.
Regards,
Frank
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · ---- "Woodruff
What is soft RDMA over ethernet, does this mean it is RDMAoE that will run over any ethernet interface that is installed in the system ?
That's correct. Currently for every ethX interface present, it creates a corresponding rxeX device. These rxeX devices behaves the same way than regular IB devices.
Frank.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 16 Dec 2009, Roland Dreier wrote:
| | > Is there any commonality that can be shared between rxe and the qlogic driver? | | Interesting to note that according to ohcount, rxe is 9591 non-blank | non-comment lines of kernel code, while qib is 36957 lines. So the | complexity of IB transport is a small part of the complexity of qib.
Yep. And a good chunk of the complexity of the qib driver (and ipath before it) was in dealing with the serdes and it's related issues (no firmware on the chip to do it, so it's all done in the driver.).
Dave Olson
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
The ipath driver, which I am more familiar with, replicated the transport for RC, UC and UD which also caused some code duplication. We tried to combine them all together in rxe.
Bob Pearson
-----Original Message-----
From: Roland Dreier [mailto:[email protected]]
Sent: Wednesday, December 16, 2009 5:53 PM
To: Sean Hefty
Cc: 'Robert Pearson'; 'Jason Gunthorpe'; 'frank zago'; [email protected]; 'John Groves'
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 1 line
Is there any commonality that can be shared between rxe and the qlogic
driver?
Interesting to note that according to ohcount, rxe is 9591 non-blank non-comment lines of kernel code, while qib is 36957 lines. So the complexity of IB transport is a small part of the complexity of qib.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
Yes, but later. For now it is still being actively developed and is changing a lot. Once it settles down we plan to convert to patches and push up stream.
I think you would probably be better off developing the kernel driver as a branch in an up-to-date kernel tree, and then worrying about backports later. You could get rid of some of the redundant infrastructure like dump_hex() and crc32 (too bad IB didn't choose the CRC32C polynomial, so you can't use the crypto API and get Intel CRC32 instruction acceleration for free).
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
I agree about the polynomial. That would have been nice. As currently proposed the embedding of the IB transport in Ethernet frames preserves the IB ICRC as part of the transport and trades the VCRC for Ethernet's CRC32 over the whole frame. This means that the packet is effectively covered by CRC32 twice except for the 802 headers and the fields that are excluded from the ICRC. CRC32C should give better protection.
We have thought that we could try CRC32C first and if that works then use that, otherwise fall back to CRC32 which would allow rxe to use CRC32C when talking to itself and get a speedup on new Intel CPUs.
The crc32 code we have is based on some suggestions from Intel and seems to be much faster than the 'standard' code in the kernel and we are/were planning to merge the crc computation with the data copy. It is actually less explicitly optimized than the code in the kernel but when we use the loop unrolling directive in the compiler we get better overall results. That core loop gives us about 5-6GB/sec for the copy/crc vs about 500MB/sec for the 'standard' crc + copy.
Our plan is to strip out all the trace and dump code when the driver is stable. We will keep the warn and error messages.
Bob
-----Original Message-----
From: Roland Dreier [mailto:[email protected]]
Sent: Thursday, December 17, 2009 3:26 PM
To: Robert Pearson
Cc: 'Jason Gunthorpe'; 'frank zago'; [email protected]; 'John Groves'
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 1 line
Yes, but later. For now it is still being actively developed and is
changing
Quoted 2 lines
a lot. Once it settles down we plan to convert to patches and push up stream.
I think you would probably be better off developing the kernel driver as a branch in an up-to-date kernel tree, and then worrying about backports later. You could get rid of some of the redundant infrastructure like dump_hex() and crc32 (too bad IB didn't choose the CRC32C polynomial, so you can't use the crypto API and get Intel CRC32 instruction acceleration for free).
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines
We have thought that we could try CRC32C first and if that works then use that, otherwise fall back to CRC32 which would allow rxe to use CRC32C when talking to itself and get a speedup on new Intel CPUs.
How would you negotiate that? Is this being worked in the definition of the IBoE spec? It seems in a final standard, having something stronger than two CRC32s and something that can handle jumbo frames would be worth it.
Quoted 7 lines
The crc32 code we have is based on some suggestions from Intel and seems to be much faster than the 'standard' code in the kernel and we are/were planning to merge the crc computation with the data copy. It is actually less explicitly optimized than the code in the kernel but when we use the loop unrolling directive in the compiler we get better overall results. That core loop gives us about 5-6GB/sec for the copy/crc vs about 500MB/sec for the 'standard' crc + copy.
Interesting ... can the upstream kernel code be improved similarly?
Quoted 2 lines
Our plan is to strip out all the trace and dump code when the driver is stable. We will keep the warn and error messages.
If you develop upstream and use the ftrace tracepoints stuff then the overhead of (runtime) disabled trace points is nearly zero (one 5-byte NOP, so a cycle or two). So you can keep them forever (which is realistically how long you're going to be debugging this :)
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Robert Pearson
I agree about the polynomial. That would have been nice. As currently proposed the embedding of the IB transport in Ethernet frames preserves the IB ICRC as part of the transport and trades the VCRC for Ethernet's CRC32 over the whole frame. This means that the packet is effectively covered by CRC32 twice except for the 802 headers and the fields that are excluded from the ICRC. CRC32C should give better protection.
The ICRC isn't even necessary from a technical sense for DCE. The underlying reasons for the ICRC/VCRC split are not really present for ethernet. Considering RDMAoE isn't interoperable with existing IB anyhow and DCE can't do routing, the best course would be to just get rid of it entirely.
IMHO, a smart thing would be to figure out how to exchange ICRC capability during the connection setup and turn it off if both sides support it. With the ICRC removed the rxe send side should be able to be completely 0 copy...
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Of course we need to interoperate with ConnectX-en and it, as far as I know, only knows how to compute the VCRC as though the packet was going to get sent to IB including a phantom 12 byte LRH that is filled with 1's without any way to turn it off. It is also a requirement that UD work well too. Instead of connection setup if we could get a capability bit in a port info record we could handle the general case.
Bob
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Thursday, December 17, 2009 4:05 PM
To: Robert Pearson
Cc: 'Roland Dreier'; 'frank zago'; [email protected]; 'John Groves'
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 2 lines · Robert Pearson
I agree about the polynomial. That would have been nice. As currently proposed the embedding of the IB transport in Ethernet frames preserves
the
Quoted 3 lines
IB ICRC as part of the transport and trades the VCRC for Ethernet's CRC32 over the whole frame. This means that the packet is effectively covered by CRC32 twice except for the 802 headers and the fields that are excluded
from
Quoted 1 line
the ICRC. CRC32C should give better protection.
The ICRC isn't even necessary from a technical sense for DCE. The underlying reasons for the ICRC/VCRC split are not really present for ethernet. Considering RDMAoE isn't interoperable with existing IB anyhow and DCE can't do routing, the best course would be to just get rid of it entirely.
IMHO, a smart thing would be to figure out how to exchange ICRC capability during the connection setup and turn it off if both sides support it. With the ICRC removed the rxe send side should be able to be completely 0 copy...
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Forgot to reply to one other point. For the most part we have been developing in vanilla recent kernel.org trees when we are testing rxe<->rxe. To talk to ConnectX we need to use the OFED-rdmaoe environment and then we have to play with symvers. If support for ConnectX-en with rdmaoe were upstream then we would only use those kernels.
-----Original Message-----
From: Roland Dreier [mailto:[email protected]]
Sent: Thursday, December 17, 2009 3:26 PM
To: Robert Pearson
Cc: 'Jason Gunthorpe'; 'frank zago'; [email protected]; 'John Groves'
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 1 line
Yes, but later. For now it is still being actively developed and is
changing
Quoted 2 lines
a lot. Once it settles down we plan to convert to patches and push up stream.
I think you would probably be better off developing the kernel driver as a branch in an up-to-date kernel tree, and then worrying about backports later. You could get rid of some of the redundant infrastructure like dump_hex() and crc32 (too bad IB didn't choose the CRC32C polynomial, so you can't use the crypto API and get Intel CRC32 instruction acceleration for free).
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Oops. I should have said ICRC not VCRC. There is no VCRC for rdmaoe. The ICRC is, however, computed exactly the same as for an IB packet. It is bit for bit identical.
-----Original Message-----
From: Robert Pearson [mailto:[email protected]]
Sent: Thursday, December 17, 2009 5:00 PM
To: 'Jason Gunthorpe'
Cc: 'Roland Dreier'; 'frank zago'; '[email protected]'; 'John Groves'
Subject: RE: [Announce] rxe dev tree available (soft RDMAoE)
Of course we need to interoperate with ConnectX-en and it, as far as I know, only knows how to compute the VCRC as though the packet was going to get sent to IB including a phantom 12 byte LRH that is filled with 1's without any way to turn it off. It is also a requirement that UD work well too. Instead of connection setup if we could get a capability bit in a port info record we could handle the general case.
Bob
-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org]
Sent: Thursday, December 17, 2009 4:05 PM
To: Robert Pearson
Cc: 'Roland Dreier'; 'frank zago'; [email protected]; 'John Groves'
Subject: Re: [Announce] rxe dev tree available (soft RDMAoE)
Quoted 2 lines · Robert Pearson
I agree about the polynomial. That would have been nice. As currently proposed the embedding of the IB transport in Ethernet frames preserves
the
Quoted 3 lines
IB ICRC as part of the transport and trades the VCRC for Ethernet's CRC32 over the whole frame. This means that the packet is effectively covered by CRC32 twice except for the 802 headers and the fields that are excluded
from
Quoted 1 line
the ICRC. CRC32C should give better protection.
The ICRC isn't even necessary from a technical sense for DCE. The underlying reasons for the ICRC/VCRC split are not really present for ethernet. Considering RDMAoE isn't interoperable with existing IB anyhow and DCE can't do routing, the best course would be to just get rid of it entirely.
IMHO, a smart thing would be to figure out how to exchange ICRC capability during the connection setup and turn it off if both sides support it. With the ICRC removed the rxe send side should be able to be completely 0 copy...
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Robert Pearson
Of course we need to interoperate with ConnectX-en and it, as far as I know, only knows how to compute the VCRC as though the packet was going to get sent to IB including a phantom 12 byte LRH that is filled with 1's without any way to turn it off. It is also a requirement that UD work well too. Instead of connection setup if we could get a capability bit in a port info record we could handle the general case.
There is no IB management layer (so no port info record) for RDMAoE..
Another good choice might be to not use IPv6 ND and include a capabilities struct of some sort in a RDMAoE specific ND packet, and, say, define GRH.nextHeader = 0x2B as meaning 'BTH with no ICRC'
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines
Of course we need to interoperate with ConnectX-en and it, as far as I know, only knows how to compute the VCRC as though the packet was going to get sent to IB including a phantom 12 byte LRH that is filled with 1's without any way to turn it off. It is also a requirement that UD work well too. Instead of connection setup if we could get a capability bit in a port info record we could handle the general case.
How do you get port info records in an IBoE fabric? There's no SA to query, right?
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines
The ICRC isn't even necessary from a technical sense for DCE. The underlying reasons for the ICRC/VCRC split are not really present for ethernet. Considering RDMAoE isn't interoperable with existing IB anyhow and DCE can't do routing, the best course would be to just get rid of it entirely.
Not sure I agree... it would be entirely possible and sensible to create an IBoE router (and if the IBoE GRH used global scope addresses, pretty much the same as routing IPv6), and I think you would want end-to-end ICRC protection in that case (since the router would be regenerating the ethernet CRC). Even in the ethernet case there are probably realistic cases of adding a VLAN tag or something that require the CRC to be recalculated by a switch -- and some switches probably cut corners and just recalculate the CRC of all packets on egress, just to avoid conditional cases.
- R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html