Linux and system area networks
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines
I'd like to find out if anyone has thought about how Linux will handle some of the new network technologies people are starting to push. Specifically I'm talking about "System Area Networks," that is, things like Infiniband, as well as TCP/IP offload.
Infiniband is doing relatively well, as much as anything can with Intel at the helm (see "Itanic"). This has nothing to do with TCP/IP offload, which is an extremely stupid idea. The whole thing stems from a desire by vendors to sell "smart" (== very expensive) NICs.
RDMA in Infiniband is, in my view, a little more than traditional DMA in any other advanced server I/O bus. Sun UPA is packetized, for instance.
Quoted 5 lines
The rough idea is that WSD is a new user space library that looks at sockets calls and decides if they have to go through the usual kernel network stack, or if they can be handed off to a "SAN service provider" which bypasses the network stack and uses hardware reliable transport and possibly RDMA.
That can be done in Linux just as easily, using same DLLs (they are called .so for "shared object"). If you look at Ashok Raj's Infi presentation, you may discern "user-level sockets", if you look hard enough. I invite you to try, if errors of others did not teach you anything.
Quoted 5 lines
This means that all applications that use Winsock benefit from the advanced network hardware. Also, it means that Windows is much easier for hardware vendors to support than other OSes. For example, Alacritech's TCP/IP offload NIC only works under Windows. Microsoft is also including Infiniband support in Windows XP and Windows 2002.
IMHO, Alacritech is about to join scores and scores of vendors who tried that before. Customers understand very soon that a properly written host based stack works much better in the face of a changing environment: Faster CPUs, new CPUs (IA-64), new network protocols (ECN). Besides, it is easy to "accelerate" a bad network stack, but try to outdo a well done stack.
Quoted 3 lines
So I guess my question is whether anyone has started thinking about the architectural changes needed to make System Area Networking and TCP/IP offload easier under Linux.
Pretty much zero-copy that DaveM and Co. do addresses this.
-- Pete
Quoted 5 lines
a properly written host based stack works much better in the face of a changing environment: Faster CPUs, new CPUs (IA-64), new network protocols (ECN). Besides, it is easy to "accelerate" a bad network stack, but try to outdo a well done stack.
Putting the stack partly in user spacd can sometimes be a benefit. Linux 8086 does this to cut down kernel size for example ;)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Pete" == Pete Zaitcev <[email protected]> writes:
Roland> The rough idea is that WSD is a new user space library Roland> that looks at sockets calls and decides if they have to go Roland> through the usual kernel network stack, or if they can be Roland> handed off to a "SAN service provider" which bypasses the Roland> network stack and uses hardware reliable transport and Roland> possibly RDMA.
Pete> That can be done in Linux just as easily, using same DLLs Pete> (they are called .so for "shared object"). If you look at Pete> Ashok Raj's Infi presentation, you may discern "user-level Pete> sockets", if you look hard enough. I invite you to try, if Pete> errors of others did not teach you anything.
I think you misunderstood the point. Microsoft is providing this WSD DLL as a standard part of W2K now. This means that hardware vendors just have to write a SAN service provider, and all Winsock-using applications benefit transparently. No matter how good your TCP/IP implementation is, you still lose (especially in latency) compared to using reliable hardware transport. Oracle-with-VI and DAFS-vs-NFS benchmarks show this quite clearly.
Linux has nothing to compare to Winsock Direct. I agree, one could put an equivalent in glibc, or one could take advantage of Linux's relatively low system call latency and put something in the kernel. The unfortunate consequence of this is that SAN (system area network) hardware vendors are not going to support Linux very well.
BTW, do you have a pointer to Ashok Raj's presentation?
Roland> This means that all applications that use Winsock benefit Roland> from the advanced network hardware. Also, it means that Roland> Windows is much easier for hardware vendors to support Roland> than other OSes. For example, Alacritech's TCP/IP offload Roland> NIC only works under Windows. Microsoft is also including Roland> Infiniband support in Windows XP and Windows 2002.
Pete> IMHO, Alacritech is about to join scores and scores of Pete> vendors who tried that before. Customers understand very Pete> soon that a properly written host based stack works much Pete> better in the face of a changing environment: Faster CPUs, Pete> new CPUs (IA-64), new network protocols (ECN). Besides, it Pete> is easy to "accelerate" a bad network stack, but try to Pete> outdo a well done stack.
OK, how about an Infiniband network with a TCP/IP gateway at the edge? Have we thought about how Linux servers should use the gateway to talk to internet hosts? Surely there's no point in running TCP/IP inside the Infiniband network, so there needs to be some concept of "socket over Infiniband."
Thanks,
Roland
Quoted 5 lines
OK, how about an Infiniband network with a TCP/IP gateway at the edge? Have we thought about how Linux servers should use the gateway to talk to internet hosts? Surely there's no point in running TCP/IP inside the Infiniband network, so there needs to be some concept of "socket over Infiniband."
So write the library, it shouldnt need the kernel involved, and you can take over AF_INET socket syscalls with an LD_PRELOAD so it can be transparent