announcing rump kernel TCP/IP stack for DPDK

Hi,

I like the opportunities that a technology like DPDK enables, and I felt that the availability of an open source TCP/IP stack for DPDK could make things even more interesting. I've been working on a concept called the anykernel, where the idea is that an OS kernel should be structured in a fashion which allows the driver components to be run independently of a monolithic kernel in so-called rump kernels. Long story short, one of the "byproducts" is a run-anywhere standalone version of NetBSD kernel TCP/IP stack, so it was a natural progression to integrate that TCP/IP stack with the NIC driver layer provided by DPDK.

The published code completes "stage 1: make it work". I want to thank another person who wished to remain anonymous for help with the implementation and testing. Next up is "stage 2: make it fast". I welcome everyone to contribute ideas / use cases / code / etc. towards that goal.

You can find the code for plugging DPDK into the rump kernel NIC layer along with instructions from:

https://github.com/anttikantee/dpdk-rumptcpip

In case anyone is interested, I tested the setup with Void Linux running in VM using an emulated 82540 NIC (cf. my previous mail to this list) and the other party tested on Ubuntu using real hardware and a 82599 NIC.

cheers,
antti

Quoted 4 lines · On Thu, 23 May 2013 18:40:38 +0200 Antti Kantee…

Hi,

I like the opportunities that a technology like DPDK enables, and I felt
that the availability of an open source TCP/IP stack for DPDK could using real hardware and a 82599 NIC.

Already done several times before.
http://savannah.nongnu.org/projects/lwip/
https://code.google.com/p/tcp-ip-in-user-space/
http://www.ohloh.net/p/tcp-ip-in-user-space
http://www.ioremap.net/projects/unetstack
http://www.joerch.org/tcpip/
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757

Of course. And many more times than that. This is one more option for us and it is BSD licensed.

Ultimately, we will probably need a TCP stack in there.

2013/5/23 Robert Bays <[email protected]>:

Quoted 3 lines

Of course. And many more times than that. This is one more option for us and it is BSD licensed.

Ultimately, we will probably need a TCP stack in there.

indeed, but what do you mean by the TCP stack? I hope not the BSD sockets :) rx/tx bufs+simple API is all you need.

Quoted 7 lines

2013/5/23 Robert Bays <[email protected]>:

Quoted 4 lines

Of course. And many more times than that. This is one more option for us and it is BSD licensed.

Ultimately, we will probably need a TCP stack in there.

Quoted 3 lines

indeed, but what do you mean by the TCP stack? I hope not the BSD sockets :)
rx/tx bufs+simple API is all you need.

Luckily, ripping out sockets and replacing them with a "high-speed aware" API is easier than starting from a simple API and implementing IPv{4,6}, TCP, IPsec, ...

Already done several times before.
http://savannah.nongnu.org/projects/lwip/
https://code.google.com/p/tcp-ip-in-user-space/
http://www.ohloh.net/p/tcp-ip-in-user-space
http://www.ioremap.net/projects/unetstack
http://www.joerch.org/tcpip/
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757

Quoted 7 lines · Stephen Hemminger

Already done several times before.
http://savannah.nongnu.org/projects/lwip/
https://code.google.com/p/tcp-ip-in-user-space/
http://www.ohloh.net/p/tcp-ip-in-user-space
http://www.ioremap.net/projects/unetstack
http://www.joerch.org/tcpip/
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.167.2757

I'm aware of the paper you linked given that I wrote it. I don't agree that it supports your claim "already done several times before".

Quoted 2 lines

I'm aware of the paper you linked given that I wrote it. I don't agree that it supports your claim "already done several times before".

I was more worried about multiple different TCP-IP stacks that seem to be only used by small number of people and not maintained. Doing full TCP/IP is hard, and there are lots of features inside.
It would be great to have one that is well supported and maintained.

Quoted 7 lines · Stephen Hemminger

Quoted 2 lines

I'm aware of the paper you linked given that I wrote it. I don't agree that it supports your claim "already done several times before".

I was more worried about multiple different TCP-IP stacks that seem to be only used by small number of people and not maintained. Doing full TCP/IP is hard, and there are lots of features inside.
It would be great to have one that is well supported and maintained.

I'm also well aware that doing TCP/IP right is very very hard. It's not possible sit down with the spec for a weekend, write up some code, and hope it will work in the real world. That's why I didn't implement TCP/IP. The implementation I mentioned is the unmodified NetBSD kernel TCP/IP stack running on a very thin hypervisor layer. It's not maintained by a few people, it's literally maintained by the entire NetBSD community. Furthermore, it's not a set of patches available for the NetBSD kernel updated every now and then, it's literally [in] the kernel. It's been working this way since 2008. What I did now was just write 200 lines of code to plug the TCP/IP stack onto DPDK.

Hopefully that convinced you that it's not just some random one-shot kinda-works-except-in-reality bitrot attractor ;)

- antti

We had to develop a complete logic from scratch (aka fast path logic) to get a TCP/IP stack with good performances for 6WIND (including some Openflow switching). Unfortunately, as part of our DPDK opensource contribution, we do not plan to publish under open source our 6WINDGate stack (yet).

However, you are very welcomed to introduce a TCP/IP stack for the DPDK. What kind of support/benchmark did you run? Would you like to get it mainstream into the DPDK?

Quoted 1 line

Would you like to get it mainstream into the DPDK?

I think currently integration makes sense only as some sort of a "reachover", i.e. source is not imported into the DPDK tree, but rather fetched on demand. I'm happy with the current state, but if someone wants to work on a reachover solution, I'll be happy to assist, though not drive the effort. Eventually integration may become more pertinent as different optimizations demand radically different code, but let's think about that if/when we get there.