L2 network namespace benchmarking

Hi,

I did some benchmarking on the existing L2 network namespaces.

These patches are included in the lxc patchset at:

    http://lxc.sourceforge.net/patches/2.6.20

The lxc7 patchset series contains Dmitry's patchset The lxc8 patchset series contains Eric's patchset

Here are the following scenarii I made in order to do some simple benchmarking on the network namespace. I tested three kernels:

* Vanilla kernel 2.6.20

* lxc7 with Dmitry's patchset based on 2.6.20
* L3 network namespace has been removed to do testing

* lxc8 with Eric's patchset based on 2.6.20

I didn't do any tests on Linux-Vserver because it is L3 namespace and it is not comparable with the L2 namespace implementation. If anyone is interessted by Linux-Vserver performances, that can be found at
http://lxc.sf.net. Roughly, we know there is no performance degradation.

For each kernel, several configurations were tested:

* vanilla, obviously, only one configuration was tested for reference values.

* lxc7, network namespace
- compiled out
- compiled in
- without container
- inside a container with ip_forward, route and veth
- inside a container with a bridge and veth

* lxc8, network namespace
- compiled out
- compiled in
- without container
- inside a container with a real network device (eth1 was moved in the container instead of using an etun device)
- inside a container with ip_forward, route and etun
- inside a container with a bridge and etun

Each benchmarking has been done with 2 machines running netperf and tbench. A dedicated machine with a RH4 kernel run the bench servers.

For each bench, netperf and tbench, the tests are ran on:

* Intel Xeon EM64T, Bi-processor 2,8GHz with hyperthreading
activated, 4GB of RAM and Gigabyte NIC (tg3)

* AMD Athlon MP 1800+, Bi-processor 1,5GHz, 1GB of RAM and Gigabyte NIC (dl2000)

Each tests are run on these machines in order to have a CPU relative overhead.

# bench on vanilla
===================

----------- --------------------------------------
| Netperf | CPU usage (%) | Throughput (Mbits/s) |
----------- --------------------------------------
| on xeon | 5.99 | 941.38 |
--------------------------------------------------
| on athlon | 28.17 | 844.82 |
--------------------------------------------------

----------- -----------------------
| Tbench | Throughput (MBytes/s) |
----------- -----------------------
| on xeon | 66.35 |
-----------------------------------
| on athlon | 65.31 |
-----------------------------------

# bench from Dmitry's patchset
==============================

1 - with net_ns compiled out
----------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 5.93 / -1 % | 941.32 / 0 % |
-----------------------------------------------------------------------
| on athlon | 28.89 / +2.5 % | 842.78 / -0.2 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 67.00 / +0.9 % |
---------------------------------------------
| on athlon | 65.45 / 0 % |
---------------------------------------------

Observation : no noticeable overhead

2 - with net_ns compiled in
---------------------------

2.1 - without container
-----------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 6.23 / +4 % | 941.35 / 0 % |
-----------------------------------------------------------------------
| on athlon | 28.83 / +2.3 % | 850.76 / +0.7 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 67.00 / 0 % |
---------------------------------------------
| on athlon | 65.45 / 0 % |
---------------------------------------------

Observation : no noticeable overhead

2.2 - inside the container with veth and routes
-----------------------------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 17.14 / +186.1 % | 941.34 / 0 % |
-----------------------------------------------------------------------
| on athlon | 49.99 / +77.45 % | 838.85 / +0.7 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 66.00 / -0.5 % |
---------------------------------------------
| on athlon | 61.00 / -6.65 % |
---------------------------------------------

Observation : CPU overhead is very big, throughput is impacted on the less powerful machine

2.3 - inside the container with veth and bridge
-----------------------------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 19.14 / +299 % | 941.18 / 0 % |
-----------------------------------------------------------------------
| on athlon | 49.98 / +77.42 % | 831.65 / -1.5 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 64.00 / -3.5 % |
---------------------------------------------
| on athlon | 60.07 / -8.3 % |
---------------------------------------------

Observation : CPU overhead is very big, throughput is impacted on the less powerful machine

# bench from Eric's patchset
============================

1 - with net_ns compiled out
----------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 6.04 / +0.8 % | 941.33 / 0 % |
-----------------------------------------------------------------------
| on athlon | 28.45 / +1 % | 840.76 / -0.5 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 65.69 / -1 % |
---------------------------------------------
| on athlon | 65.35 / -0.2 % |
---------------------------------------------

Observation : no noticeable overhead

2 - with net_ns compiled in
---------------------------

2.1 - without container
-----------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 6.02 / +0.5 % | 941.34 / 0 % |
-----------------------------------------------------------------------
| on athlon | 27.93 / -0.8 % | 833.53 / -1.3 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 66.00 / -0.5 % |
---------------------------------------------
| on athlon | 64.94 / -0.9 % |
---------------------------------------------

Observation : no noticeable overhead

2.2 - inside the container with real device
-------------------------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 5.60 / -6.5 % | 941.42 / 0 % |
-----------------------------------------------------------------------
| on athlon | 27.73 / -1.5 % | 835.11 / +1.5 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 74.36 / +12 % |
---------------------------------------------
| on athlon | 70.87 / +8.2 % |
---------------------------------------------

Observation : no noticeable overhead. The network interface is only used by the container, so I guess it does not interact with another network traffic and that explains the performances are better.

2.3 - inside the container with etun and routes
-----------------------------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 16.25 / +171 % | 941.31 / 0 % |
-----------------------------------------------------------------------
| on athlon | 49.99 / +77 % | 828.94 / -1.9 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 65.61 / -1.1 % |
---------------------------------------------
| on athlon | 62.58 / -4.5 % |
---------------------------------------------

Observation : The CPU overhead is very big. Throughput is a little impacted on the less powerful machine.

2.4 - inside the container with etun and bridge
-----------------------------------------------

----------- -----------------------------------------------------------
| Netperf | CPU usage (%) / overhead | Throughput (Mbits/s) / changed |
----------- -----------------------------------------------------------
| on xeon | 18.39 / +207 % | 941.30 / 0 % |
-----------------------------------------------------------------------
| on athlon | 49.94 / +77 % | 823.75 / -2.5 % |
-----------------------------------------------------------------------

----------- ---------------------------------
| Tbench | Throughput (MBytes/s) / changed |
----------- ---------------------------------
| on xeon | 66.52 / +0.2 % |
---------------------------------------------
| on athlon | 61.07 / -6.8 % |
---------------------------------------------

Observation : The CPU overhead is very big. Throughput is a little impacted on the less powerful machine.

3. General observations
-----------------------

The objective to have no performances degrations, when the network namespace is off in the kernel, is reached in both solutions.

When the network is used outside the container and the network namespace are compiled in, there is no performance degradations.

Eric's patchset allows to move network devices between namespaces and this is clearly a good feature, missing in the Dmitry's patchset. This feature helps us to see that the network namespace code does not add overhead when using directly the physical network device into the container.

The loss of performances is very noticeable inside the container and seems to be directly related to the usage of the pair device and the specific network configuration needed for the container. When the packets are sent by the container, the mac address is for the pair device but the IP address is not owned by the host. That directly implies to have the host to act as a router and the packets to be forwarded. That adds a lot of overhead.

A hack has been made in the ip_forward function to avoid useless skb_cow when using the pair device/tunnel device and the overhead is reduced by the half.

Regards.

-- Daniel

_______________________________________________
Containers mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/containers

3. General observations
-----------------------

The objective to have no performances degrations, when the network namespace is off in the kernel, is reached in both solutions.

When the network is used outside the container and the network namespace are compiled in, there is no performance degradations.

Eric's patchset allows to move network devices between namespaces and this is clearly a good feature, missing in the Dmitry's patchset. This feature helps us to see that the network namespace code does not add overhead when using directly the physical network device into the container.

The loss of performances is very noticeable inside the container and seems to be directly related to the usage of the pair device and the specific network configuration needed for the container. When the packets are sent by the container, the mac address is for the pair device but the IP address is not owned by the host. That directly implies to have the host to act as a router and the packets to be forwarded. That adds a lot of overhead.

A hack has been made in the ip_forward function to avoid useless skb_cow when using the pair device/tunnel device and the overhead is reduced by the half.

Quoted 1 line · Daniel Lezcano

Hi,

Quoted 23 lines

3. General observations
-----------------------

The objective to have no performances degrations, when the network namespace is off in the kernel, is reached in both solutions.

When the network is used outside the container and the network namespace are compiled in, there is no performance degradations.

Eric's patchset allows to move network devices between namespaces and this is clearly a good feature, missing in the Dmitry's patchset. This feature helps us to see that the network namespace code does not add overhead when using directly the physical network device into the container.

The loss of performances is very noticeable inside the container and seems to be directly related to the usage of the pair device and the specific network configuration needed for the container. When the packets are sent by the container, the mac address is for the pair device but the IP address is not owned by the host. That directly implies to have the host to act as a router and the packets to be forwarded. That adds a lot of overhead.

Quoted 3 lines

A hack has been made in the ip_forward function to avoid useless skb_cow when using the pair device/tunnel device and the overhead is reduced by the half.

would it be possible to do some tests regarding scalability?

i.e. I would be interested how the following would look like:

10 connections on a single host (in parallel, overall performance) 10 connections from the same net space
10 connections from 10 different net spaces
(i.e. one connection from each space)

we can assume that L3 isolation will give similar results to the first case, but if needed, we can provide a patch to test this too ...

PS: great work! tx!

Quoted 7 lines

The loss of performances is very noticeable inside the container and seems to be directly related to the usage of the pair device and the specific network configuration needed for the container. When the packets are sent by the container, the mac address is for the pair device but the IP address is not owned by the host. That directly implies to have the host to act as a router and the packets to be forwarded. That adds a lot of overhead.

Well it adds measurable overhead.

Quoted 3 lines

A hack has been made in the ip_forward function to avoid useless skb_cow when using the pair device/tunnel device and the overhead is reduced by the half.

To be fully satisfactory how we get the packets to the namespace still appears to need work.

We have overhead in routing. That may simply be the cost of performing routing or there may be some optimizations opportunities there.

We have about the same overhead when performing bridging which I actually find more surprising, as the bridging code should involve less packet handling.

Ideally we can optimize the bridge code or something equivalent to it so that we can take one look at the destination mac address and know which network namespace we should be in. Potentially moving this work to hardware when the hardware supports multiple queues.

If we can get the overhead out of the routing code that would be tremendous. However I think it may be more realistic to get the overhead out of the ethernet bridging code where we know we don't need to modify the packet.

Quoted 6 lines · Daniel Lezcano

3. General observations
-----------------------

The objective to have no performances degrations, when the network namespace is off in the kernel, is reached in both solutions.

Quoted 8 lines

When the network is used outside the container and the network namespace are compiled in, there is no performance degradations.

Eric's patchset allows to move network devices between namespaces and this is clearly a good feature, missing in the Dmitry's patchset. This feature helps us to see that the network namespace code does not add overhead when using directly the physical network device into the container.

Assuming these results are not contradicted this says that the extra dereference where we need it does not add measurable to the overhead in the Linus network stack. Performance wise this should be good enough to allow merging the code into the linux kernel, as it does not measurably affect networking when we do not have multiple containers in use.

Things are good enough that we can even consider not providing an option to compile the support out.

Quoted 7 lines

The loss of performances is very noticeable inside the container and seems to be directly related to the usage of the pair device and the specific network configuration needed for the container. When the packets are sent by the container, the mac address is for the pair device but the IP address is not owned by the host. That directly implies to have the host to act as a router and the packets to be forwarded. That adds a lot of overhead.

Well it adds measurable overhead.

Quoted 3 lines

A hack has been made in the ip_forward function to avoid useless skb_cow when using the pair device/tunnel device and the overhead is reduced by the half.

To be fully satisfactory how we get the packets to the namespace still appears to need work.

We have overhead in routing. That may simply be the cost of performing routing or there may be some optimizations opportunities there.
We have about the same overhead when performing bridging which I actually find more surprising, as the bridging code should involve less packet handling.

Ideally we can optimize the bridge code or something equivalent to it so that we can take one look at the destination mac address and know which network namespace we should be in. Potentially moving this work to hardware when the hardware supports multiple queues.

If we can get the overhead out of the routing code that would be tremendous. However I think it may be more realistic to get the overhead out of the ethernet bridging code where we know we don't need to modify the packet.

Quoted 17 lines · Eric W. Biederman

Quoted 6 lines · Daniel Lezcano

3. General observations
-----------------------

The objective to have no performances degrations, when the network namespace is off in the kernel, is reached in both solutions.

Quoted 8 lines

When the network is used outside the container and the network namespace are compiled in, there is no performance degradations.

Eric's patchset allows to move network devices between namespaces and this is clearly a good feature, missing in the Dmitry's patchset. This feature helps us to see that the network namespace code does not add overhead when using directly the physical network device into the container.

Quoted 6 lines

Assuming these results are not contradicted this says that the extra dereference where we need it does not add measurable to the overhead in the Linus network stack. Performance wise this should be good enough to allow merging the code into the linux kernel, as it does not measurably affect networking when we do not have multiple containers in use.

I have a few questions about merging code into the linux kernel.

* How do you plan to do that ?

* When do you expect to have the network namespace into mainline ?

* Are Dave Miller and Alexey Kuznetov aware of the network namespace ?

* Did they saw your patchset or ever know it exists ?

* Do you have any feedbacks from netdev about the network namespace ?

Quoted 9 lines

To be fully satisfactory how we get the packets to the namespace still appears to need work.

We have overhead in routing. That may simply be the cost of performing routing or there may be some optimizations opportunities there.
We have about the same overhead when performing bridging which I actually find more surprising, as the bridging code should involve less packet handling.

Yep. I will try to figure out what is happening.

Quoted 5 lines

Ideally we can optimize the bridge code or something equivalent to it so that we can take one look at the destination mac address and know which network namespace we should be in. Potentially moving this work to hardware when the hardware supports multiple queues.

Quoted 4 lines

If we can get the overhead out of the routing code that would be tremendous. However I think it may be more realistic to get the overhead out of the ethernet bridging code where we know we don't need to modify the packet.

The routing was optimized for the loopback, no ? Why can't we do the same for the etun device ?

Quoted 4 lines

Ideally we can optimize the bridge code or something equivalent to it so that we can take one look at the destination mac address and know which network namespace we should be in. Potentially moving this work to hardware when the hardware supports multiple queues.

yes, we can hack the bridge, so that packets coming out of eth devices can go directly to the container and get out of veth devices from inside the container.

Quoted 4 lines

If we can get the overhead out of the routing code that would be tremendous. However I think it may be more realistic to get the overhead out of the ethernet bridging code where we know we don't need to modify the packet.

Why not optimize both? :)

If I read the results right it took a 32bit machine from AMD with a gigabit interface before you could measure a throughput difference. That isn't shabby for a non-optimized code path.

Quoted 3 lines

If I read the results right it took a 32bit machine from AMD with a gigabit interface before you could measure a throughput difference. That isn't shabby for a non-optimized code path.

Just some paranoid ramblings - one needs to look beyond just whether or not the performance of a bulk transfer test (eg TCP_STREAM) remains able to hit link-rate. One has to also consider the change in service demand (the normalization of CPU util and throughput). Also, with functionality like TSO in place, the ability to pass very large things down the stack can help cover for a multitude of path-length sins. And with either multiple 1G or 10G NICs becoming more and more prevalent, we have another one of those "NIC speed vs CPU speed" switch-overs, so maintaining single-NIC 1 gigabit throughput, while necessary, isn't (IMO) sufficient.

Soooo, it becomes very important to go beyond just TCP_STREAM tests when evaluating these sorts of things. Another test to run would be the TCP_RR test. TCP_RR with single-byte request/response sizes will "bypass" the TSO stuff, and the transaction rate will be more directly affected by the change in path length than a TCP_STREAM test. It will also show-up quite clearly in the service demand. Now, with NICs doing interrupt coalescing, if the NIC is strapped "poorly" (IMO) then you may not see a change in transaction rate - it may be getting limited artifically by the NIC's interrupt coalescing. So, one has to fall-back on service demand, or better yet, disable the interrupt coalescing.

Otherwise, measuring peak aggregate request/response becomes necessary.

rick jones
don't be blinded by bit-rate

Do you have any pointer to help on benchmarking the network, perhaps a checklist or some scripts for netperf ?

Daniel Lezcano <[email protected]> writes:

Quoted 1 line

* When do you expect to have the network namespace into mainline ?

My current goal is to finish my rebase against 2.6.linus_lastest in the next couple of days after having figured out how to deal with sysfs.

I have been doing reviewing in more code then I know what to do with, and fighting some very strange bugs during the stabilization window. Which has kept me from doing additional development. Plus I have had a cold.

If I read the results right it took a 32bit machine from AMD with a gigabit interface before you could measure a throughput difference. That isn't shabby for a non-optimized code path.

Quoted 1 line · Eric W. Biederman

Daniel Lezcano <[email protected]> writes:

[...]

Quoted 1 line

Quoted 1 line

* When do you expect to have the network namespace into mainline ?

Quoted 2 lines

My current goal is to finish my rebase against 2.6.linus_lastest in the next couple of days after having figured out how to deal with sysfs.

Great news!
I also have some questions about this updated version:

- Have you integrated the bug fixes and cleanups(*) Daniel wrote for your previous netns patchset (and the few glitches I reported too)?

(*) available in LXC8 patchset

- Do you already have a public git repository set up for the rebase?
- If it is private, any plan to make it public soon? (That would be great)

Quoted 4 lines

I have been doing reviewing in more code then I know what to do with, and fighting some very strange bugs during the stabilization window. Which has kept me from doing additional development. Plus I have had a cold.

I hope you're getting better... and you'll be able to provide us the updated patchset very soon :)

[...]

Quoted 3 lines

If I read the results right it took a 32bit machine from AMD with a gigabit interface before you could measure a throughput difference. That isn't shabby for a non-optimized code path.

Indeed the throughput difference is not significant.
This is very good to see that it stays constant when using the container. What I'm more worried about is the CPU load increase. But it seems we've identified some of the culprits.

This afternoon I had a look at why the bridge setup isn't better than the route setup (section 2.3 and 2.4 of Daniel's report).

In the bridge case, we encounter the same problems as the routes case.
The oprofile profile is the same: the most demanding routines are
pskb_expand_head and csum_partial_copy_generic. pskb_expand_head() is also called by skb_cow(), but this time skb_cow() is called by netfilter's nf_bridge_copy_header().

We can avoid this copy by removing option CONFIG_BRIDGE_NETFILTER. This copy is made even if netfilter is not used on the host. Maybe some optimizations can be made in netfilter's code to prevent this.