8% performance improved by change tap interact with kernel stack
Subsystems (3)
according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.
so I changed the function which cause skb transmitted more quickly. from
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();
The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782pps
8% performance gained from this change,
Is there any problem for this patch ?
Quoted 24 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782pps
Quoted 2 lines
8% performance gained from this change,
Is there any problem for this patch ?
I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.
Could you please check latency results?
Signature · 1 line
MST
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782pps
Quoted 9 lines
Quoted 2 lines
8% performance gained from this change,
Is there any problem for this patch ?I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
netperf UDP_RR 512
test model: VM->host->host
modified before : 11108
modified after : 11480
3% gained by this patch
Quoted 35 lines · Qin Chuanyu
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782ppsQuoted 9 lines
Quoted 2 lines
8% performance gained from this change,
Is there any problem for this patch ?I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
Quoted 9 lines
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
Quoted 9 lines
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
Quoted 9 lines
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.
Quoted 8 lines
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?
Thanks!
Signature · 1 line
MST
Quoted 5 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
Quoted 7 lines
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();
No idea why you use rcu here ?
Quoted 12 lines
The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782pps
Quoted 2 lines
8% performance gained from this change,
Is there any problem for this patch ?
Quoted 10 lines · Qin Chuanyu
Quoted 2 lines · Michael S. Tsirkin
Could you please check latency results?
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Netperf UDP_RR can be very sensitive to packet losses. Not that there were necessarily any in your tests, but to further confirm the 3% improvement in latency, I would suggest using the confidence intervals functionality in your before/after netperf testing. And to get at what Michael asks about CPU utilization I would suggest:
netperf -H <otherguy> -c -C -l 30 -i 30,3 -t UDP_RR -- -r 512
(I was guessing as to what netperf options you may have been using already)
happy benchmarking,
rick jones
Quoted 35 lines · On Tue, 28 Jan 2014 12:33:25 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
Quoted 9 lines
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.Quoted 8 lines
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Quoted 7 lines
Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?Thanks!
There are a couple potential problems with this. The primary one is that now you are violating the explicit assumptions about when netif_receive_skb() can be called and because of that it may break things all over the place.
*
* netif_receive_skb() is the main receive data processing function.
* It always succeeds. The buffer may be dropped during processing
* for congestion control or by the protocol layers.
*
* This function may only be called from softirq context and interrupts
* should be enabled.
At a minimum, softirq (BH) and preempt must be disabled.
Another potential problem is that since a softirq is not used, the kernel stack maybe much larger.
Maybe a better way would be implementing some form of NAPI in the TUN device?
Quoted 43 lines · Stephen Hemminger
Quoted 40 lines · On Tue, 28 Jan 2014 12:33:25 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines
I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.Could you please check latency results?
Quoted 9 lines
netperf
UDP_RR512
test model:VM->host->hostmodified before : 11108
modified after : 114803% gained by this patch
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.Quoted 8 lines
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?
Quoted 6 lines
Quoted 2 lines
Thanks!
There are a couple potential problems with this. The primary one is that now you are violating the explicit assumptions about when
netif_receive_skb()can be called and because of that it may break things all over the place.
Specifically http://patchwork.ozlabs.org/patch/52963/
mentions cls_cgroup_classify which has this code:
if (in_serving_softirq()) {
/* If there is an sk_classid we'll use that. */
if (!skb->sk)
return -1;
classid = skb->sk->sk_classid;
}in_serving_softirq now checks flag so we could thinkably set it just like softirq does.
Quoted 9 lines
*
*netif_receive_skb()is the main receive data processing function.
* It always succeeds. The buffer may be dropped during processing
* for congestion control or by the protocol layers.
*
* This function may only be called from softirq context and interrupts
* should be enabled.At a minimum, softirq (BH) and preempt must be disabled.
Yes.
Quoted 2 lines
Another potential problem is that since a softirq is not used, the kernel stack maybe much larger.
tun itself is pretty modest in its stack use - as thread linked above says it might not be a big issue.
Quoted 2 lines
Maybe a better way would be implementing some form of NAPI in the TUN device?
We can't always do this.
regular devices get skbs from card or RAM so they can do this in softirq context.
tun gets skbs from userspace memory so it needs to run in process context, at least sometimes.
Signature · 1 line
MST
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
Quoted 7 lines
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();
Quoted 1 line
No idea why you use rcu here ?
In my first version, I forgot to add lock when called netif_receive_skb then I met a dad spinlock when using tcpdump.
tcpdump receive skb in netif_receive_skb but also in dev_queue_xmit.
and I have notice dev_queue_xmit add rcu_read_lock_bh before transmitting skb, and this lock avoid race between softirq and transmit thread.
/* Disable soft irqs for various locks below. Also * stops preemption for RCU. */ rcu_read_lock_bh();
Now I try to xmit skb in vhost thread, so I did the same thing.
Quoted 18 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.Quoted 8 lines
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Quoted 7 lines
Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?Thanks!
because of spring festival of china, the test result would be given two week later. throughput would be test by netperf, and latency would be tested by qperf. Is that OK?
Quoted 25 lines · Qin Chuanyu
Quoted 23 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines
Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.Quoted 8 lines
Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.
modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing
TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.
Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?
Quoted 6 lines
Quoted 2 lines
Thanks!
because of spring festival of china, the test result would be given two week later.
throughput would be test by netperf, and latency would be tested by qperf. Is that OK?
For testing - sounds good. Run vmstat in host to check host cpu utilization. Pls don't forget to address all issues raised in this thread and in the old one Eric mentioned:
http://patchwork.ozlabs.org/patch/52963/
either address in code or address in commit log why it doesn't apply anymore.
Signature · 1 line
MST
Quoted 25 lines · Eric Dumazet
Quoted 5 lines · Qin Chuanyu
according perf test result,I found that there are 5%-8% cpu cost on softirq by use
netif_rx_nicalled intun_get_user.so I changed the function which cause skb transmitted more quickly. from
Quoted 7 lines
tun_get_user -> netif_rx_ni(skb); to tun_get_user -> rcu_read_lock_bh(); netif_receive_skb(skb); rcu_read_unlock_bh();No idea why you use rcu here ?
Quoted 9 lines
The test result is as below:
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz NIC: intel 82599 Host OS/Guest OS:suse11sp3 Qemu-1.6 netperf udp 512(VM tx) test model: VM->host->host
Quoted 7 lines
Quoted 5 lines
modified before : 2.00Gbps 461146pps modified after : 2.16Gbps 498782pps8% performance gained from this change,
Is there any problem for this patch ?
the problem list above show that:
Since the cgroup classifier has this check:
if (softirq_count() != SOFTIRQ_OFFSET)) return -1;
We still fail to classify the frame.
but the source in the recently version has changed as below:
if (in_serving_softirq()) {
/* If there is an sk_classid we'll use that. */
if (!skb->sk)
return -1;
classid = skb->sk->sk_classid;
}skb is allocated by tun_alloc_skb, so skb->sk is not NULL. I think the problem is not existed anymore.