Skip to contentSkip to conversation

8% performance improved by change tap interact with kernel stack

DISCUSSION Qin Chuanyu 5 participants 14 messages Jan 28, 2014 – Feb 11, 2014

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host

	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps

8% performance gained from this change,
Is there any problem for this patch ?

#1 lore raw
Quoted 24 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host

	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps
Quoted 2 lines

8% performance gained from this change,
Is there any problem for this patch ?

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Signature · 1 line
MST
#2 lore raw
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host

	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps
Quoted 9 lines
Quoted 2 lines

8% performance gained from this change,
Is there any problem for this patch ?

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

#3 lore raw
Quoted 35 lines · Qin Chuanyu
Quoted 25 lines · Michael S. Tsirkin
Quoted 24 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host

	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps
Quoted 9 lines
Quoted 2 lines

8% performance gained from this change,
Is there any problem for this patch ?

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Quoted 9 lines

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

#4 lore raw
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Quoted 9 lines

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

#5 lore raw
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Quoted 9 lines

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Quoted 8 lines

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?

Thanks!

Signature · 1 line
MST
#11 lore raw
In reply to Qin Chuanyu
Quoted 5 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

Quoted 7 lines
	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

No idea why you use rcu here ?

Quoted 12 lines

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host

	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps
Quoted 2 lines

8% performance gained from this change,
Is there any problem for this patch ?

http://patchwork.ozlabs.org/patch/52963/

#10 lore raw
In reply to Qin Chuanyu
Quoted 10 lines · Qin Chuanyu
Quoted 2 lines · Michael S. Tsirkin

Could you please check latency results?

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Netperf UDP_RR can be very sensitive to packet losses. Not that there were necessarily any in your tests, but to further confirm the 3% improvement in latency, I would suggest using the confidence intervals functionality in your before/after netperf testing. And to get at what Michael asks about CPU utilization I would suggest:

netperf -H <otherguy> -c -C -l 30 -i 30,3 -t UDP_RR -- -r 512

(I was guessing as to what netperf options you may have been using already)

happy benchmarking,

rick jones

#6 lore raw
In reply to Michael S. Tsirkin
Quoted 35 lines · On Tue, 28 Jan 2014 12:33:25 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Quoted 9 lines

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Quoted 8 lines

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

Quoted 7 lines

Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?

Thanks!

There are a couple potential problems with this. The primary one is that now you are violating the explicit assumptions about when netif_receive_skb() can be called and because of that it may break things all over the place.

*
* netif_receive_skb() is the main receive data processing function.
* It always succeeds. The buffer may be dropped during processing
* for congestion control or by the protocol layers.
*
* This function may only be called from softirq context and interrupts
* should be enabled.

At a minimum, softirq (BH) and preempt must be disabled.

Another potential problem is that since a softirq is not used, the kernel stack maybe much larger.

Maybe a better way would be implementing some form of NAPI in the TUN device?

#7 lore raw
Quoted 43 lines · Stephen Hemminger
Quoted 40 lines · On Tue, 28 Jan 2014 12:33:25 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 25 lines · Qin Chuanyu
Quoted 15 lines · Michael S. Tsirkin
Quoted 6 lines
Quoted 6 lines

I think it's okay - IIUC this way we are processing xmit directly instead of going through softirq.
Was meaning to try this - I'm glad you are looking into this.

Could you please check latency results?

Quoted 9 lines

netperf UDP_RR 512
test model: VM->host->host

modified before : 11108
modified after : 11480

3% gained by this patch

Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Quoted 8 lines

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?

Quoted 6 lines
Quoted 2 lines

Thanks!

There are a couple potential problems with this. The primary one is that now you are violating the explicit assumptions about when netif_receive_skb() can be called and because of that it may break things all over the place.

Specifically http://patchwork.ozlabs.org/patch/52963/
mentions cls_cgroup_classify which has this code:

        if (in_serving_softirq()) {
                /* If there is an sk_classid we'll use that. */
                if (!skb->sk)
                        return -1;
                classid = skb->sk->sk_classid;
        }

in_serving_softirq now checks flag so we could thinkably set it just like softirq does.

Quoted 9 lines

*
* netif_receive_skb() is the main receive data processing function.
* It always succeeds. The buffer may be dropped during processing
* for congestion control or by the protocol layers.
*
* This function may only be called from softirq context and interrupts
* should be enabled.

At a minimum, softirq (BH) and preempt must be disabled.

Yes.

Quoted 2 lines

Another potential problem is that since a softirq is not used, the kernel stack maybe much larger.

tun itself is pretty modest in its stack use - as thread linked above says it might not be a big issue.

Quoted 2 lines

Maybe a better way would be implementing some form of NAPI in the TUN device?

We can't always do this.

regular devices get skbs from card or RAM so they can do this in softirq context.
tun gets skbs from userspace memory so it needs to run in process context, at least sometimes.

Signature · 1 line
MST
#12 lore raw
In reply to Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

Quoted 7 lines
	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();
Quoted 1 line

No idea why you use rcu here ?

In my first version, I forgot to add lock when called netif_receive_skb then I met a dad spinlock when using tcpdump.

tcpdump receive skb in netif_receive_skb but also in dev_queue_xmit.
and I have notice dev_queue_xmit add rcu_read_lock_bh before transmitting skb, and this lock avoid race between softirq and transmit thread.

	/* Disable soft irqs for various locks below. Also
	 * stops preemption for RCU.
	 */
	rcu_read_lock_bh();

Now I try to xmit skb in vhost thread, so I did the same thing.

#8 lore raw
In reply to Michael S. Tsirkin
Quoted 18 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Quoted 8 lines

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

Quoted 7 lines

Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?

Thanks!

because of spring festival of china, the test result would be given two week later. throughput would be test by netperf, and latency would be tested by qperf. Is that OK?

#9 lore raw
Quoted 25 lines · Qin Chuanyu
Quoted 23 lines · Michael S. Tsirkin
Quoted 9 lines
Quoted 9 lines

Nice.
What about CPU utilization?
It's trivially easy to speed up networking by burning up a lot of CPU so we must make sure it's not doing that.
And I think we should see some tests with TCP as well, and try several message sizes.

Quoted 8 lines

Yes, by burning up more CPU we could get better performance easily. So I have bond vhost thread and interrupt of nic on CPU1 while testing.

modified before, the idle of CPU1 is 0%-1% while testing. and after modify, the idle of CPU1 is 2%-3% while testing

TCP also could gain from this, but pps is less than UDP, so I think the improvement would be not so obviously.

Still need to test this doesn't regress but overall looks convincing to me. Could you send a patch, accompanied by testing results for throughput latency and cpu utilization for tcp and udp
with various message sizes?

Quoted 6 lines
Quoted 2 lines

Thanks!

because of spring festival of china, the test result would be given two week later.
throughput would be test by netperf, and latency would be tested by qperf. Is that OK?

For testing - sounds good. Run vmstat in host to check host cpu utilization. Pls don't forget to address all issues raised in this thread and in the old one Eric mentioned:
http://patchwork.ozlabs.org/patch/52963/
either address in code or address in commit log why it doesn't apply anymore.

Signature · 1 line
MST
#13 lore raw
In reply to Eric Dumazet
Quoted 25 lines · Eric Dumazet
Quoted 5 lines · Qin Chuanyu

according perf test result,I found that there are 5%-8% cpu cost on softirq by use netif_rx_ni called in tun_get_user.

so I changed the function which cause skb transmitted more quickly. from

Quoted 7 lines
	tun_get_user	->
		 netif_rx_ni(skb);
to
	tun_get_user	->
		rcu_read_lock_bh();
		netif_receive_skb(skb);
		rcu_read_unlock_bh();

No idea why you use rcu here ?

Quoted 9 lines

The test result is as below:

	CPU: Intel(R) Xeon(R) CPU E5620  @ 2.40GHz
	NIC: intel 82599
	Host OS/Guest OS:suse11sp3
	Qemu-1.6
	netperf udp 512(VM tx)
	test model: VM->host->host
Quoted 7 lines
Quoted 5 lines
	modified before : 2.00Gbps 461146pps
	modified after  : 2.16Gbps 498782pps

8% performance gained from this change,
Is there any problem for this patch ?

http://patchwork.ozlabs.org/patch/52963/

the problem list above show that:

Since the cgroup classifier has this check:

    if (softirq_count() != SOFTIRQ_OFFSET))
	   return -1;

We still fail to classify the frame.

but the source in the recently version has changed as below:

	if (in_serving_softirq()) {
		/* If there is an sk_classid we'll use that. */
		if (!skb->sk)
			return -1;
		classid = skb->sk->sk_classid;
	}

skb is allocated by tun_alloc_skb, so skb->sk is not NULL. I think the problem is not existed anymore.