DPDK Performance issue with l2fwd

Hey Guys,

Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I got a strange pheromone in my test.

When I used 12 ports to test the performance of l2fwd, it can work fine and achieve 120G.
But it got abnormal when I using over than 12 port. Part of ports seems something wrong and no any Tx/Rx.
Has anyone know about this?

My testing Environment.
1. E5-2658 v2 (10 cores) * 2

http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-v2-25M-Cache-2_40-GHz
2. one core handle one port. (In order to get best performance.)
3. No any QPI crossing issue.
4. l2fwd parameters

     4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
     4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
work well.
     4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can

work well.

BTW, I have tried lots of parameter sets and if I set the ports number over than 12 ports, it only first 10 ports got work.
Else, everything got well.

Can anyone help me to solve the issue? Or DPDK only can set less equal than 12 ports?
Or DPDK max throughput is 120G?

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain confidential information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zachary.Jen@cas- well.com
Sent: Thursday, July 10, 2014 2:29 AM
To: [email protected]
Cc: [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Hi Alex:

Thanks for your help.

I forget to describe some criteria in my original post.

At first, I has confirmed my 82599 has connected by PCIe Gen3 (Gen3 x8) speed. The theoretical bandwidth can support over 160G in total.
Hence, It should get full speed in my test.

Second, I have ever check the performance w/o DPDK in packet size 1518 in the same environment, and indeed it can get 160G totally (by IRQ balance method). So, I was so surprised to get this kinds of result in DPDK (I also use size 1518 to test DPDK).

BTW, I can get 120G throughput in 12 ports already. But when I add more than 12 ports, I only can get 100G.
Why the performance gets less than 120G? Why only 10 ports works fine and NO Tx and Rx in the others?
Is it bugs or limitations in DPDK?

Has anyone every do the similar or the same test?

On 07/10/2014 04:40 PM, Alex Markuze wrote:
Hi Zachary,
Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to 128Gb/s[3].
Now, AFAIK[1] the CPU is connected to the I/O with a single PCI-E slot.

Several thoughts that may help you:

1. You can figure out the max b/w by running netsurf over the kernel interfaces (w/o DPDK). Each CPU can handle the Netperf and the Completion interrupts with grace (packets of 64K and all offloads on) for 10Gb nics. With more then 10 Nics I would disable the IRQ balancer and make sure interrupts are spread evenly by setting the IRQ affinity manually [2]. As long as you have a physical core(NO hyperthreading) per NIC port you can figure out the MAX B/W you can get with all the nics.

2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics (Mellanox), In this case for each Netperf flow you will need to separate each Netperf Stream and the interrupts to different Cores to Reach wire speed as long as both cores are on the same NUMA node(lscpu).

Hope this helps.

[1]http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revis ion_3.0.pdf [2]http://h50146.www5.hp.com/products/software/oe/linux/mainstream/supp ort/whitepaper/pdfs/4AA4-9294ENW.pdf [3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x

On Thu, Jul 10, 2014 at 11:07 AM, <Zachary.Jen@cas- well.com<mailto:[email protected]>> wrote:
Hey Guys,

Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I got a strange pheromone in my test.

When I used 12 ports to test the performance of l2fwd, it can work fine and achieve 120G.
But it got abnormal when I using over than 12 port. Part of ports seems something wrong and no any Tx/Rx.
Has anyone know about this?

My testing Environment.
1. E5-2658 v2 (10 cores) * 2
http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-v2-
25M-Cache-2_40-GHz
2. one core handle one port. (In order to get best performance.)
3. No any QPI crossing issue.
4. l2fwd parameters

     4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
     4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
work well.
     4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can

work well.

BTW, I have tried lots of parameter sets and if I set the ports number over than 12 ports, it only first 10 ports got work.
Else, everything got well.

Can anyone help me to solve the issue? Or DPDK only can set less equal than 12 ports?
Or DPDK max throughput is 120G?

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

--
Best Regards,
Zachary Jen

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

Hi,

Have you tried running a test with 16 ports using any other applications, for example testpmd?

Regards,
/Bruce

Quoted 117 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zachary.Jen@cas- well.com
Sent: Thursday, July 10, 2014 2:29 AM
To: [email protected]
Cc: [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Hi Alex:

Thanks for your help.

I forget to describe some criteria in my original post.

At first, I has confirmed my 82599 has connected by PCIe Gen3 (Gen3 x8) speed. The theoretical bandwidth can support over 160G in total.
Hence, It should get full speed in my test.

Second, I have ever check the performance w/o DPDK in packet size 1518 in the same environment, and indeed it can get 160G totally (by IRQ balance method). So, I was so surprised to get this kinds of result in DPDK (I also use size 1518 to test DPDK).

BTW, I can get 120G throughput in 12 ports already. But when I add more than 12 ports, I only can get 100G.
Why the performance gets less than 120G? Why only 10 ports works fine and NO Tx and Rx in the others?
Is it bugs or limitations in DPDK?

Has anyone every do the similar or the same test?

On 07/10/2014 04:40 PM, Alex Markuze wrote:
Hi Zachary,
Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to 128Gb/s[3].
Now, AFAIK[1] the CPU is connected to the I/O with a single PCI-E slot.

Several thoughts that may help you:

1. You can figure out the max b/w by running netsurf over the kernel interfaces (w/o DPDK). Each CPU can handle the Netperf and the Completion interrupts with grace (packets of 64K and all offloads on) for 10Gb nics. With more then 10 Nics I would disable the IRQ balancer and make sure interrupts are spread evenly by setting the IRQ affinity manually [2]. As long as you have a physical core(NO hyperthreading) per NIC port you can figure out the MAX B/W you can get with all the nics.

2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics (Mellanox), In this case for each Netperf flow you will need to separate each Netperf Stream and the interrupts to different Cores to Reach wire speed as long as both cores are on the same NUMA node(lscpu).

Hope this helps.

[1]http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revis ion_3.0.pdf [2]http://h50146.www5.hp.com/products/software/oe/linux/mainstream/supp ort/whitepaper/pdfs/4AA4-9294ENW.pdf [3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x

On Thu, Jul 10, 2014 at 11:07 AM, <Zachary.Jen@cas- well.com<mailto:[email protected]>> wrote:
Hey Guys,

Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I got a strange pheromone in my test.

When I used 12 ports to test the performance of l2fwd, it can work fine and achieve 120G.
But it got abnormal when I using over than 12 port. Part of ports seems something wrong and no any Tx/Rx.
Has anyone know about this?

My testing Environment.
1. E5-2658 v2 (10 cores) * 2
http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-v2-
25M-Cache-2_40-GHz
2. one core handle one port. (In order to get best performance.)
3. No any QPI crossing issue.
4. l2fwd parameters

      4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
      4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
work well.
      4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can

work well.

BTW, I have tried lots of parameter sets and if I set the ports number over than 12 ports, it only first 10 ports got work.
Else, everything got well.

Can anyone help me to solve the issue? Or DPDK only can set less equal than 12 ports?
Or DPDK max throughput is 120G?

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

--
Best Regards,
Zachary Jen

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Friday, July 11, 2014 4:05 AM
To: [email protected]
Cc: Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Hi Bruce:

Thanks for your suggestion.
I have tried to use testpmd to test 16 ports today.
The result is so Interested. It can work, although some ports get low performance (only get 80%).

Besides, I also do another test in l2fwd.
I tried to use 82580 * 16 ports in the same platform with l2fwd test. The same situation happened again.
So, it seems a big bug hidden in the l2fwd.

Have someone get the similar case?

BTW, may this issue relate with DPDK version?

Quoted 3 lines · Bruce

Hi,

Have you tried running a test with 16 ports using any other applications, for

example testpmd?

Quoted 19 lines

Regards,
/Bruce

Quoted 15 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zachary.Jen@cas- well.com
Sent: Thursday, July 10, 2014 2:29 AM
To: [email protected]
Cc: [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Hi Alex:

Thanks for your help.

I forget to describe some criteria in my original post.

At first, I has confirmed my 82599 has connected by PCIe Gen3 (Gen3 x8)

speed.

Quoted 4 lines

Quoted 4 lines

The theoretical bandwidth can support over 160G in total. Hence, It should get full speed in my test.

Second, I have ever check the performance w/o DPDK in packet size 1518 in

the

Quoted 1 line

Quoted 1 line

same environment, and indeed it can get 160G totally (by IRQ balance

method).

Quoted 1 line

Quoted 1 line

So, I was so surprised to get this kinds of result in DPDK (I also use size 1518

to

Quoted 3 lines

Quoted 3 lines

test DPDK).

BTW, I can get 120G throughput in 12 ports already. But when I add more

than

Quoted 2 lines

Quoted 2 lines

12 ports, I only can get 100G.
Why the performance gets less than 120G? Why only 10 ports works fine and

NO

Quoted 12 lines

Quoted 12 lines

Tx and Rx in the others?
Is it bugs or limitations in DPDK?

Has anyone every do the similar or the same test?

On 07/10/2014 04:40 PM, Alex Markuze wrote:
Hi Zachary,
Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to 128Gb/s[3].
Now, AFAIK[1] the CPU is connected to the I/O with a single PCI-E slot.

Quoted 3 lines

Quoted 3 lines

Several thoughts that may help you:

1. You can figure out the max b/w by running netsurf over the kernel

interfaces

Quoted 7 lines

Quoted 7 lines

(w/o DPDK). Each CPU can handle the Netperf and the Completion interrupts with grace (packets of 64K and all offloads on) for 10Gb nics. With more then 10 Nics I would disable the IRQ balancer and make sure interrupts are spread evenly by setting the IRQ affinity manually [2]. As long as you have a physical core(NO hyperthreading) per NIC port you can figure out the MAX B/W you can get with all the nics.

Quoted 8 lines

Quoted 8 lines

2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics (Mellanox), In this case for each Netperf flow you will need to separate each Netperf Stream and the interrupts to different Cores to Reach wire speed as long as both cores are on the same NUMA node(lscpu).

Hope this helps.

[1]http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revis

Quoted 2 lines

Quoted 2 lines

ion_3.0.pdf

[2]http://h50146.www5.hp.com/products/software/oe/linux/mainstream/supp

Quoted 17 lines

Quoted 17 lines

ort/whitepaper/pdfs/4AA4-9294ENW.pdf [3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x

On Thu, Jul 10, 2014 at 11:07 AM, <Zachary.Jen@cas- well.com<mailto:[email protected]>> wrote:
Hey Guys,

Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I got a strange pheromone in my test.

When I used 12 ports to test the performance of l2fwd, it can work fine and achieve 120G.
But it got abnormal when I using over than 12 port. Part of ports seems something wrong and no any Tx/Rx.
Has anyone know about this?

Quoted 3 lines

Quoted 3 lines

My testing Environment.
1. E5-2658 v2 (10 cores) * 2
http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-

v2-

Quoted 14 lines

Quoted 14 lines

25M-Cache-2_40-GHz
2. one core handle one port. (In order to get best performance.)
3. No any QPI crossing issue.
4. l2fwd parameters

      4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
      4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
work well.
      4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can

work well.

BTW, I have tried lots of parameter sets and if I set the ports number over than 12 ports, it only first 10 ports got work.
Else, everything got well.

Quoted 5 lines

Quoted 5 lines

Can anyone help me to solve the issue? Or DPDK only can set less equal than 12 ports?
Or DPDK max throughput is 120G?

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露

本

Quoted 2 lines

Quoted 2 lines

信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you

are

Quoted 8 lines

Quoted 8 lines

not the intended recipient.

--
Best Regards,
Zachary Jen

Quoted 7 lines

Quoted 7 lines

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露

本

Quoted 2 lines

Quoted 2 lines

信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you

are

Quoted 1 line

Quoted 1 line

not the intended recipient.

--
Best Regards,
Zachary Jen

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988
本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.

Quoted 8 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Friday, July 11, 2014 4:05 AM
To: [email protected]
Cc: Richardson, Bruce; [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Quoted 6 lines

Hi Bruce:

Thanks for your suggestion.
I have tried to use testpmd to test 16 ports today.
The result is so Interested. It can work, although some ports get low performance (only get 80%).

For small packets, PCI bandwidth can be an issue. Do you get 80% of line rate with 64-byte packets or with larger packet sizes? How many testpmd cores were you using, and what parameters were you setting in testpmd?

Quoted 4 lines

Besides, I also do another test in l2fwd.
I tried to use 82580 * 16 ports in the same platform with l2fwd test. The same situation happened again.
So, it seems a big bug hidden in the l2fwd.

From what you report, that does indeed seem to be the case. We'll have to take another look at the code to see if there is an issue there.

/Bruce

Quoted 157 lines

Have someone get the similar case?

BTW, may this issue relate with DPDK version?

Quoted 3 lines · Bruce

Hi,

Have you tried running a test with 16 ports using any other applications, for

example testpmd?

Quoted 18 lines

Regards,
/Bruce

Quoted 15 lines

-----Original Message-----
From: dev [mailto:[email protected]] On Behalf Of Zachary.Jen@cas- well.com
Sent: Thursday, July 10, 2014 2:29 AM
To: [email protected]
Cc: [email protected]
Subject: Re: [dpdk-dev] DPDK Performance issue with l2fwd

Hi Alex:

Thanks for your help.

I forget to describe some criteria in my original post.

At first, I has confirmed my 82599 has connected by PCIe Gen3 (Gen3 x8)

speed.

Quoted 4 lines

Quoted 4 lines

The theoretical bandwidth can support over 160G in total. Hence, It should get full speed in my test.

Second, I have ever check the performance w/o DPDK in packet size 1518 in

the

Quoted 1 line

Quoted 1 line

same environment, and indeed it can get 160G totally (by IRQ balance

method).

Quoted 1 line

Quoted 1 line

So, I was so surprised to get this kinds of result in DPDK (I also use size 1518

to

Quoted 3 lines

Quoted 3 lines

test DPDK).

BTW, I can get 120G throughput in 12 ports already. But when I add more

than

Quoted 2 lines

Quoted 2 lines

12 ports, I only can get 100G.
Why the performance gets less than 120G? Why only 10 ports works fine and

NO

Quoted 12 lines

Quoted 12 lines

Tx and Rx in the others?
Is it bugs or limitations in DPDK?

Has anyone every do the similar or the same test?

On 07/10/2014 04:40 PM, Alex Markuze wrote:
Hi Zachary,
Your issue may be with the PCI-e 3, with 16 lanes Each slot is limited to 128Gb/s[3].
Now, AFAIK[1] the CPU is connected to the I/O with a single PCI-E slot.

Quoted 3 lines

Quoted 3 lines

Several thoughts that may help you:

1. You can figure out the max b/w by running netsurf over the kernel

interfaces

Quoted 7 lines

Quoted 7 lines

(w/o DPDK). Each CPU can handle the Netperf and the Completion interrupts with grace (packets of 64K and all offloads on) for 10Gb nics. With more then 10 Nics I would disable the IRQ balancer and make sure interrupts are spread evenly by setting the IRQ affinity manually [2]. As long as you have a physical core(NO hyperthreading) per NIC port you can figure out the MAX B/W you can get with all the nics.

Quoted 8 lines

Quoted 8 lines

2. You can try using (If available to you , obviously) 40Gb and 56Gb Nics (Mellanox), In this case for each Netperf flow you will need to separate each Netperf Stream and the interrupts to different Cores to Reach wire speed as long as both cores are on the same NUMA node(lscpu).

Hope this helps.

[1]http://komposter.com.ua/documents/PCI_Express_Base_Specification_Revis

Quoted 2 lines

Quoted 2 lines

ion_3.0.pdf

[2]http://h50146.www5.hp.com/products/software/oe/linux/mainstream/supp

Quoted 17 lines

Quoted 17 lines

ort/whitepaper/pdfs/4AA4-9294ENW.pdf [3]http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_3.x

On Thu, Jul 10, 2014 at 11:07 AM, <Zachary.Jen@cas- well.com<mailto:[email protected]>> wrote:
Hey Guys,

Recently, I have used l2fwd to test 160G (82599 10G * 16 ports), but I got a strange pheromone in my test.

When I used 12 ports to test the performance of l2fwd, it can work fine and achieve 120G.
But it got abnormal when I using over than 12 port. Part of ports seems something wrong and no any Tx/Rx.
Has anyone know about this?

Quoted 3 lines

Quoted 3 lines

My testing Environment.
1. E5-2658 v2 (10 cores) * 2
http://ark.intel.com/zh-tw/products/76160/Intel-Xeon-Processor-E5-2658-

v2-

Quoted 14 lines

Quoted 14 lines

25M-Cache-2_40-GHz
2. one core handle one port. (In order to get best performance.)
3. No any QPI crossing issue.
4. l2fwd parameters

       4.1 -c 0xF0FF -- -P 0xF00FF  => 120G get!
       4.2 -c 0xFF0FF -- -P 0xFF0FF => Failed! Only first 10 ports can
work well.
       4.3 -c 0x3F3FF -- -P 0x3F3FF => Failed! Only first 10 ports can

work well.

BTW, I have tried lots of parameter sets and if I set the ports number over than 12 ports, it only first 10 ports got work.
Else, everything got well.

Quoted 5 lines

Quoted 5 lines

Can anyone help me to solve the issue? Or DPDK only can set less equal than 12 ports?
Or DPDK max throughput is 120G?

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露

本

Quoted 2 lines

Quoted 2 lines

信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you

are

Quoted 8 lines

Quoted 8 lines

not the intended recipient.

--
Best Regards,
Zachary Jen

Quoted 7 lines

Quoted 7 lines

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988

本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露

本

Quoted 2 lines

Quoted 2 lines

信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you

are

Quoted 1 line

Quoted 1 line

not the intended recipient.

--
Best Regards,
Zachary Jen

Software RD
CAS-WELL Inc.
8th Floor, No. 242, Bo-Ai St., Shu-Lin City, Taipei County 238, Taiwan
Tel: +886-2-7705-8888#6305
Fax: +886-2-7731-9988
本信件可能包含瑞祺電通機密資訊，非指定之收件者，請勿使用或揭露本
信件內容，並請銷毀此信件。 This email may contain confidential
information. Please do not use or disclose it in any way and delete it if you are not the intended recipient.