ConnectX6 / mlx5 DPDK - bad RSS/ rte_flow performance on mixed traffic ( rxq_cqe_comp_en=4 )

sudo ./dpdk-testpmd -n 8 -l 4,6,8,10,12,14,16,18,20 -a > 0000:4b:00.0,rxq_cqe_comp_en=4 -a 0000:4b:00.1,rxq_cqe_comp_en=4 -- --forward-> mode=mac --rxq=8 --txq=8 --nb-cores=8 --numa -i -a --disable-rss

flow create 0 ingress pattern eth / ipv4 / tcp / end actions rss types ipv4-tcp > end queues 0 1 2 3 4 5 6 7 end key > 6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A / > end

flow create 0 ingress pattern eth / ipv4 / udp / end actions rss types ipv4-udp > end queues 0 1 2 3 4 5 6 7 end key > 6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A / > end

flow create 1 ingress pattern eth / ipv4 / tcp / end actions rss types ipv4-tcp > end queues 0 1 2 3 4 5 6 7 end key > 6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A / > end

flow create 1 ingress pattern eth / ipv4 / udp / end actions rss types ipv4-udp > end queues 0 1 2 3 4 5 6 7 end key > 6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A / > end

{code}
ConnectX6DX

<Devices>

    <Device pciName="0000:4b:00.0" type="ConnectX6DX" psid="DEL0000000027" >
    partNumber="0F6FXM_08P2T2_Ax">
      <Versions>
        <FW current="22.31.1014" available="N/A"/>
        <PXE current="3.6.0403" available="N/A"/>
        <UEFI current="14.24.0013" available="N/A"/>
     </Versions>

{code}

-----Original Message-----
From: [email protected] <[email protected]>
Sent: Friday, July 15, 2022 7:07 PM
To: [email protected]
Subject: [Bug 1053] ConnectX6 / mlx5 DPDK - bad RSS/ rte_flow performance on mixed traffic ( rxq_cqe_comp_en=4 )

https://bugs.dpdk.org/show_bug.cgi?id=1053

Bug ID: 1053
Summary: ConnectX6 / mlx5 DPDK - bad RSS/ rte_flow performance
on mixed traffic ( rxq_cqe_comp_en=4 )
Product: DPDK
Version: 21.11
Hardware: x86
OS: Linux
Status: UNCONFIRMED
Severity: normal
Priority: Normal
Component: ethdev
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---

Our team has been chasing major performance issues with ConnectX6 cards.

*Customer challenge:*
Flow stable ( symmetric RSS) load-balancing of flows to 8 worker lcores.

*Observation:*
Performance is fine up to 100Gbps using either tcp *or* udp-only traffic profiles.
Mixed traffic drops down to 50% loss with all packets showing up as xstats:
rx_phy_discard_packets

card infos at end of email.

There appears to be a huge performance issue on mixed UDP/TCP using symmetric load-balancing accross multiple workers.
E.g. compiling a DPDK v20.11 or newer test-pmd apps:

Quoted 4 lines

sudo ./dpdk-testpmd -n 8 -l 4,6,8,10,12,14,16,18,20 -a > 0000:4b:00.0,rxq_cqe_comp_en=4 -a 0000:4b:00.1,rxq_cqe_comp_en=4 -- --forward-> mode=mac --rxq=8 --txq=8 --nb-cores=8 --numa -i -a --forward-> --disable-rss

and configuring:

Quoted 3 lines

flow create 0 ingress pattern eth / ipv4 / tcp / end actions rss types ipv4-tcp > end queues 0 1 2 3 4 5 6 7 end key >

6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5 A6D5A6D5A6D

Quoted 6 lines

5A6D5A6D5A
/ > end

flow create 0 ingress pattern eth / ipv4 / udp / end actions rss types ipv4-udp > end queues 0 1 2 3 4 5 6 7 end key >

6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5 A6D5A6D5A6D

Quoted 6 lines

5A6D5A6D5A
/ > end

flow create 1 ingress pattern eth / ipv4 / tcp / end actions rss types ipv4-tcp > end queues 0 1 2 3 4 5 6 7 end key >

6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5 A6D5A6D5A6D

Quoted 6 lines

5A6D5A6D5A
/ > end

flow create 1 ingress pattern eth / ipv4 / udp / end actions rss types ipv4-udp > end queues 0 1 2 3 4 5 6 7 end key >

6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5A6D5 A6D5A6D5A6D

Quoted 2 lines

5A6D5A6D5A
/ > end

will see *significant* packet drops at a load > 50 Gbps on any type of mixed UDP/TCP traffic. E.g.

https://github.com/cisco-system-traffic-generator/trex-
core/blob/master/scripts/cap2/sfr3.yaml
Whenever those packet drops occur, I see those in the xstats as "rx_phy_discard_packets"

On the other hand using a TCP-or UDP-only traffic profile perfectly scales up to 100Gbps w/o drops.

Thanks for your help!

Quoted 7 lines

{code}
ConnectX6DX

<Devices>

    <Device pciName="0000:4b:00.0" type="ConnectX6DX"

psid="DEL0000000027" >

Quoted 7 lines

    partNumber="0F6FXM_08P2T2_Ax">
      <Versions>
        <FW current="22.31.1014" available="N/A"/>
        <PXE current="3.6.0403" available="N/A"/>
        <UEFI current="14.24.0013" available="N/A"/>
     </Versions>

{code}

--
You are receiving this mail because:
You are the assignee for the bug.