significant multi-core performance degradation with SR-IOV/macswap in some cases

https://bugs.dpdk.org/show_bug.cgi?id=188

Bug ID: 188
Summary: significant multi-core performance degradation with
SR-IOV/macswap in some cases
Product: DPDK
Version: 19.02
Hardware: x86
OS: Linux
Status: CONFIRMED
Severity: critical
Priority: Normal
Component: testpmd
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---

Hi,

I'm seeing a performance degradation with testpmd running with multiple cores. I saw the issue with ConnectX-3, ConnectX-4 Lx, and ConnectX-5.

### For ConnectX-3:
- With 2 Cores, I see a degradation from 19.2 mpps to 13.5 mpps.

testpmd command:
`MLX4_INLINE_RECV_SIZE=64 ./build/app/test-pmd/testpmd -c 0x7 -n 4 -w 0002:00:02.0 --vdev=net_vdev_netvsc,ignore=1 --socket-mem=2048 -- --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=256 --rxd=256 --mbcache=512 --rxq=2 --txq=2 --nb-cores=2 -i -a --forward-mode=macswap`

- With 4 Cores, I see a degradation from 19.2 mpps - 13.9 mpps.

testpmd command:
`MLX4_INLINE_RECV_SIZE=64 ./build/app/test-pmd/testpmd -c 0x1f -n 4 -w 0002:00:02.0 --vdev=net_vdev_netvsc,ignore=1 --socket-mem=2048 -- --port-numa-config=0,0 --socket-num=0 --burst=64 --txd=256 --rxd=256 --mbcache=512 --rxq=2 --txq=2 --nb-cores=4 -i -a --forward-mode=macswap`

### For ConnectX-4 Lx:

- With 2 Cores, I see a degradation from 32.9 mpps to 26.8 mpps.

testpmd command:
`./build/app/test-pmd/testpmd -c 0x7 -n 4 -w 0003:00:02.0 --vdev="net_vdev_netvsc,ignore=1" -- --burst=64 --txd=1024 --rxd=256 --mbcache=512 --rxq=2 --txq=2 --nb-cores=2 --rss-udp --port-topology=chained --forward-mode=macswap -i -a`

- With 4 Cores, I see a degradation from 34.8 mpps to 19 mpps.

testpmd command:
`./build/app/test-pmd/testpmd -c 0x1f -n 4 -w 0003:00:02.0 --vdev="net_vdev_netvsc,ignore=1" -- --burst=64 --txd=8192 --rxd=256 --mbcache=512 --rxq=4 --txq=4 --nb-cores=4 --rss-udp --port-topology=chained --forward-mode=macswap -i -a`

Server specs:
OS: Ubuntu 16.04.2 LTS
Kernel: 4.15.0-rc3

I'm using IXIA as a traffic generator, and to get the forwarding performance.

DPDK version: master:a89fbda

The cause of the degradation is the patch:

```
commit 62b52877adbe8d3ad19615b0827e7725e51feadc
Author: Qi Zhang <[email protected]>
Date:   Sun Dec 16 08:58:35 2018 +0800

    app/testpmd: batch MAC swap for performance on x86

    Do four packets macswap in same loop iterate to squeeze more
    CPU cycles.

    Signed-off-by: Qi Zhang <[email protected]>
    Reviewed-by: Ferruh Yigit <[email protected]>
    Acked-by: Bernard Iremonger <[email protected]>
```

You are receiving this mail because:
You are the assignee for the bug.