Skip to contentSkip to conversation

128k sequential read / write (fio) performance with spdk+vpp is not as good as the one with "kernel TCP"

SPDK sejun.kwon at samsung.com 2 participants 2 messages Feb 14, 2020

lore
Share & tools
lore

sejun.kwon at samsung.com

#0 lore raw

Hello,
I'm working on SPDK library + VPP, because some report said that VPP reduces the overhead of network.
When I test with VPP (with mlx5 poll mode driver) and null device with spdk, 4k performance with VPP is much better than the default(kTCP). But, 128k write performance with VPP is 30 percent lower than the one with kTCP.
The configuration when VPP starts is as below.
Is there anyone why 128k write performance with VPP is not good as kernel ?

I increase num-rx-desc and there is improvment, but it still lower than kernel. Thanks in advance.

unix {
        nodaemon
        cli-listen /run/vpp/cli.sock

        # cli-listen localhost:5002

          full-coredump

}
session {
evt_qs_memfd_seg
}

dpdk {
        dev 0000:03:00.0
        {
        }

}
socksvr {
# socket-name /tmp/vpp-api.sock
socket-name /run/vpp-api.sock
}

plugins {
        plugin default { disable }
        plugin dpdk_plugin.so { enable }
}
#1 lore raw

When I tuned vpp for our NIC's I found that the default vpp configuration wasn't sufficient. Doesn't look like you're controlling the worker threads. When vpp uses dpdk unless you control which cores the workers use they'll compete with spdk cores and I found letting them compete doesn't work well. Instead, assign vpp to dedicated cores and control the spdk coremask so that it doesn't use the vpp cores. I matched the vpp worker threads with a number of RSS queues from the NIC. Other hindrances to performance were multi-seg and tx checksum offload. You want to use tx cksum offload because the s/w overhead is very high. This will depend on your h/w but are things to look at. With this config I was able to get ~800k iops for 4k reads (to nvme not null). Can't remember the numbers for other sized reads/writes. It wasn't great but the kernel was worse.

cpu {
## Set logical CPU core where main thread runs, if main core is not set ## VPP will use core 1 if available
main-core 0

## Set logical CPU core(s) where worker threads are running

  # corelist-workers 2-3,18-19
  corelist-workers 1-3

## Specify a number of workers to be created
## Workers are pinned to N consecutive CPU cores while skipping
"skip-cores" CPU core(s)
## and main thread's CPU core
workers 3

}

dpdk {
dev default {
## Number of receive queues, enables RSS
## Default is 1
num-rx-queues 3
...
}
## Disable multi-segment buffers, improves performance but ## disables Jumbo MTU support
no-multi-seg

## Disables UDP / TCP TX checksum offload. Typically needed for use ## faster vector PMDs (together with no-multi-seg) #no-tx-checksum-offload

}

Quoted 41 lines · On Thu, Feb 13, 2020 at 7:57 PM <sejun.kwon(a)samsung.com> wrote:

Hello,
I'm working on SPDK library + VPP, because some report said that VPP reduces the overhead of network.
When I test with VPP (with mlx5 poll mode driver) and null device with spdk, 4k performance with VPP is much better than the default(kTCP). But, 128k write performance with VPP is 30 percent lower than the one with kTCP.
The configuration when VPP starts is as below.
Is there anyone why 128k write performance with VPP is not good as kernel ?

I increase num-rx-desc and there is improvment, but it still lower than kernel. Thanks in advance.

unix {
        nodaemon
        cli-listen /run/vpp/cli.sock

        # cli-listen localhost:5002

          full-coredump

}
session {
evt_qs_memfd_seg
}

dpdk {
        dev 0000:03:00.0
        {
        }

}
socksvr {
# socket-name /tmp/vpp-api.sock
socket-name /run/vpp-api.sock
}

plugins {
        plugin default { disable }
        plugin dpdk_plugin.so { enable }
}

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org