Using SPDK IOAT driver to drive data to PCIe space
nufosmatic at nufosmatic.com
Leaving aside for the moment that the IOAT example (build/examples/ioat_perf) is using SPDK library calls that do not differentiate between the memory that is used for managing a mempool and the data that serves as buffers in a mempool, why would the SPDK IOAT perform very, very badly addressing memory in PCIe space.
I have a device that I know will run 2-6 GB/s on a x8 interface from other DMA engines in this system. And the IOAT example will run as fast as 5 GB/s into the default DPDK memory reserves. And I know that the IOAT engine (at least the Crystal Beach IOAT engine on an Intel Cascade Lake SuperMicro X11DPG-QT with Sky Lake-E CBDMA) using another commercial middleware will run as much as 9 GB/s in a x16 interface.
But the IOAT example, modified to separate the memory management from the data (see rte_mempool_empty() and rte_mempool_populate_virt()), runs <<0.1 GB/s into x8 and x4 PCIe spaces. Is there some other feature that needs to be enabled in the IOAT driver to allow it to drive to PCIe space fastly?
Luse
Nothing that I can think of… maybe you can expand a little on what modifications you made? Slack might be a faster way to get some conversations going too but this works as well ☺
From: nufosmatic(a)nufosmatic.com <nufosmatic(a)nufosmatic.com>
Date: Monday, June 14, 2021 at 3:12 AM
To: spdk(a)lists.01.org <spdk(a)lists.01.org>
Subject: [SPDK] Using SPDK IOAT driver to drive data to PCIe space
Leaving aside for the moment that the IOAT example (build/examples/ioat_perf) is using SPDK library calls that do not differentiate between the memory that is used for managing a mempool and the data that serves as buffers in a mempool, why would the SPDK IOAT perform very, very badly addressing memory in PCIe space.
I have a device that I know will run 2-6 GB/s on a x8 interface from other DMA engines in this system. And the IOAT example will run as fast as 5 GB/s into the default DPDK memory reserves. And I know that the IOAT engine (at least the Crystal Beach IOAT engine on an Intel Cascade Lake SuperMicro X11DPG-QT with Sky Lake-E CBDMA) using another commercial middleware will run as much as 9 GB/s in a x16 interface.
But the IOAT example, modified to separate the memory management from the data (see rte_mempool_empty() and rte_mempool_populate_virt()), runs <<0.1 GB/s into x8 and x4 PCIe spaces. Is there some other feature that needs to be enabled in the IOAT driver to allow it to drive to PCIe space fastly?
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org