Content-Type: multipart/mixed; boundary="===============3213463133415542841==" MIME-Version: 1.0 From: nufosmatic at nufosmatic.com Subject: [SPDK] Using SPDK IOAT driver to drive data to PCIe space Date: Mon, 14 Jun 2021 10:11:24 +0000 Message-ID: <20210614101124.2820.734@ml01.vlan13.01.org> List-ID: To: spdk@lists.01.org --===============3213463133415542841== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Leaving aside for the moment that the IOAT example (build/examples/ioat_per= f) is using SPDK library calls that do not differentiate between the memory= that is used for managing a mempool and the data that serves as buffers in= a mempool, why would the SPDK IOAT perform very, very badly addressing me= mory in PCIe space. I have a device that I know will run 2-6 GB/s on a x8 interface from other = DMA engines in this system. And the IOAT example will run as fast as 5 GB/s= into the default DPDK memory reserves. And I know that the IOAT engine (a= t least the Crystal Beach IOAT engine on an Intel Cascade Lake SuperMicro X= 11DPG-QT with Sky Lake-E CBDMA) using another commercial middleware will ru= n as much as 9 GB/s in a x16 interface. But the IOAT example, modified to separate the memory management from the d= ata (see rte_mempool_empty() and rte_mempool_populate_virt()), runs <<0.1 = GB/s into x8 and x4 PCIe spaces. Is there some other feature that needs to = be enabled in the IOAT driver to allow it to drive to PCIe space fastly? --===============3213463133415542841==--