Registering other than NVMe device as a PCI memory buffer
rmbhagwat at gmail.com
It looks like in SPDK we can register only handful of devices (nvme, ioat, idxd, virtio, vmd).
Is there any way to register Pure PCI Memory device (say 1 to 8 GB of DDR behind the PCIe)?
My intention is to register the PCI memory and use it as a data buffer which will be passed to either spdk_nvme_ctrlr_cmd_io_raw() command or spdk_nvme_ns_cmd_read() or spdk_nvme_ns_cmd_write(). Basically instead of using Host DDR memory, I wish to use memory on PCIe. While doing so, I do not intend to use CMB as my SSDs do not have CMB. Please let me know if there is any possibility in SPDK. If yes, what would be the procedure?
Thank you!
Walker
Quoted 6 lines
From: rmbhagwat(a)gmail.com <rmbhagwat(a)gmail.com>
It looks like in SPDK we can register only handful of devices (nvme, ioat, idxd, virtio, vmd).
Is there any way to register Pure PCI Memory device (say 1 to 8 GB of DDR behind the PCIe)?
Is this memory exposed as a PCI BAR? Is there a driver for this device? You have two options:
1) If you have a kernel driver for the device, have that kernel driver expose the memory region as a file descriptor that can be passed to mmap
2) Write a userspace driver against vfio-pci. https://www.kernel.org/doc/Documentation/vfio.txt. This is how SPDK works.
The drivers in SPDK are all userspace drivers written against the vfio interface in Linux (we actually make DPDK calls which then uses vfio-pci). You can probably copy SPDK's lib/ioat or lib/idxd as a good template for writing a simple user space driver. Once the BAR is mapped into the process address space, you can call spdk_mem_register() on the address to configure it for DMA access and use the virtual addresses as input to calls like spdk_nvme_ns_cmd_write().
Quoted 8 lines
My intention is to register the PCI memory and use it as a data buffer which will be passed to either
spdk_nvme_ctrlr_cmd_io_raw()command orspdk_nvme_ns_cmd_read()orspdk_nvme_ns_cmd_write(). Basically instead of using Host DDR memory, I wish to use memory on PCIe. While doing so, I do not intend to use CMB as my SSDs do not have CMB.
Please let me know if there is any possibility in SPDK. If yes, what would be the procedure?
Thank you!