Skip to contentSkip to conversation

Registering other than NVMe device as a PCI memory buffer

SPDK rmbhagwat at gmail.com 1 participant 2 messages Sep 2, 2021

lore
Share & tools
lore

rmbhagwat at gmail.com

#0 lore raw

It looks like in SPDK we can register only handful of devices (nvme, ioat, idxd, virtio, vmd).
Is there any way to register Pure PCI Memory device (say 1 to 8 GB of DDR behind the PCIe)?
My intention is to register the PCI memory and use it as a data buffer which will be passed to either spdk_nvme_ctrlr_cmd_io_raw() command or spdk_nvme_ns_cmd_read() or spdk_nvme_ns_cmd_write(). Basically instead of using Host DDR memory, I wish to use memory on PCIe. While doing so, I do not intend to use CMB as my SSDs do not have CMB. Please let me know if there is any possibility in SPDK. If yes, what would be the procedure?
Thank you!

#1 lore raw
Quoted 6 lines

From: rmbhagwat(a)gmail.com <rmbhagwat(a)gmail.com>

It looks like in SPDK we can register only handful of devices (nvme, ioat, idxd, virtio, vmd).
Is there any way to register Pure PCI Memory device (say 1 to 8 GB of DDR behind the PCIe)?

Is this memory exposed as a PCI BAR? Is there a driver for this device? You have two options:

1) If you have a kernel driver for the device, have that kernel driver expose the memory region as a file descriptor that can be passed to mmap
2) Write a userspace driver against vfio-pci. https://www.kernel.org/doc/Documentation/vfio.txt. This is how SPDK works.

The drivers in SPDK are all userspace drivers written against the vfio interface in Linux (we actually make DPDK calls which then uses vfio-pci). You can probably copy SPDK's lib/ioat or lib/idxd as a good template for writing a simple user space driver. Once the BAR is mapped into the process address space, you can call spdk_mem_register() on the address to configure it for DMA access and use the virtual addresses as input to calls like spdk_nvme_ns_cmd_write().

Quoted 8 lines

My intention is to register the PCI memory and use it as a data buffer which will be passed to either spdk_nvme_ctrlr_cmd_io_raw() command or spdk_nvme_ns_cmd_read() or spdk_nvme_ns_cmd_write(). Basically instead of using Host DDR memory, I wish to use memory on PCIe. While doing so, I do not intend to use CMB as my SSDs do not have CMB.
Please let me know if there is any possibility in SPDK. If yes, what would be the procedure?
Thank you!