Skip to contentSkip to conversation

nvme-ioctl: propagate PRP1 from ioctl to admin cmd

RFC PATCH In review David Epping 4 participants 13 messages Jun 22 – Jun 29 NVMe

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 nvme-ioctl: propagate PRP1 from ioctl to admin cmd

Hello,

some NVMe admin commands (like IO queue creation) require information to be provided via the PRP1 field of the command.
This information has not previously been passed from the ioctl to the NVMe command structure, which prevented execution of such commands from userspace. Likely this is not an oversight but intentional because of security concerns?

With this email I'm seeking to see if there is interest in getting support for userspace NVMe IO queue creation in the upstream driver.
The attached patch is one way to do it and has been used for multiple years with multiple Kernel versions now. There are additional patches required to for example allocate sufficiently many IO queues at initialization time, but I would like to focus on the ability to create queues for now.
A final patch would also not ignore nvme_user_cmd64().

The system setup where this patch has been used is as follows:
- P2P PCIe capable CPU (currently also IOMMU disabled)
- patched Linux in-Kernel NVMe driver for local PCIe NVMe SSDs
- FPGA accelerator implementing NVMe IO queue memory and IO queue handling, exposed via PCIe BAR
- vfio-pcie Kernel driver plus vfio userspace FPGA driver / application
- The userspace application creates new NVMe IO queues at the SSD using the patched admin ioctl and points them towards the FPGA BAR. It then informs the FPGA about the SSD BAR address and IO queue ID. From then on the FPGA can access the SSD storage entirely without software interaction.

Since the omition of PRP1 access from userspace is likely intentional, maybe the discussion and patches by Leon Romanovsky for making dmabuf and p2pdma available via vfio lead in the right direction:
https://lore.kernel.org/all/[email protected]/

It currently seems focused on SPDK for handling the SSDs in userspace, but the author also describes it as a general mechanism that can support other scenarios.
How about having the in-Kernel NVMe driver accept a dmabuf as IO queue location for creating a new IO queue? And in turn it likely has to provide a small dmabuf to the FPGA VFIO world for access to the queue doorbells on the SSD.

Looking forward to your feedback,
David

---
 drivers/nvme/host/ioctl.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/host/ioctl.c +1 -0
index ca86d3bf7ea4..d5d740f1b554 100644
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);
 	c.common.cdw10 = cpu_to_le32(cmd.cdw10);
 	c.common.cdw11 = cpu_to_le32(cmd.cdw11);
 	c.common.cdw12 = cpu_to_le32(cmd.cdw12);
Signature · 1 line
2.43.0
maintainer
#1 lore raw

drivers/nvme/host/ioctl.c static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 309

Quoted 5 lines · David Epping
drivers/nvme/host/ioctl.c +1 -0
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);

This is not correct: the user space virtual address isn't the device DMA'able address. The driver already handles mapping the user address to kernel space, then to dma, then sets the PRP accordingly.

#2 lore raw
Quoted 10 lines · Keith Busch

drivers/nvme/host/ioctl.c static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 309

Quoted 5 lines · David Epping
drivers/nvme/host/ioctl.c +1 -0
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);

This is not correct: the user space virtual address isn't the device DMA'able address. The driver already handles mapping the user address to kernel space, then to dma, then sets the PRP accordingly.

To clarify, the ioctl struct addr field is not filled with a memory buffer address by the userspace, but a PCIe mapped BAR address plus offset. It is obtained by the userspace application operating the FPGA vfio device by reading from PCI config space via VFIO_PCI_CONFIG_REGION_INDEX. So it is the address Linux assigned to that BAR (plus offset).

maintainer
#3 lore raw
Quoted 12 lines · David Epping
Quoted 10 lines · Keith Busch

drivers/nvme/host/ioctl.c static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 309

Quoted 5 lines · David Epping
drivers/nvme/host/ioctl.c +1 -0
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);

This is not correct: the user space virtual address isn't the device DMA'able address. The driver already handles mapping the user address to kernel space, then to dma, then sets the PRP accordingly.

Quoted 5 lines

To clarify, the ioctl struct addr field is not filled with a memory buffer address by the userspace, but a PCIe mapped BAR address plus offset. It is obtained by the userspace application operating the FPGA vfio device by reading from PCI config space via VFIO_PCI_CONFIG_REGION_INDEX. So it is the address Linux assigned to that BAR (plus offset).

The driver and block layer should already handle PCIe addresses. You're supposed to mmap it to user space first though, and pass that address in instead. And you'd also need to set cmd.data_len to a non-zero value so the driver doesn't skip setting up the data pointers.

Note, creating IO queues from user space, while not explicitly prevented today, is not supported. The driver doesn't know you've done this so the queue isn't properly handled on a controller reset.

#4 lore raw
Quoted 19 lines · Keith Busch
Quoted 12 lines · David Epping
Quoted 10 lines · Keith Busch

drivers/nvme/host/ioctl.c static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns, · line 309

Quoted 5 lines · David Epping
drivers/nvme/host/ioctl.c +1 -0
@@ -306,6 +306,7 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	c.common.nsid = cpu_to_le32(cmd.nsid);
 	c.common.cdw2[0] = cpu_to_le32(cmd.cdw2);
 	c.common.cdw2[1] = cpu_to_le32(cmd.cdw3);
+	c.common.dptr.prp1 = cpu_to_le64(cmd.addr);

This is not correct: the user space virtual address isn't the device DMA'able address. The driver already handles mapping the user address to kernel space, then to dma, then sets the PRP accordingly.

Quoted 5 lines

To clarify, the ioctl struct addr field is not filled with a memory buffer address by the userspace, but a PCIe mapped BAR address plus offset. It is obtained by the userspace application operating the FPGA vfio device by reading from PCI config space via VFIO_PCI_CONFIG_REGION_INDEX. So it is the address Linux assigned to that BAR (plus offset).

Quoted 9 lines

The driver and block layer should already handle PCIe addresses. You're supposed to mmap it to user space first though, and pass that address in instead. And you'd also need to set cmd.data_len to a non-zero value so the driver doesn't skip setting up the data pointers.

Note, creating IO queues from user space, while not explicitly prevented today, is not supported. The driver doesn't know you've done this so the queue isn't properly handled on a controller reset.

Hi Keith, I understand that creating IO queues from user space is not supported by the current driver. That's why we created patches for that a couple of years ago and ported them to new Kernels since.

My question is, and maybe I should have put this in my initial email explicitely, is there interest in having such functionality in the upstream Linux in-Kernel NVMe driver? An interface and mechanism to request and manage IO queues that are not used by the Linux NVMe driver to perform IO, but handed to a separate entity for this purpose.

Of course an upstream implementation would have to take many more things into account, like the reset you mentioned, and IOMMU setup, and much more. But that's only worth looking at if there is upstream interest in it.

Thanks for your feedback, David

maintainer
#5 lore raw
Quoted 5 lines · David Epping

My question is, and maybe I should have put this in my initial email explicitely, is there interest in having such functionality in the upstream Linux in-Kernel NVMe driver? An interface and mechanism to request and manage IO queues that are not used by the Linux NVMe driver to perform IO, but handed to a separate entity for this purpose.

Partitioning device resources to assign to special purposes should be under a well defined framework. Unfortunately the only thing I know of approaching this is SIOV. :) Not sure how other maintainers and developers feel about it, but that's the route I would go for this. It at least provides memory access on a queue granularity and neatly separates the control plane.

#6 lore raw
Quoted 7 lines · Keith Busch
Quoted 5 lines · David Epping

My question is, and maybe I should have put this in my initial email explicitely, is there interest in having such functionality in the upstream Linux in-Kernel NVMe driver? An interface and mechanism to request and manage IO queues that are not used by the Linux NVMe driver to perform IO, but handed to a separate entity for this purpose.

Quoted 6 lines

Partitioning device resources to assign to special purposes should be under a well defined framework. Unfortunately the only thing I know of approaching this is SIOV. :) Not sure how other maintainers and developers feel about it, but that's the route I would go for this. It at least provides memory access on a queue granularity and neatly separates the control plane.

Yeah, we can't just hand out queues. I/O to all namespaces can be done on queue, and any queue can address any IOVA, so this is fundamentally unsafe. Add to that fun like abort handling and it's just not going to work at all. We had at least to previous public attempts at such schemes (Damiens' libvnme back in the day, and the Mellanox nvmet offloading) that were rejected for the same reason.

#7 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 6 lines · Keith Busch

Partitioning device resources to assign to special purposes should be under a well defined framework. Unfortunately the only thing I know of approaching this is SIOV. :) Not sure how other maintainers and developers feel about it, but that's the route I would go for this. It at least provides memory access on a queue granularity and neatly separates the control plane.

Quoted 7 lines

Yeah, we can't just hand out queues. I/O to all namespaces can be done on queue, and any queue can address any IOVA, so this is fundamentally unsafe. Add to that fun like abort handling and it's just not going to work at all. We had at least to previous public attempts at such schemes (Damiens' libvnme back in the day, and the Mellanox nvmet offloading) that were rejected for the same reason.

Thank you both for your feedback, I get the point. I'll definitely look into using SRIOV or SPDK to migrate the system to an unmodified upstream NVMe driver mid-term.
Thank you for such a stable base to build upon, David

maintainer
#8 lore raw
Quoted 3 lines · David Epping

Thank you both for your feedback, I get the point. I'll definitely look into using SRIOV or SPDK to migrate the system to an unmodified upstream NVMe driver mid-term.

SRIOV could definitely get you there with existing capable hardware and software as long as you don't need to exceed the VF count, but it is a bit heavy for what you're describing.

My SIOV suggestion is more fine grained for similar use cases, however there's no nvme standard or kernel support for the feature, so anything using the concepts would be a custom solution; NVMe would need some mechanism to associate an IO queue to a PASID, then attach namespace access to that queue. After that it's just a matter of implementing the "mediated" device.

If you're interested, this is a recent proposal to generically setup SIOV, but it needs some work:

https://lore.kernel.org/linux-pci/[email protected]/

#9 lore raw
Quoted 16 lines · Keith Busch
Quoted 3 lines · David Epping

Thank you both for your feedback, I get the point. I'll definitely look into using SRIOV or SPDK to migrate the system to an unmodified upstream NVMe driver mid-term.

SRIOV could definitely get you there with existing capable hardware and software as long as you don't need to exceed the VF count, but it is a bit heavy for what you're describing.

My SIOV suggestion is more fine grained for similar use cases, however there's no nvme standard or kernel support for the feature, so anything using the concepts would be a custom solution; NVMe would need some mechanism to associate an IO queue to a PASID, then attach namespace access to that queue. After that it's just a matter of implementing the "mediated" device.

Quoted 5 lines

If you're interested, this is a recent proposal to generically setup SIOV, but it needs some work:

https://lore.kernel.org/linux-pci/[email protected]/

Keith, thank you for the follow up and catching my mistake. I noticed the missing R and just assumed its a typo... Sorry.
I will absolutely look into SIOV to understand the concept!

#10 lore raw
Quoted 3 lines · David Epping

Keith, thank you for the follow up and catching my mistake. I noticed the missing R and just assumed its a typo... Sorry.
I will absolutely look into SIOV to understand the concept!

Note that SIOV would indeed be very interesting for these use cases, but it requires spec work and hardware support. The NVMe technical working group is looking into a proposal for it at the moment.

#11 lore raw
In reply to David Epping
Quoted 10 lines

The system setup where this patch has been used is as follows:
- P2P PCIe capable CPU (currently also IOMMU disabled)
- patched Linux in-Kernel NVMe driver for local PCIe NVMe SSDs
- FPGA accelerator implementing NVMe IO queue memory and IO queue handling, exposed via PCIe BAR
- vfio-pcie Kernel driver plus vfio userspace FPGA driver / application
- The userspace application creates new NVMe IO queues at the SSD using the patched admin ioctl and points them towards the FPGA BAR. It then informs the FPGA about the SSD BAR address and IO queue ID. From then on the FPGA can access the SSD storage entirely without software interaction.

Hi David,

I would like to ask for your insight on one point about the FPGA queue-offload setup described in the RFC. This is not about the PRP1 ioctl change itself; I am personally interested in FPGA/NVMe datapath offload and would like to better understand how your setup handled this.

For the I/O queues handled by the FPGA, how does the FPGA learn that the SSD has posted new CQEs?

Did your implementation disable interrupts for those CQs and let the FPGA poll the CQ phase tag, or did you use MSI/MSI-X with the corresponding NVMe MSI-X vector targeting an FPGA BAR event register instead of the host interrupt controller?

I also wonder how the I/O work was submitted to the FPGA in this model. Does the CPU still provide the FPGA with per-I/O information such as the data buffer address and the NVMe namespace/LBA range, while the FPGA then builds and submits the NVMe commands? Or is the FPGA able to derive most of that by itself after the initial queue setup?

Thanks,
Guzebing

#12 lore raw
Quoted 26 lines · 顾泽兵
Quoted 10 lines

The system setup where this patch has been used is as follows:
- P2P PCIe capable CPU (currently also IOMMU disabled)
- patched Linux in-Kernel NVMe driver for local PCIe NVMe SSDs
- FPGA accelerator implementing NVMe IO queue memory and IO queue handling, exposed via PCIe BAR
- vfio-pcie Kernel driver plus vfio userspace FPGA driver / application
- The userspace application creates new NVMe IO queues at the SSD using the patched admin ioctl and points them towards the FPGA BAR. It then informs the FPGA about the SSD BAR address and IO queue ID. From then on the FPGA can access the SSD storage entirely without software interaction.

Hi David,

I would like to ask for your insight on one point about the FPGA queue-offload setup described in the RFC. This is not about the PRP1 ioctl change itself; I am personally interested in FPGA/NVMe datapath offload and would like to better understand how your setup handled this.

For the I/O queues handled by the FPGA, how does the FPGA learn that the SSD has posted new CQEs?

Did your implementation disable interrupts for those CQs and let the FPGA poll the CQ phase tag, or did you use MSI/MSI-X with the corresponding NVMe MSI-X vector targeting an FPGA BAR event register instead of the host interrupt controller?

Quoted 9 lines

I also wonder how the I/O work was submitted to the FPGA in this model. Does the CPU still provide the FPGA with per-I/O information such as the data buffer address and the NVMe namespace/LBA range, while the FPGA then builds and submits the NVMe commands? Or is the FPGA able to derive most of that by itself after the initial queue setup?

Thanks,
Guzebing

Hi Guzebing,

the I/O queues managed by the FPGA are implemented as FPGA internal SRAM, and thus the FPGA sees and performs every single queue memory access. As you assumed, interrupts are disabled for these queues, and software would call this polling, but for the FPGA it is instantaneous knowledge about the access.

After initial I/O queue setup the FPGA operates completely autonomous as far as NVMe is concerened.
There is additional Linux userspace software controlling the operation and telling the FPGA which linear range of LBAs it is allowed to access, but that is not a NVMe driver/protocol level knowledge or enforcement. As such, Linux simultaneous access to the same LBAs is technically possible, but does not make sense because of caching.

We use the FPGA to record data from external sources (FPGA attached network interfaces, high-speed ADCs, ...) to a set of NVMe SSDs in RAID configuration. Linux never gets to see this data (or even knows this is happening). Only after the recording Linux may open and use the RAID block device (we use mdraid structures). This mutually exclusive access scheduling is managed by userspace software.

Best regards, David