Skip to contentSkip to conversation

NVMe PCI endpoint function driver

PATCH Superseded Damien Le Moal 3 participants 16 messages Oct 11, 2024 – Oct 14, 2024

Subsystems (5)
Documentation PCI NVMe NVMe target +1 more subsystems

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover NVMe PCI endpoint function driver

This patch series implements an NVMe PCI endpoint driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. This series is based on the improved PCI endpoint API patches "Improve PCI memory mapping API" (see [1]).

The first 3 patches of this series are changes to the NVMe target and fabrics code to facilitate reusing the NVMe code and an NVMe host controller from other (non NVMe) drivers.

Patch 4 is the main patch which introduces the NVMe PCI endpoint driver. This patch commit message provides and overview of the driver design and operation.

Finally, patch 5 adds documentation files to document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe PCI endpoint device and describe the NVMe endpoint function driver binding attributes.

This driver has been extensively tested using a Radxa Rock5B board (rk3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board (however, this board does not support PCI DMA, leading to very poor performance).

Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:

 +----------------------------------+------------------------+
 | Workload                         | IOPS (BW)              |
 +----------------------------------+------------------------+
 | Rand read, 4KB, QD=1, 1 job      | 7382 IOPS              |
 | Rand read, 4KB, QD=32, 1 job     | 45.5k IOPS             |
 | Rand read, 4KB, QD=32, 4 jobs    | 49.7k IOPS             |
 | Rand read, 128KB, QD=32, 1 job   | 10.0k IOPS (1.31 GB/s) |
 | Rand read, 128KB, QD=32, 4 jobs  | 10.2k IOPS (1.33 GB/s) |
 | Seq read, 128KB, QD=32, 1 job    | 1.28 GB/s              |
 | Seq read, 512KB, QD=32, 1 job    | 1.28 GB/s              |
 | Rand write, 128KB, QD=32, 1 job  | 8713 IOPS (1.14 GB/s)  |
 | Rand write, 128KB, QD=32, 4 jobs | 8103 IOPS (1.06 GB/s)  |
 | Seq write, 128KB, QD=32, 1 job   | 8557 IOPS (1.12 GB/s)  |
 | Seq write, 512KB, QD=32, 1 job   | 2069 IOPS (1.08 GB/s)  |
 +----------------------------------+------------------------+

These results use the default MDTS of the NVMe enpoint driver of 128 KB. Setting the NVMe endpoint device with a larger MDTS of 512 KB leads to improved maximum throughput of up to 2.4 GB/s (e.g. for the 512K random read workloads and sequential read workloads). The maximum IOPS achieved with this larger MDTS does not change significantly.

This driver is not intended for production use but rather to be a playground for learning NVMe and NVMe over fabrics and exploring/testing new NVMe features while providing reasonably good performance.

[1] https://lore.kernel.org/linux-pci/[email protected]/T/#t

Changes from v1:
- Added review tag to patch 1
- Modified patch 4 to:
- Add Rick's copyright notice
- Improve admin command handling (set_features command) to handle the number of queues feature (among others) to enable Windows host
- Improved SQ and CQ work items handling

Damien Le Moal (5):
nvmet: rename and move nvmet_get_log_page_len()
nvmef: export nvmef_create_ctrl()
nvmef: Introduce the NVME_OPT_HIDDEN_NS option
PCI: endpoint: Add NVMe endpoint function driver
PCI: endpoint: Document the NVMe endpoint function driver

 .../endpoint/function/binding/pci-nvme.rst    |   34 +
 Documentation/PCI/endpoint/index.rst          |    3 +
 .../PCI/endpoint/pci-nvme-function.rst        |  151 +
 Documentation/PCI/endpoint/pci-nvme-howto.rst |  189 ++
 MAINTAINERS                                   |    9 +
 drivers/nvme/host/core.c                      |   17 +-
 drivers/nvme/host/fabrics.c                   |   11 +-
 drivers/nvme/host/fabrics.h                   |    5 +
 drivers/nvme/target/admin-cmd.c               |   20 +-
 drivers/nvme/target/discovery.c               |    4 +-
 drivers/nvme/target/nvmet.h                   |    3 -
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2591 +++++++++++++++++
 include/linux/nvme.h                          |   19 +
 15 files changed, 3036 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
Signature · 1 line
2.47.0
#1 lore raw

PATCH v2 1/5 nvmet: rename and move nvmet_get_log_page_len()

The code for nvmet_get_log_page_len() has no pedendency on nvme target code and only depends on struct nvme_command. Move this helper function out of drivers/nvme/target/admin-cmd.c and inline it as part of the generic definitions in include/linux/nvme.h. Apply the same modification to nvmet_get_log_page_offset().

---
 drivers/nvme/target/admin-cmd.c | 20 +-------------------
 drivers/nvme/target/discovery.c |  4 ++--
 drivers/nvme/target/nvmet.h     |  3 ---
 include/linux/nvme.h            | 19 +++++++++++++++++++
 4 files changed, 22 insertions(+), 24 deletions(-)
drivers/nvme/target/admin-cmd.c +1 -19
index 081f0473cd9e..64434654b713 100644
@@ -12,19 +12,6 @@
 #include <linux/unaligned.h>
 #include "nvmet.h"
 
-u32 nvmet_get_log_page_len(struct nvme_command *cmd)
-{
-	u32 len = le16_to_cpu(cmd->get_log_page.numdu);
-
-	len <<= 16;
-	len += le16_to_cpu(cmd->get_log_page.numdl);
-	/* NUMD is a 0's based value */
-	len += 1;
-	len *= sizeof(u32);
-
-	return len;
-}
-
 static u32 nvmet_feat_data_len(struct nvmet_req *req, u32 cdw10)
 {
 	switch (cdw10 & 0xff) {
@@ -35,11 +22,6 @@ static u32 nvmet_feat_data_len(struct nvmet_req *req, u32 cdw10)
 	}
 }
 
-u64 nvmet_get_log_page_offset(struct nvme_command *cmd)
-{
-	return le64_to_cpu(cmd->get_log_page.lpo);
-}
-
 static void nvmet_execute_get_log_page_noop(struct nvmet_req *req)
 {
 	nvmet_req_complete(req, nvmet_zero_sgl(req, 0, req->transfer_len));
@@ -319,7 +301,7 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 
 static void nvmet_execute_get_log_page(struct nvmet_req *req)
 {
-	if (!nvmet_check_transfer_len(req, nvmet_get_log_page_len(req->cmd)))
+	if (!nvmet_check_transfer_len(req, nvme_get_log_page_len(req->cmd)))
 		return;
 
 	switch (req->cmd->get_log_page.lid) {
drivers/nvme/target/discovery.c +2 -2
index 28843df5fa7c..71c94a54bcd8 100644
@@ -163,8 +163,8 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
 	const int entry_size = sizeof(struct nvmf_disc_rsp_page_entry);
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmf_disc_rsp_page_hdr *hdr;
-	u64 offset = nvmet_get_log_page_offset(req->cmd);
-	size_t data_len = nvmet_get_log_page_len(req->cmd);
+	u64 offset = nvme_get_log_page_offset(req->cmd);
+	size_t data_len = nvme_get_log_page_len(req->cmd);
 	size_t alloc_len;
 	struct nvmet_subsys_link *p;
 	struct nvmet_port *r;
drivers/nvme/target/nvmet.h +0 -3
index 190f55e6d753..6e9499268c28 100644
@@ -541,9 +541,6 @@ u16 nvmet_copy_from_sgl(struct nvmet_req *req, off_t off, void *buf,
 		size_t len);
 u16 nvmet_zero_sgl(struct nvmet_req *req, off_t off, size_t len);
 
-u32 nvmet_get_log_page_len(struct nvme_command *cmd);
-u64 nvmet_get_log_page_offset(struct nvme_command *cmd);
-
 extern struct list_head *nvmet_ports;
 void nvmet_port_disc_changed(struct nvmet_port *port,
 		struct nvmet_subsys *subsys);
include/linux/nvme.h +19 -0
index b58d9405d65e..1f6d8cd0389a 100644
@@ -10,6 +10,7 @@
 #include <linux/bits.h>
 #include <linux/types.h>
 #include <linux/uuid.h>
+#include <asm/byteorder.h>
 
 /* NQN names in commands fields specified one size */
 #define NVMF_NQN_FIELD_LEN	256
@@ -1856,6 +1857,24 @@ static inline bool nvme_is_write(const struct nvme_command *cmd)
 	return cmd->common.opcode & 1;
 }
 
+static inline __u32 nvme_get_log_page_len(struct nvme_command *cmd)
+{
+	__u32 len = le16_to_cpu(cmd->get_log_page.numdu);
+
+	len <<= 16;
+	len += le16_to_cpu(cmd->get_log_page.numdl);
+	/* NUMD is a 0's based value */
+	len += 1;
+	len *= sizeof(__u32);
+
+	return len;
+}
+
+static inline __u64 nvme_get_log_page_offset(struct nvme_command *cmd)
+{
+	return le64_to_cpu(cmd->get_log_page.lpo);
+}
+
 enum {
 	/*
 	 * Generic Command Status:
Signature · 1 line
2.47.0
Reviewed-by on v2 1/5
#2 lore raw

Re: [PATCH v2 1/5] nvmet: rename and move nvmet_get_log_page_len()

Quoted 8 lines · Damien Le Moal

The code for nvmet_get_log_page_len() has no pedendency on nvme target code and only depends on struct nvme_command. Move this helper function out of drivers/nvme/target/admin-cmd.c and inline it as part of the generic definitions in include/linux/nvme.h. Apply the same modification to nvmet_get_log_page_offset().

Looks good.

-ck

#3 lore raw

PATCH v2 2/5 nvmef: export nvmef_create_ctrl()

In reply to Damien Le Moal

Export nvmef_create_ctrl() to allow drivers to directly call this function instead of forcing the creation of a fabrics host controller with a write to /dev/nvmef. The export is restricted to the NVME_FABRICS namespace.

---
 drivers/nvme/host/fabrics.c | 4 ++--
 drivers/nvme/host/fabrics.h | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fabrics.c +2 -2
index 432efcbf9e2f..e3c990d50704 100644
@@ -1276,8 +1276,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
 				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
-static struct nvme_ctrl *
-nvmf_create_ctrl(struct device *dev, const char *buf)
+struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf)
 {
 	struct nvmf_ctrl_options *opts;
 	struct nvmf_transport_ops *ops;
@@ -1346,6 +1345,7 @@ nvmf_create_ctrl(struct device *dev, const char *buf)
 	nvmf_free_options(opts);
 	return ERR_PTR(ret);
 }
+EXPORT_SYMBOL_NS_GPL(nvmf_create_ctrl, NVME_FABRICS);
 
 static const struct class nvmf_class = {
 	.name = "nvme-fabrics",
drivers/nvme/host/fabrics.h +1 -0
index 21d75dc4a3a0..2dd3aeb8c53a 100644
@@ -214,6 +214,7 @@ static inline unsigned int nvmf_nr_io_queues(struct nvmf_ctrl_options *opts)
 		min(opts->nr_poll_queues, num_online_cpus());
 }
 
+struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf);
 int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val);
 int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val);
Signature · 1 line
2.47.0
Reviewed-by on v2 2/5
#4 lore raw

Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()

On 10/11/24 05:19, Damien Le Moal wrote:

nit:- please consider following subject line as nvmef_create_ctrl()
      doesn't exists :-

    nvme-fabrics: export nvmf_create_ctrl()

Quoted 3 lines

Export nvmef_create_ctrl() to allow drivers to directly call this function instead of forcing the creation of a fabrics host controller with a write to /dev/nvmef. The export is restricted to the NVME_FABRICS

I'm not sure /dev/nvmef exists I think you meant /dev/nvme-fabrics above ?

Quoted 3 lines

namespace.

With above fix, looks good.

-ck

#5 lore raw

Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()

In reply to Damien Le Moal

s/nvmef_create_ctrl/nvmf_create_ctrl/

But evem looking at the code later using it I fail how this could work.

nvmf_create_ctrl is used to implement writes to the /dev/nvme-fabrics control device to create a fabrics controller out of thin air. The biggest part of it is parsing the options provided as a string, which most importantly includes the actual transport used.

But you really need to force a pcie transport type here, which as far as I can tell isn't even added anywhere.

#6 lore raw

Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()

Quoted 5 lines · Christoph Hellwig

s/nvmef_create_ctrl/nvmf_create_ctrl/

But evem looking at the code later using it I fail how this could work.

Quoted 7 lines

nvmf_create_ctrl is used to implement writes to the /dev/nvme-fabrics control device to create a fabrics controller out of thin air. The biggest part of it is parsing the options provided as a string, which most importantly includes the actual transport used.

But you really need to force a pcie transport type here, which as far as I can tell isn't even added anywhere.

Nope, the PCIe "transport" is the front-end of the endpoint driver. What this patch does is to allow this driver to be created by passing it a string that is normally used for "nvme connect" command, which calls nvmf_create_ctrl().

The entire thing was created to not add a PCIe host transport but rather to use a locally created fabric host controller which connect to whatever the user wants. That is: the PCI endpoint driver can implement a PCI interface which uses a loop nvme device, or a tcp nvme device (and if you have a board that can do rdma or fc, that can also be used as the backend nvme device used from the pci endpoint driver).

I only need this function exported so that I can call it from the configfs writes of the pci endpoint when the user sets up the endpoint.

Signature · 2 lines
Damien Le Moal
Western Digital Research
#7 lore raw

Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()

Quoted 4 lines · Damien Le Moal

Nope, the PCIe "transport" is the front-end of the endpoint driver. What this patch does is to allow this driver to be created by passing it a string that is normally used for "nvme connect" command, which calls nvmf_create_ctrl().

That's a pretty wrong and broken layering. PCIe isn't any different than RDMA or TCP in being a NVMe transport.

Quoted 6 lines

The entire thing was created to not add a PCIe host transport but rather to use a locally created fabric host controller which connect to whatever the user wants. That is: the PCI endpoint driver can implement a PCI interface which uses a loop nvme device, or a tcp nvme device (and if you have a board that can do rdma or fc, that can also be used as the backend nvme device used from the pci endpoint driver).

You can always use the nvmet passthrough backend to directly use a nvme controller to export through nvmet.

#8 lore raw

PATCH v2 3/5 nvmef: Introduce the NVME_OPT_HIDDEN_NS option

In reply to Damien Le Moal

Introduce the NVME fabrics option NVME_OPT_HIDDEN_NS to allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.

This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.

---
 drivers/nvme/host/core.c    | 17 ++++++++++++++---
 drivers/nvme/host/fabrics.c |  7 ++++++-
 drivers/nvme/host/fabrics.h |  4 ++++
 3 files changed, 24 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +14 -3
index 43d73d31c66f..6559e1028685 100644
@@ -1714,11 +1714,17 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)
 	queue_work(nvme_wq, &ctrl->async_event_work);
 }
 
+static inline bool nvme_hidden_ns(struct nvme_ctrl *ctrl)
+{
+	return ctrl->opts && ctrl->opts->hidden_ns;
+}
+
 static int nvme_ns_open(struct nvme_ns *ns)
 {
 
 	/* should never be called due to GENHD_FL_HIDDEN */
-	if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head)))
+	if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head) ||
+			 nvme_hidden_ns(ns->ctrl)))
 		goto fail;
 	if (!nvme_get_ns(ns))
 		goto fail;
@@ -3828,6 +3834,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	disk->fops = &nvme_bdev_ops;
 	disk->private_data = ns;
 
+	if (nvme_hidden_ns(ctrl))
+		disk->flags |= GENHD_FL_HIDDEN;
+
 	ns->disk = disk;
 	ns->queue = disk->queue;
 	ns->ctrl = ctrl;
@@ -3879,7 +3888,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (device_add_disk(ctrl->device, ns->disk, nvme_ns_attr_groups))
 		goto out_cleanup_ns_from_list;
 
-	if (!nvme_ns_head_multipath(ns->head))
+	if (!nvme_ns_head_multipath(ns->head) &&
+	    !nvme_hidden_ns(ctrl))
 		nvme_add_ns_cdev(ns);
 
 	nvme_mpath_add_disk(ns, info->anagrpid);
@@ -3945,7 +3955,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	/* guarantee not available in head->list */
 	synchronize_srcu(&ns->head->srcu);
 
-	if (!nvme_ns_head_multipath(ns->head))
+	if (!nvme_ns_head_multipath(ns->head) &&
+	    !nvme_hidden_ns(ns->ctrl))
 		nvme_cdev_del(&ns->cdev, &ns->cdev_device);
 	del_gendisk(ns->disk);
 
drivers/nvme/host/fabrics.c +6 -1
index e3c990d50704..64e95727ae2a 100644
@@ -707,6 +707,7 @@ static const match_table_t opt_tokens = {
 #ifdef CONFIG_NVME_TCP_TLS
 	{ NVMF_OPT_TLS,			"tls"			},
 #endif
+	{ NVMF_OPT_HIDDEN_NS,		"hidden_ns"		},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -1053,6 +1054,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tls = true;
 			break;
+		case NVMF_OPT_HIDDEN_NS:
+			opts->hidden_ns = true;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -1274,7 +1278,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
 				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
-				 NVMF_OPT_DHCHAP_CTRL_SECRET)
+				 NVMF_OPT_DHCHAP_CTRL_SECRET | \
+				 NVMF_OPT_HIDDEN_NS)
 
 struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf)
 {
drivers/nvme/host/fabrics.h +4 -0
index 2dd3aeb8c53a..5388610e475d 100644
@@ -66,6 +66,7 @@ enum {
 	NVMF_OPT_TLS		= 1 << 25,
 	NVMF_OPT_KEYRING	= 1 << 26,
 	NVMF_OPT_TLS_KEY	= 1 << 27,
+	NVMF_OPT_HIDDEN_NS	= 1 << 28,
 };
 
 /**
@@ -108,6 +109,8 @@ enum {
  * @nr_poll_queues: number of queues for polling I/O
  * @tos: type of service
  * @fast_io_fail_tmo: Fast I/O fail timeout in seconds
+ * @fast_io_fail_tmo: Fast I/O fail timeout in seconds
+ * @hide_dev: Hide block devices for the namesapces of the controller
  */
 struct nvmf_ctrl_options {
 	unsigned		mask;
@@ -133,6 +136,7 @@ struct nvmf_ctrl_options {
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
+	bool			hidden_ns;
 	unsigned int		nr_write_queues;
 	unsigned int		nr_poll_queues;
 	int			tos;
Signature · 1 line
2.47.0
#9 lore raw

Re: [PATCH v2 3/5] nvmef: Introduce the NVME_OPT_HIDDEN_NS option

Quoted 9 lines · Damien Le Moal

Introduce the NVME fabrics option NVME_OPT_HIDDEN_NS to allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.

This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.

That doesn't make any sense whatsover. Why would you create a passthrough controller to support PCIe?

#10 lore raw

Re: [PATCH v2 3/5] nvmef: Introduce the NVME_OPT_HIDDEN_NS option

Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Damien Le Moal

Introduce the NVME fabrics option NVME_OPT_HIDDEN_NS to allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.

Quoted 6 lines
Quoted 3 lines

This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.

That doesn't make any sense whatsover. Why would you create a passthrough controller to support PCIe?

PCIe is handled by the PCI endpoint driver and that driver only passes the nvme commands it gets to the local fabrics controller that is being used (which can be a loopback or tcp or whatever nvmf supports).

Signature · 2 lines
Damien Le Moal
Western Digital Research
#11 lore raw

PATCH v2 4/5 PCI: endpoint: Add NVMe endpoint function driver

In reply to Damien Le Moal

From: Damien Le Moal <[email protected]>

Add a Linux PCI Endpoint function driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. The NVMe endpoint function driver relies as most as possible on the NVMe fabrics driver for executing NVMe commands received from the host, to minimize NVMe command parsing. However, some admin commands must be modified to satisfy PCI transport specifications constraints (e.g. queue management commands support and the optional SGL support).

The NVMe endpoint function drivers is created as follows:
1) Upon binding of the endpoint function driver to the endpoint controller (pci_epf_nvme_bind()), the function driver sets up BAR 0 for the NVMe PCI controller with enough doorbell space to accommodate up to PCI_EPF_NVME_MAX_NR_QUEUES (16) queue pairs. The DMA channels that will be used to exchange data with the host over the PCI link are also initialized.
2) The endpoint function driver then creates the NVMe host fabrics controller using nvmef_create_ctrl() (called from pci_epf_nvme_create_ctrl()), which connects the host controller to its target (e.g. a loop target with a file or block device or a TCP remote target).
3) Once the PCI link status is detected to be up, the endpoint controller initializes IRQ management and BAR 0 content to advertize its capabilities. The capabilities of the fabrics controller are mostly used unmodified (pci_epf_nvme_init_ctrl_regs()). With that, the endpoint controller starts a delayed task to poll the BAR 0 register bar to detect changes to the CC register.
4) When the PCI host enables the controller, pci_epf_nvme_enable_ctrl() is called to create the admin submission and completion queues and start the fabrics controller with nvme_start_ctrl(). The endpoint controller then starts a delayed work to poll the admin submission queue doorbell to detect commands from the PCI host.
5) Admin commands received from the PCI host are retrieved from the admin queue by mapping the queue memory to PCI memory space, copying the command locally using a struct pci_epf_nvme_cmd, and proccess the command using pci_epf_nvme_process_admin_cmd().
6) I/O commands are similarly handled: each I/O submission queue uses a delayed work to poll the queue doorbell and upon detection of a command being issued by the host, the I/O command is copied locally and processed using pci_epf_nvme_process_io_cmd().

I/O and admin commands are processed as follows:
1) A minimal parsing of the command is done to determine the command buffer size and data transfer direction. The command processing then continues using a command work scheduled using a per queue-pair high-priority workqueue (pci_epf_nvme_exec_cmd_work()).
2) The command execution work calls pci_epf_nvme_exec_cmd() which will retrieve and parse the command PRPs to determine the PCI address location of the command buffer segments, and retrieve the command data if the command is a write command. The command is then executed using the host fabrics controller by calling __nvme_submit_sync_cmd(). Once done, pci_epf_nvme_complete_cmd() is called to complete the command, after having transferred the command data back to the PCI host in the case of a read command.
3) pci_epf_nvme_complete_cmd() queues the command in a completion list for the completion queue of the command and schedules the queue completion work which will batch CQ entry transfers to the PCI host with the completion queue memory mapped to the host PCI address of the completion queue.

With this processing, most of the command parsing and handling is left to the NVMe fabrics code. The only NVMe specific parsing implemented in the endpoint driver is the command PRP parsing. Of note is that the current code does not support SGL (this capability is thus not advertized).

For data transfers, the endpoint driver relies by default on the DMA RX and TX channels of the hardware endpoint PCI controller. If no DMA channels are available, the NVMe endpoint function driver falls back to using mmio, which degrades performance significantly but keeps the function working.

The BAR register polling work also monitors for controller-disable events (e.g. the PCI host reboots or shutdown). Such events trigger calls to pci_epf_nvme_disable_ctrl() which drains, cleanups and destroys the local queue pairs.

The configuration and enablement of this NVMe endpoint function driver can be fully controlled using configfs, once a NVMe fabrics target is also setup. The available configfs parameters are:
- ctrl_opts: Fabrics controller connection arguments, as formatted for the nvme cli "connect" command.
- dma_enable: Enable (default) or disable DMA data transfers.
- mdts_kb: Change the maximum data transfer size (default: 128 KB).

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2591 +++++++++++++++++
 4 files changed, 2608 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
MAINTAINERS +7 -0
index a097afd76ded..301e0a1b56e8 100644
@@ -16558,6 +16558,13 @@ S:	Supported
 F:	drivers/platform/x86/nvidia-wmi-ec-backlight.c
 F:	include/linux/platform_data/x86/nvidia-wmi-ec-backlight.h
 
+NVME ENDPOINT DRIVER
+M:	Damien Le Moal <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/pci/endpoint/functions/pci-epf-nvme.c
+
 NVM EXPRESS DRIVER
 M:	Keith Busch <[email protected]>
 M:	Jens Axboe <[email protected]>
drivers/pci/endpoint/functions/Kconfig +9 -0
index 0c9cea0698d7..ea641d558fb8 100644
@@ -47,3 +47,12 @@ config PCI_EPF_MHI
 	   devices such as SDX55.
 
 	   If in doubt, say "N" to disable Endpoint driver for MHI bus.
+
+config PCI_EPF_NVME
+	tristate "PCI Endpoint NVMe function driver"
+	depends on PCI_ENDPOINT && NVME_TARGET
+	help
+	   Enable this configuration option to enable the NVMe PCI endpoint
+	   function driver.
+
+	   If in doubt, say "N".
drivers/pci/endpoint/functions/Makefile +1 -0
index 696473fce50e..fe2d6cf8c502 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
 obj-$(CONFIG_PCI_EPF_MHI)		+= pci-epf-mhi.o
+obj-$(CONFIG_PCI_EPF_NVME)		+= pci-epf-nvme.o
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +2591 -0

2,591 lines not on this page · raw message

Signature · 1 line
2.47.0
#12 lore raw

Re: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver

For one please keep nvme target code in drivers/nvme/ PCI endpoint is just another transport and should not have device class logic.

But I also really fail to understand the architecture of the whole thing. It is a target driver and should in no way tie into the NVMe host code, the host code runs on the other side of the PCIe wire.

#13 lore raw

Re: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 6 lines · Christoph Hellwig

For one please keep nvme target code in drivers/nvme/ PCI endpoint is just another transport and should not have device class logic.

But I also really fail to understand the architecture of the whole thing. It is a target driver and should in no way tie into the NVMe host code, the host code runs on the other side of the PCIe wire.

Nope, it is not a target driver. It is a PCI endpoint driver which turns the host running it into a PCIe NVMe device. But the NVMe part implementation is minimal. Instead I use an endpoint local fabrics host controller which is itself connected to whatever target you want (loop, tcp, ...).

Overall, it looks like this:

         +-----------------------------------+
         | PCIe Host Machine (Root-Complex)  |
         | (BIOS, Grub, Linux, Windows, ...) |
         |                                   |
         |       +------------------+        |
         |       | NVMe PCIe driver |        |
         +-------+------------------+--------+
                           |
                 PCIe bus  |
                           |
        +----+---------------------------+-----+
        |    | PCIe NVMe endpoint driver |     |
	|    | (Handles BAR registers,   |     |
	|    | doorbells, IRQs, SQs, CQs |     |
	|    | and DMA transfers)        |     |
        |    +---------------------------+     |
        |                  |                   |
        |    +---------------------------+     |
        |    |     NVMe fabrics host     |     |
        |    +---------------------------+     |
        |                  |                   |
        |    +---------------------------+     |
        |    |     NVMe fabrics target   |     |
        |    |      (loop, TCP, ...)     |     |
        |    +---------------------------+     |
        |                                      |
        | PCIe Endpoint Machine (e.g. Rock 5B) |
        +--------------------------------------+

The nvme target can be anything that can be supported with the PCI Endpoint Machine. With a small board like the Rock 5B, it is loop (file and block device), TCP target or NVMe passtrhough (using the PCIe Gen2 M.2 E-Key slot).

Unless I am mistaken, if I use a PCI transport as the base for the endpoint driver, I would be able to connect only to a PCIe nvme device as the backend, no ? With the above design, I can use anything support by nvmf as backend and that is exposed to the root-complex host through the nvme endpoint PCIe driver. To do that, the PCI endpoint driver mostly need only to create the fabrics host with nvmf_create_ctrl(), which connects to the target and then the nvme endpoint driver can execute the nvme commands with __nvme_submit_sync_cmd(). Only some admin commands need special handling (e.g. create sq/cq).

Signature · 2 lines
Damien Le Moal
Western Digital Research
#14 lore raw

Re: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 9 lines · Damien Le Moal
        |    | PCIe NVMe endpoint driver |     |
	|    | (Handles BAR registers,   |     |
	|    | doorbells, IRQs, SQs, CQs |     |
	|    | and DMA transfers)        |     |
        |    +---------------------------+     |
        |                  |                   |
        |    +---------------------------+     |
        |    |     NVMe fabrics host     |     |
        |    +---------------------------+     |

That whole fabrics host here is broken. The PCI frontend needs to talk directly to the target code instead of doing a completely pointless roundtrip through the block layer.

Quoted 3 lines

Unless I am mistaken, if I use a PCI transport as the base for the endpoint driver, I would be able to connect only to a PCIe nvme device as the backend, no ?

No. Any fabrisc transport (or frontend) can you any of the the backends (file / bdev and passthrough).

#15 lore raw

PATCH v2 5/5 PCI: endpoint: Document the NVMe endpoint function driver

In reply to Damien Le Moal

Add the documentation files:
- Documentation/PCI/endpoint/pci-nvme-function.rst
- Documentation/PCI/endpoint/pci-nvme-howto.rst
- Documentation/PCI/endpoint/function/binding/pci-nvme.rst

To respectively document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe endpoint device and describe the NVMe endpoint function driver binding attributes.

---
 .../endpoint/function/binding/pci-nvme.rst    |  34 ++++
 Documentation/PCI/endpoint/index.rst          |   3 +
 .../PCI/endpoint/pci-nvme-function.rst        | 151 ++++++++++++++
 Documentation/PCI/endpoint/pci-nvme-howto.rst | 189 ++++++++++++++++++
 MAINTAINERS                                   |   2 +
 5 files changed, 379 insertions(+)
 create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst
Documentation/PCI/endpoint/function/binding/pci-nvme.rst new file +34 -0
new file mode 100644
index 000000000000..d80293c08bcd
@@ -0,0 +1,34 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================
+PCI NVMe Endpoint Function
+==========================
+
+1) Create the function subdirectory pci_epf_nvme.0 in the
+pci_ep/functions/pci_epf_nvme directory of configfs.
+
+Standard EPF Configurable Fields:
+
+================   ===========================================================
+vendorid           Do not care (e.g. PCI_ANY_ID)
+deviceid           Do not care (e.g. PCI_ANY_ID)
+revid              Do not care
+progif_code	   Must be 0x02 (NVM Express)
+baseclass_code     Must be 0x1 (PCI_BASE_CLASS_STORAGE)
+subclass_code      Must be 0x08 (Non-Volatile Memory controller)
+cache_line_size    Do not care
+subsys_vendor_id   Do not care (e.g. PCI_ANY_ID)
+subsys_id          Do not care (e.g. PCI_ANY_ID)
+msi_interrupts     At least equal to the number of queue pairs desired
+msix_interrupts    At least equal to the number of queue pairs desired
+interrupt_pin      Interrupt PIN to use if MSI and MSI-X are not supported
+================   ===========================================================
+
+The NVMe EPF specific configurable fields are in the nvme subdirectory of the
+directory created in 1
+
+================   ===========================================================
+ctrl_opts          NVMe target connection parameters
+dma_enable         Enable (1) or disable (0) DMA transfers; default = 1
+mdts_kb            Maximum data transfer size in KiB; default = 128
+================   ===========================================================
Documentation/PCI/endpoint/index.rst +3 -0
index 4d2333e7ae06..764f1e8f81f2 100644
@@ -15,6 +15,9 @@ PCI Endpoint Framework
    pci-ntb-howto
    pci-vntb-function
    pci-vntb-howto
+   pci-nvme-function
+   pci-nvme-howto
 
    function/binding/pci-test
    function/binding/pci-ntb
+   function/binding/pci-nvme
Documentation/PCI/endpoint/pci-nvme-function.rst new file +151 -0

151 lines not on this page · raw message

Documentation/PCI/endpoint/pci-nvme-howto.rst new file +189 -0

189 lines not on this page · raw message

MAINTAINERS +2 -0
index 301e0a1b56e8..48431d2aa751 100644
@@ -16563,6 +16563,8 @@ M:	Damien Le Moal <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Supported
+F:	Documentation/PCI/endpoint/function/binding/pci-nvme.rst
+F:	Documentation/PCI/endpoint/pci-nvme-*.rst
 F:	drivers/pci/endpoint/functions/pci-epf-nvme.c
 
 NVM EXPRESS DRIVER
Signature · 1 line
2.47.0