NVMe PCI endpoint function driver
Subsystems (5)
v2 cover NVMe PCI endpoint function driver
This patch series implements an NVMe PCI endpoint driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. This series is based on the improved PCI endpoint API patches "Improve PCI memory mapping API" (see [1]).
The first 3 patches of this series are changes to the NVMe target and fabrics code to facilitate reusing the NVMe code and an NVMe host controller from other (non NVMe) drivers.
Patch 4 is the main patch which introduces the NVMe PCI endpoint driver. This patch commit message provides and overview of the driver design and operation.
Finally, patch 5 adds documentation files to document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe PCI endpoint device and describe the NVMe endpoint function driver binding attributes.
This driver has been extensively tested using a Radxa Rock5B board (rk3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board (however, this board does not support PCI DMA, leading to very poor performance).
Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:
+----------------------------------+------------------------+ | Workload | IOPS (BW) | +----------------------------------+------------------------+ | Rand read, 4KB, QD=1, 1 job | 7382 IOPS | | Rand read, 4KB, QD=32, 1 job | 45.5k IOPS | | Rand read, 4KB, QD=32, 4 jobs | 49.7k IOPS | | Rand read, 128KB, QD=32, 1 job | 10.0k IOPS (1.31 GB/s) | | Rand read, 128KB, QD=32, 4 jobs | 10.2k IOPS (1.33 GB/s) | | Seq read, 128KB, QD=32, 1 job | 1.28 GB/s | | Seq read, 512KB, QD=32, 1 job | 1.28 GB/s | | Rand write, 128KB, QD=32, 1 job | 8713 IOPS (1.14 GB/s) | | Rand write, 128KB, QD=32, 4 jobs | 8103 IOPS (1.06 GB/s) | | Seq write, 128KB, QD=32, 1 job | 8557 IOPS (1.12 GB/s) | | Seq write, 512KB, QD=32, 1 job | 2069 IOPS (1.08 GB/s) | +----------------------------------+------------------------+
These results use the default MDTS of the NVMe enpoint driver of 128 KB. Setting the NVMe endpoint device with a larger MDTS of 512 KB leads to improved maximum throughput of up to 2.4 GB/s (e.g. for the 512K random read workloads and sequential read workloads). The maximum IOPS achieved with this larger MDTS does not change significantly.
This driver is not intended for production use but rather to be a playground for learning NVMe and NVMe over fabrics and exploring/testing new NVMe features while providing reasonably good performance.
[1] https://lore.kernel.org/linux-pci/[email protected]/T/#t
Changes from v1:
- Added review tag to patch 1
- Modified patch 4 to:
- Add Rick's copyright notice
- Improve admin command handling (set_features command) to handle the number of queues feature (among others) to enable Windows host
- Improved SQ and CQ work items handling
Damien Le Moal (5):
nvmet: rename and move nvmet_get_log_page_len()
nvmef: export nvmef_create_ctrl()
nvmef: Introduce the NVME_OPT_HIDDEN_NS option
PCI: endpoint: Add NVMe endpoint function driver
PCI: endpoint: Document the NVMe endpoint function driver
.../endpoint/function/binding/pci-nvme.rst | 34 + Documentation/PCI/endpoint/index.rst | 3 + .../PCI/endpoint/pci-nvme-function.rst | 151 + Documentation/PCI/endpoint/pci-nvme-howto.rst | 189 ++ MAINTAINERS | 9 + drivers/nvme/host/core.c | 17 +- drivers/nvme/host/fabrics.c | 11 +- drivers/nvme/host/fabrics.h | 5 + drivers/nvme/target/admin-cmd.c | 20 +- drivers/nvme/target/discovery.c | 4 +- drivers/nvme/target/nvmet.h | 3 - drivers/pci/endpoint/functions/Kconfig | 9 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-nvme.c | 2591 +++++++++++++++++ include/linux/nvme.h | 19 + 15 files changed, 3036 insertions(+), 30 deletions(-) create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
Signature · 1 line
2.47.0
PATCH v2 1/5 nvmet: rename and move nvmet_get_log_page_len()
The code for nvmet_get_log_page_len() has no pedendency on nvme target code and only depends on struct nvme_command. Move this helper function out of drivers/nvme/target/admin-cmd.c and inline it as part of the generic definitions in include/linux/nvme.h. Apply the same modification to nvmet_get_log_page_offset().
--- drivers/nvme/target/admin-cmd.c | 20 +------------------- drivers/nvme/target/discovery.c | 4 ++-- drivers/nvme/target/nvmet.h | 3 --- include/linux/nvme.h | 19 +++++++++++++++++++ 4 files changed, 22 insertions(+), 24 deletions(-)
drivers/nvme/target/admin-cmd.c +1 -19
@@ -12,19 +12,6 @@#include <linux/unaligned.h> #include "nvmet.h" -u32 nvmet_get_log_page_len (struct nvme_command *cmd) -{ -u32 len =le16_to_cpu (cmd->get_log_page.numdu); - - len <<=16 ; - len +=le16_to_cpu (cmd->get_log_page.numdl); -/* NUMD is a 0's based value */ - len +=1 ; - len *=sizeof (u32); - -return len; -} -static u32 nvmet_feat_data_len (struct nvmet_req *req,u32 cdw10) {switch (cdw10 &0xff ) {
@@ -35,11 +22,6 @@ static u32 nvmet_feat_data_len(struct nvmet_req *req, u32 cdw10) } } -u64 nvmet_get_log_page_offset (struct nvme_command *cmd) -{ -return le64_to_cpu (cmd->get_log_page.lpo); -} -static void nvmet_execute_get_log_page_noop (struct nvmet_req *req) {nvmet_req_complete (req,nvmet_zero_sgl (req,0 , req->transfer_len));
@@ -319,7 +301,7 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)static void nvmet_execute_get_log_page (struct nvmet_req *req) { -if (!nvmet_check_transfer_len (req,nvmet_get_log_page_len (req->cmd))) +if (!nvmet_check_transfer_len (req,nvme_get_log_page_len (req->cmd)))return ;switch (req->cmd->get_log_page.lid) {
drivers/nvme/target/discovery.c +2 -2
@@ -163,8 +163,8 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)const int entry_size =sizeof (struct nvmf_disc_rsp_page_entry );struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvmf_disc_rsp_page_hdr *hdr; -u64 offset =nvmet_get_log_page_offset (req->cmd); -size_t data_len =nvmet_get_log_page_len (req->cmd); +u64 offset =nvme_get_log_page_offset (req->cmd); +size_t data_len =nvme_get_log_page_len (req->cmd);size_t alloc_len;struct nvmet_subsys_link *p;struct nvmet_port *r;
drivers/nvme/target/nvmet.h +0 -3
@@ -541,9 +541,6 @@ u16 nvmet_copy_from_sgl(struct nvmet_req *req, off_t off, void *buf, size_t len);u16 nvmet_zero_sgl (struct nvmet_req *req,off_t off,size_t len); -u32 nvmet_get_log_page_len (struct nvme_command *cmd); -u64 nvmet_get_log_page_offset (struct nvme_command *cmd); -extern struct list_head *nvmet_ports;void nvmet_port_disc_changed (struct nvmet_port *port,struct nvmet_subsys *subsys);
include/linux/nvme.h +19 -0
@@ -10,6 +10,7 @@#include <linux/bits.h> #include <linux/types.h> #include <linux/uuid.h> +#include <asm/byteorder.h> /* NQN names in commands fields specified one size */ #define NVMF_NQN_FIELD_LEN 256
@@ -1856,6 +1857,24 @@ static inline bool nvme_is_write(const struct nvme_command *cmd)return cmd->common.opcode &1 ; } +static inline __u32 nvme_get_log_page_len (struct nvme_command *cmd) +{ +__u32 len =le16_to_cpu (cmd->get_log_page.numdu); + + len <<=16 ; + len +=le16_to_cpu (cmd->get_log_page.numdl); +/* NUMD is a 0's based value */ + len +=1 ; + len *=sizeof (__u32); + +return len; +} + +static inline __u64 nvme_get_log_page_offset (struct nvme_command *cmd) +{ +return le64_to_cpu (cmd->get_log_page.lpo); +} +enum {/* * Generic Command Status:
Signature · 1 line
2.47.0
Re: [PATCH v2 1/5] nvmet: rename and move nvmet_get_log_page_len()
Quoted 8 lines · Damien Le Moal
The code for
nvmet_get_log_page_len()has no pedendency on nvme target code and only depends on structnvme_command. Move this helper function out ofdrivers/nvme/target/admin-cmd.cand inline it as part of the generic definitions ininclude/linux/nvme.h. Apply the same modification tonvmet_get_log_page_offset().Signed-off-by Damien Le Moal<[email protected]>Reviewed-by Christoph Hellwig<[email protected]>
Looks good.
-ck
PATCH v2 2/5 nvmef: export nvmef_create_ctrl()
In reply to Damien Le MoalExport nvmef_create_ctrl() to allow drivers to directly call this function instead of forcing the creation of a fabrics host controller with a write to /dev/nvmef. The export is restricted to the NVME_FABRICS namespace.
--- drivers/nvme/host/fabrics.c | 4 ++-- drivers/nvme/host/fabrics.h | 1 + 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fabrics.c +2 -2
@@ -1276,8 +1276,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options); NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\ NVMF_OPT_DHCHAP_CTRL_SECRET) -static struct nvme_ctrl * -nvmf_create_ctrl (struct device *dev,const char *buf) +struct nvme_ctrl *nvmf_create_ctrl (struct device *dev,const char *buf) {struct nvmf_ctrl_options *opts;struct nvmf_transport_ops *ops;
@@ -1346,6 +1345,7 @@ nvmf_create_ctrl(struct device *dev, const char *buf)nvmf_free_options (opts);return ERR_PTR (ret); } +EXPORT_SYMBOL_NS_GPL (nvmf_create_ctrl,NVME_FABRICS );static const struct class nvmf_class = { .name ="nvme-fabrics" ,
drivers/nvme/host/fabrics.h +1 -0
@@ -214,6 +214,7 @@ static inline unsigned int nvmf_nr_io_queues(struct nvmf_ctrl_options *opts)min (opts->nr_poll_queues,num_online_cpus ()); } +struct nvme_ctrl *nvmf_create_ctrl (struct device *dev,const char *buf);int nvmf_reg_read32 (struct nvme_ctrl *ctrl,u32 off,u32 *val);int nvmf_reg_read64 (struct nvme_ctrl *ctrl,u32 off,u64 *val);int nvmf_reg_write32 (struct nvme_ctrl *ctrl,u32 off,u32 val);
Signature · 1 line
2.47.0
Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()
On 10/11/24 05:19, Damien Le Moal wrote:
nit:- please consider following subject line as nvmef_create_ctrl()
doesn't exists :-
nvme-fabrics: export nvmf_create_ctrl()
Quoted 3 lines
Export
nvmef_create_ctrl()to allow drivers to directly call this function instead of forcing the creation of a fabrics host controller with a write to/dev/nvmef. The export is restricted to theNVME_FABRICS
I'm not sure /dev/nvmef exists I think you meant /dev/nvme-fabrics above ?
Quoted 3 lines
namespace.
Signed-off-by Damien Le Moal <[email protected]>
With above fix, looks good.
-ck
Christoph Hellwig
maintainerRe: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()
In reply to Damien Le Moals/nvmef_create_ctrl/nvmf_create_ctrl/
But evem looking at the code later using it I fail how this could work.
nvmf_create_ctrl is used to implement writes to the /dev/nvme-fabrics control device to create a fabrics controller out of thin air. The biggest part of it is parsing the options provided as a string, which most importantly includes the actual transport used.
But you really need to force a pcie transport type here, which as far as I can tell isn't even added anywhere.
Re: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()
Quoted 5 lines · Christoph Hellwig
s/nvmef_create_ctrl/nvmf_create_ctrl/
But evem looking at the code later using it I fail how this could work.
Quoted 7 lines
nvmf_create_ctrlis used to implement writes to the/dev/nvme-fabricscontrol device to create a fabrics controller out of thin air. The biggest part of it is parsing the options provided as a string, which most importantly includes the actual transport used.But you really need to force a pcie transport type here, which as far as I can tell isn't even added anywhere.
Nope, the PCIe "transport" is the front-end of the endpoint driver. What this patch does is to allow this driver to be created by passing it a string that is normally used for "nvme connect" command, which calls nvmf_create_ctrl().
The entire thing was created to not add a PCIe host transport but rather to use a locally created fabric host controller which connect to whatever the user wants. That is: the PCI endpoint driver can implement a PCI interface which uses a loop nvme device, or a tcp nvme device (and if you have a board that can do rdma or fc, that can also be used as the backend nvme device used from the pci endpoint driver).
I only need this function exported so that I can call it from the configfs writes of the pci endpoint when the user sets up the endpoint.
Signature · 2 lines
Damien Le Moal Western Digital Research
Christoph Hellwig
maintainerRe: [PATCH v2 2/5] nvmef: export nvmef_create_ctrl()
Quoted 4 lines · Damien Le Moal
Nope, the PCIe "transport" is the front-end of the endpoint driver. What this patch does is to allow this driver to be created by passing it a string that is normally used for "nvme connect" command, which calls
nvmf_create_ctrl().
That's a pretty wrong and broken layering. PCIe isn't any different than RDMA or TCP in being a NVMe transport.
Quoted 6 lines
The entire thing was created to not add a PCIe host transport but rather to use a locally created fabric host controller which connect to whatever the user wants. That is: the PCI endpoint driver can implement a PCI interface which uses a loop nvme device, or a tcp nvme device (and if you have a board that can do rdma or fc, that can also be used as the backend nvme device used from the pci endpoint driver).
You can always use the nvmet passthrough backend to directly use a nvme controller to export through nvmet.
PATCH v2 3/5 nvmef: Introduce the NVME_OPT_HIDDEN_NS option
In reply to Damien Le MoalIntroduce the NVME fabrics option NVME_OPT_HIDDEN_NS to allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.
This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.
--- drivers/nvme/host/core.c | 17 ++++++++++++++--- drivers/nvme/host/fabrics.c | 7 ++++++- drivers/nvme/host/fabrics.h | 4 ++++ 3 files changed, 24 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +14 -3
@@ -1714,11 +1714,17 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)queue_work (nvme_wq, &ctrl->async_event_work); } +static inline bool nvme_hidden_ns (struct nvme_ctrl *ctrl) +{ +return ctrl->opts && ctrl->opts->hidden_ns; +} +static int nvme_ns_open (struct nvme_ns *ns) {/* should never be called due to GENHD_FL_HIDDEN */ -if (WARN_ON_ONCE (nvme_ns_head_multipath (ns->head))) +if (WARN_ON_ONCE (nvme_ns_head_multipath (ns->head) || +nvme_hidden_ns (ns->ctrl))) goto fail;if (!nvme_get_ns (ns)) goto fail;
@@ -3828,6 +3834,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) disk->fops = &nvme_bdev_ops; disk->private_data = ns; +if (nvme_hidden_ns (ctrl)) + disk->flags |=GENHD_FL_HIDDEN ; + ns->disk = disk; ns->queue = disk->queue; ns->ctrl = ctrl;
@@ -3879,7 +3888,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)if (device_add_disk (ctrl->device, ns->disk, nvme_ns_attr_groups)) goto out_cleanup_ns_from_list; -if (!nvme_ns_head_multipath (ns->head)) +if (!nvme_ns_head_multipath (ns->head) && + !nvme_hidden_ns (ctrl))nvme_add_ns_cdev (ns);nvme_mpath_add_disk (ns, info->anagrpid);
@@ -3945,7 +3955,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)/* guarantee not available in head->list */ synchronize_srcu (&ns->head->srcu); -if (!nvme_ns_head_multipath (ns->head)) +if (!nvme_ns_head_multipath (ns->head) && + !nvme_hidden_ns (ns->ctrl))nvme_cdev_del (&ns->cdev, &ns->cdev_device);del_gendisk (ns->disk);
drivers/nvme/host/fabrics.c +6 -1
@@ -707,6 +707,7 @@ static const match_table_t opt_tokens = { #ifdef CONFIG_NVME_TCP_TLS { NVMF_OPT_TLS, "tls" }, #endif + { NVMF_OPT_HIDDEN_NS, "hidden_ns" }, { NVMF_OPT_ERR, NULL } };
@@ -1053,6 +1054,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } opts->tls = true;break ; +case NVMF_OPT_HIDDEN_NS : + opts->hidden_ns = true; +break ;default :pr_warn ("unknown parameter or missing value '%s' in ctrl creation request\n" , p);
@@ -1274,7 +1278,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options); NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\ NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\ NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\ - NVMF_OPT_DHCHAP_CTRL_SECRET) + NVMF_OPT_DHCHAP_CTRL_SECRET | \ + NVMF_OPT_HIDDEN_NS)struct nvme_ctrl *nvmf_create_ctrl (struct device *dev,const char *buf) {
drivers/nvme/host/fabrics.h +4 -0
@@ -66,6 +66,7 @@ enum {NVMF_OPT_TLS =1 <<25 ,NVMF_OPT_KEYRING =1 <<26 ,NVMF_OPT_TLS_KEY =1 <<27 , + NVMF_OPT_HIDDEN_NS = 1 << 28, };/**
@@ -108,6 +109,8 @@ enum {* @nr_poll_queues: number of queues for polling I/O * @tos: type of service * @fast_io_fail_tmo: Fast I/O fail timeout in seconds +* @fast_io_fail_tmo: Fast I/O fail timeout in seconds +* @hide_dev: Hide block devices for the namesapces of the controller */ struct nvmf_ctrl_options {unsigned mask;
@@ -133,6 +136,7 @@ struct nvmf_ctrl_options {bool disable_sqflow;bool hdr_digest;bool data_digest; +bool hidden_ns;unsigned int nr_write_queues;unsigned int nr_poll_queues;int tos;
Signature · 1 line
2.47.0
Christoph Hellwig
maintainerRe: [PATCH v2 3/5] nvmef: Introduce the NVME_OPT_HIDDEN_NS option
Quoted 9 lines · Damien Le Moal
Introduce the NVME fabrics option
NVME_OPT_HIDDEN_NSto allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.
That doesn't make any sense whatsover. Why would you create a passthrough controller to support PCIe?
Re: [PATCH v2 3/5] nvmef: Introduce the NVME_OPT_HIDDEN_NS option
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Damien Le Moal
Introduce the NVME fabrics option
NVME_OPT_HIDDEN_NSto allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.
Quoted 6 lines
Quoted 3 lines
This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.
That doesn't make any sense whatsover. Why would you create a passthrough controller to support PCIe?
PCIe is handled by the PCI endpoint driver and that driver only passes the nvme commands it gets to the local fabrics controller that is being used (which can be a loopback or tcp or whatever nvmf supports).
Signature · 2 lines
Damien Le Moal Western Digital Research
PATCH v2 4/5 PCI: endpoint: Add NVMe endpoint function driver
In reply to Damien Le MoalFrom: Damien Le Moal <[email protected]>
Add a Linux PCI Endpoint function driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. The NVMe endpoint function driver relies as most as possible on the NVMe fabrics driver for executing NVMe commands received from the host, to minimize NVMe command parsing. However, some admin commands must be modified to satisfy PCI transport specifications constraints (e.g. queue management commands support and the optional SGL support).
The NVMe endpoint function drivers is created as follows:
1) Upon binding of the endpoint function driver to the endpoint controller (pci_epf_nvme_bind()), the function driver sets up BAR 0 for the NVMe PCI controller with enough doorbell space to accommodate up to PCI_EPF_NVME_MAX_NR_QUEUES (16) queue pairs. The DMA channels that will be used to exchange data with the host over the PCI link are also initialized.
2) The endpoint function driver then creates the NVMe host fabrics controller using nvmef_create_ctrl() (called from pci_epf_nvme_create_ctrl()), which connects the host controller to its target (e.g. a loop target with a file or block device or a TCP remote target).
3) Once the PCI link status is detected to be up, the endpoint controller initializes IRQ management and BAR 0 content to advertize its capabilities. The capabilities of the fabrics controller are mostly used unmodified (pci_epf_nvme_init_ctrl_regs()). With that, the endpoint controller starts a delayed task to poll the BAR 0 register bar to detect changes to the CC register.
4) When the PCI host enables the controller, pci_epf_nvme_enable_ctrl() is called to create the admin submission and completion queues and start the fabrics controller with nvme_start_ctrl(). The endpoint controller then starts a delayed work to poll the admin submission queue doorbell to detect commands from the PCI host.
5) Admin commands received from the PCI host are retrieved from the admin queue by mapping the queue memory to PCI memory space, copying the command locally using a struct pci_epf_nvme_cmd, and proccess the command using pci_epf_nvme_process_admin_cmd().
6) I/O commands are similarly handled: each I/O submission queue uses a delayed work to poll the queue doorbell and upon detection of a command being issued by the host, the I/O command is copied locally and processed using pci_epf_nvme_process_io_cmd().
I/O and admin commands are processed as follows:
1) A minimal parsing of the command is done to determine the command buffer size and data transfer direction. The command processing then continues using a command work scheduled using a per queue-pair high-priority workqueue (pci_epf_nvme_exec_cmd_work()).
2) The command execution work calls pci_epf_nvme_exec_cmd() which will retrieve and parse the command PRPs to determine the PCI address location of the command buffer segments, and retrieve the command data if the command is a write command. The command is then executed using the host fabrics controller by calling __nvme_submit_sync_cmd(). Once done, pci_epf_nvme_complete_cmd() is called to complete the command, after having transferred the command data back to the PCI host in the case of a read command.
3) pci_epf_nvme_complete_cmd() queues the command in a completion list for the completion queue of the command and schedules the queue completion work which will batch CQ entry transfers to the PCI host with the completion queue memory mapped to the host PCI address of the completion queue.
With this processing, most of the command parsing and handling is left to the NVMe fabrics code. The only NVMe specific parsing implemented in the endpoint driver is the command PRP parsing. Of note is that the current code does not support SGL (this capability is thus not advertized).
For data transfers, the endpoint driver relies by default on the DMA RX and TX channels of the hardware endpoint PCI controller. If no DMA channels are available, the NVMe endpoint function driver falls back to using mmio, which degrades performance significantly but keeps the function working.
The BAR register polling work also monitors for controller-disable events (e.g. the PCI host reboots or shutdown). Such events trigger calls to pci_epf_nvme_disable_ctrl() which drains, cleanups and destroys the local queue pairs.
The configuration and enablement of this NVMe endpoint function driver can be fully controlled using configfs, once a NVMe fabrics target is also setup. The available configfs parameters are:
- ctrl_opts: Fabrics controller connection arguments, as formatted for the nvme cli "connect" command.
- dma_enable: Enable (default) or disable DMA data transfers.
- mdts_kb: Change the maximum data transfer size (default: 128 KB).
Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.
--- MAINTAINERS | 7 + drivers/pci/endpoint/functions/Kconfig | 9 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-nvme.c | 2591 +++++++++++++++++ 4 files changed, 2608 insertions(+) create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
MAINTAINERS +7 -0
@@ -16558,6 +16558,13 @@ S: Supported F: drivers/platform/x86/nvidia-wmi-ec-backlight.c F: include/linux/platform_data/x86/nvidia-wmi-ec-backlight.h +NVME ENDPOINT DRIVER +M: Damien Le Moal <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: drivers/pci/endpoint/functions/pci-epf-nvme.c + NVM EXPRESS DRIVER M: Keith Busch <[email protected]> M: Jens Axboe <[email protected]>
drivers/pci/endpoint/functions/Kconfig +9 -0
@@ -47,3 +47,12 @@ config PCI_EPF_MHI devices such as SDX55. If in doubt, say "N" to disable Endpoint driver for MHI bus. + +config PCI_EPF_NVME + tristate "PCI Endpoint NVMe function driver" + depends on PCI_ENDPOINT && NVME_TARGET + help + Enable this configuration option to enable the NVMe PCI endpoint + function driver. + + If in doubt, say "N".
drivers/pci/endpoint/functions/Makefile +1 -0
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST) += pci-epf-test.o obj-$(CONFIG_PCI_EPF_NTB) += pci-epf-ntb.o obj-$(CONFIG_PCI_EPF_VNTB) += pci-epf-vntb.o obj-$(CONFIG_PCI_EPF_MHI) += pci-epf-mhi.o +obj-$(CONFIG_PCI_EPF_NVME) += pci-epf-nvme.o
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +2591 -0
2,591 lines not on this page · raw message
Signature · 1 line
2.47.0
Christoph Hellwig
maintainerRe: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver
For one please keep nvme target code in drivers/nvme/ PCI endpoint is just another transport and should not have device class logic.
But I also really fail to understand the architecture of the whole thing. It is a target driver and should in no way tie into the NVMe host code, the host code runs on the other side of the PCIe wire.
Re: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver
Quoted 6 lines · Christoph Hellwig
For one please keep nvme target code in
drivers/nvme/PCI endpoint is just another transport and should not have device class logic.But I also really fail to understand the architecture of the whole thing. It is a target driver and should in no way tie into the NVMe host code, the host code runs on the other side of the PCIe wire.
Nope, it is not a target driver. It is a PCI endpoint driver which turns the host running it into a PCIe NVMe device. But the NVMe part implementation is minimal. Instead I use an endpoint local fabrics host controller which is itself connected to whatever target you want (loop, tcp, ...).
Overall, it looks like this:
+-----------------------------------+
| PCIe Host Machine (Root-Complex) |
| (BIOS, Grub, Linux, Windows, ...) |
| |
| +------------------+ |
| | NVMe PCIe driver | |
+-------+------------------+--------+
|
PCIe bus |
|
+----+---------------------------+-----+
| | PCIe NVMe endpoint driver | |
| | (Handles BAR registers, | |
| | doorbells, IRQs, SQs, CQs | |
| | and DMA transfers) | |
| +---------------------------+ |
| | |
| +---------------------------+ |
| | NVMe fabrics host | |
| +---------------------------+ |
| | |
| +---------------------------+ |
| | NVMe fabrics target | |
| | (loop, TCP, ...) | |
| +---------------------------+ |
| |
| PCIe Endpoint Machine (e.g. Rock 5B) |
+--------------------------------------+The nvme target can be anything that can be supported with the PCI Endpoint Machine. With a small board like the Rock 5B, it is loop (file and block device), TCP target or NVMe passtrhough (using the PCIe Gen2 M.2 E-Key slot).
Unless I am mistaken, if I use a PCI transport as the base for the endpoint driver, I would be able to connect only to a PCIe nvme device as the backend, no ? With the above design, I can use anything support by nvmf as backend and that is exposed to the root-complex host through the nvme endpoint PCIe driver. To do that, the PCI endpoint driver mostly need only to create the fabrics host with nvmf_create_ctrl(), which connects to the target and then the nvme endpoint driver can execute the nvme commands with __nvme_submit_sync_cmd(). Only some admin commands need special handling (e.g. create sq/cq).
Signature · 2 lines
Damien Le Moal Western Digital Research
Christoph Hellwig
maintainerRe: [PATCH v2 4/5] PCI: endpoint: Add NVMe endpoint function driver
Quoted 9 lines · Damien Le Moal
| | PCIe NVMe endpoint driver | |
| | (Handles BAR registers, | |
| | doorbells, IRQs, SQs, CQs | |
| | and DMA transfers) | |
| +---------------------------+ |
| | |
| +---------------------------+ |
| | NVMe fabrics host | |
| +---------------------------+ |That whole fabrics host here is broken. The PCI frontend needs to talk directly to the target code instead of doing a completely pointless roundtrip through the block layer.
Quoted 3 lines
Unless I am mistaken, if I use a PCI transport as the base for the endpoint driver, I would be able to connect only to a PCIe nvme device as the backend, no ?
No. Any fabrisc transport (or frontend) can you any of the the backends (file / bdev and passthrough).
PATCH v2 5/5 PCI: endpoint: Document the NVMe endpoint function driver
In reply to Damien Le MoalAdd the documentation files:
- Documentation/PCI/endpoint/pci-nvme-function.rst
- Documentation/PCI/endpoint/pci-nvme-howto.rst
- Documentation/PCI/endpoint/function/binding/pci-nvme.rst
To respectively document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe endpoint device and describe the NVMe endpoint function driver binding attributes.
--- .../endpoint/function/binding/pci-nvme.rst | 34 ++++ Documentation/PCI/endpoint/index.rst | 3 + .../PCI/endpoint/pci-nvme-function.rst | 151 ++++++++++++++ Documentation/PCI/endpoint/pci-nvme-howto.rst | 189 ++++++++++++++++++ MAINTAINERS | 2 + 5 files changed, 379 insertions(+) create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst
Documentation/PCI/endpoint/function/binding/pci-nvme.rst new file +34 -0
@@ -0,0 +1,34 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================== +PCI NVMe Endpoint Function +========================== + +1) Create the function subdirectory pci_epf_nvme.0 in the +pci_ep/functions/pci_epf_nvme directory of configfs. + +Standard EPF Configurable Fields: + +================ =========================================================== +vendorid Do not care (e.g. PCI_ANY_ID) +deviceid Do not care (e.g. PCI_ANY_ID) +revid Do not care +progif_code Must be 0x02 (NVM Express) +baseclass_code Must be 0x1 (PCI_BASE_CLASS_STORAGE) +subclass_code Must be 0x08 (Non-Volatile Memory controller) +cache_line_size Do not care +subsys_vendor_id Do not care (e.g. PCI_ANY_ID) +subsys_id Do not care (e.g. PCI_ANY_ID) +msi_interrupts At least equal to the number of queue pairs desired +msix_interrupts At least equal to the number of queue pairs desired +interrupt_pin Interrupt PIN to use if MSI and MSI-X are not supported +================ =========================================================== + +The NVMe EPF specific configurable fields are in the nvme subdirectory of the +directory created in 1 + +================ =========================================================== +ctrl_opts NVMe target connection parameters +dma_enable Enable (1) or disable (0) DMA transfers; default = 1 +mdts_kb Maximum data transfer size in KiB; default = 128 +================ ===========================================================
Documentation/PCI/endpoint/index.rst +3 -0
@@ -15,6 +15,9 @@ PCI Endpoint Framework pci-ntb-howto pci-vntb-function pci-vntb-howto + pci-nvme-function + pci-nvme-howto function/binding/pci-test function/binding/pci-ntb + function/binding/pci-nvme
Documentation/PCI/endpoint/pci-nvme-function.rst new file +151 -0
151 lines not on this page · raw message
Documentation/PCI/endpoint/pci-nvme-howto.rst new file +189 -0
189 lines not on this page · raw message
MAINTAINERS +2 -0
@@ -16563,6 +16563,8 @@ M: Damien Le Moal <[email protected]> L: [email protected] L: [email protected] S: Supported +F: Documentation/PCI/endpoint/function/binding/pci-nvme.rst +F: Documentation/PCI/endpoint/pci-nvme-*.rst F: drivers/pci/endpoint/functions/pci-epf-nvme.c NVM EXPRESS DRIVER
Signature · 1 line
2.47.0