Skip to contentSkip to conversation

NVMe PCI endpoint function driver

PATCH Superseded Damien Le Moal 6 participants 14 messages Oct 7, 2024 – Oct 20, 2024

Subsystems (5)
Documentation PCI NVMe NVMe target +1 more subsystems

v1 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

cover NVMe PCI endpoint function driver

This patch series implements an NVMe PCI endpoint driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. This series is based on the improved PCI endpoint API patches "Improve PCI memory mapping API" (see [1]).

The first 3 patches of this series are changes to the NVMe target and fabrics code to facilitate reusing the NVMe code and an NVMe host controller from other (non NVMe) drivers.

Patch 4 is the main patch which introduces the NVMe PCI endpoint driver. This patch commit message provides and overview of the driver design and operation.

Finally, patch 5 adds documentation files to document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe PCI endpoint device and describe the NVMe endpoint function driver binding attributes.

This driver has been extensively tested using a Radxa Rock5B board (rk3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board (however, this board does not support PCI DMA, leading to very poor performance).

Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:

 +----------------------------------+------------------------+
 | Workload                         | IOPS (BW)              |
 +----------------------------------+------------------------+
 | Rand read, 4KB, QD=1, 1 job      | 7382 IOPS              |
 | Rand read, 4KB, QD=32, 1 job     | 45.5k IOPS             |
 | Rand read, 4KB, QD=32, 4 jobs    | 49.7k IOPS             |
 | Rand read, 128KB, QD=32, 1 job   | 10.0k IOPS (1.31 GB/s) |
 | Rand read, 128KB, QD=32, 4 jobs  | 10.2k IOPS (1.33 GB/s) |
 | Seq read, 128KB, QD=32, 1 job    | 1.28 GB/s              |
 | Seq read, 512KB, QD=32, 1 job    | 1.28 GB/s              |
 | Rand write, 128KB, QD=32, 1 job  | 8713 IOPS (1.14 GB/s)  |
 | Rand write, 128KB, QD=32, 4 jobs | 8103 IOPS (1.06 GB/s)  |
 | Seq write, 128KB, QD=32, 1 job   | 8557 IOPS (1.12 GB/s)  |
 | Seq write, 512KB, QD=32, 1 job   | 2069 IOPS (1.08 GB/s)  |
 +----------------------------------+------------------------+

These results use the default MDTS of the NVMe enpoint driver of 128 KB. Setting the NVMe endpoint device with a larger MDTS of 512 KB leads to improved maximum throughput of up to 2.4 GB/s (e.g. for the 512K random read workloads and sequential read workloads). The maximum IOPS achieved with this larger MDTS does not change significantly.

This driver is not intended for production use but rather to be a playground for learning NVMe and NVMe over fabrics and exploring/testing new NVMe features while providing reasonably good performance.

[1] https://lore.kernel.org/linux-pci/[email protected]/T/#t

Damien Le Moal (5):
nvmet: rename and move nvmet_get_log_page_len()
nvmef: export nvmef_create_ctrl()
nvmef: Introduce the NVME_OPT_HIDDEN_NS option
PCI: endpoint: Add NVMe endpoint function driver
PCI: endpoint: Document the NVMe endpoint function driver

 .../endpoint/function/binding/pci-nvme.rst    |   34 +
 Documentation/PCI/endpoint/index.rst          |    3 +
 .../PCI/endpoint/pci-nvme-function.rst        |  151 +
 Documentation/PCI/endpoint/pci-nvme-howto.rst |  190 ++
 MAINTAINERS                                   |    9 +
 drivers/nvme/host/core.c                      |   17 +-
 drivers/nvme/host/fabrics.c                   |   11 +-
 drivers/nvme/host/fabrics.h                   |    5 +
 drivers/nvme/target/admin-cmd.c               |   20 +-
 drivers/nvme/target/discovery.c               |    4 +-
 drivers/nvme/target/nvmet.h                   |    3 -
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 include/linux/nvme.h                          |   19 +
 15 files changed, 2935 insertions(+), 30 deletions(-)
 create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
Signature · 1 line
2.46.2
#1 lore raw

PATCH 1/5 nvmet: rename and move nvmet_get_log_page_len()

The code for nvmet_get_log_page_len() has no pedendency on nvme target code and only depends on struct nvme_command. Move this helper function out of drivers/nvme/target/admin-cmd.c and inline it as part of the generic definitions in include/linux/nvme.h. Apply the same modification to nvmet_get_log_page_offset().

---
 drivers/nvme/target/admin-cmd.c | 20 +-------------------
 drivers/nvme/target/discovery.c |  4 ++--
 drivers/nvme/target/nvmet.h     |  3 ---
 include/linux/nvme.h            | 19 +++++++++++++++++++
 4 files changed, 22 insertions(+), 24 deletions(-)
drivers/nvme/target/admin-cmd.c +1 -19
index 954d4c074770..b20cd1dba207 100644
@@ -12,19 +12,6 @@
 #include <asm/unaligned.h>
 #include "nvmet.h"
 
-u32 nvmet_get_log_page_len(struct nvme_command *cmd)
-{
-	u32 len = le16_to_cpu(cmd->get_log_page.numdu);
-
-	len <<= 16;
-	len += le16_to_cpu(cmd->get_log_page.numdl);
-	/* NUMD is a 0's based value */
-	len += 1;
-	len *= sizeof(u32);
-
-	return len;
-}
-
 static u32 nvmet_feat_data_len(struct nvmet_req *req, u32 cdw10)
 {
 	switch (cdw10 & 0xff) {
@@ -35,11 +22,6 @@ static u32 nvmet_feat_data_len(struct nvmet_req *req, u32 cdw10)
 	}
 }
 
-u64 nvmet_get_log_page_offset(struct nvme_command *cmd)
-{
-	return le64_to_cpu(cmd->get_log_page.lpo);
-}
-
 static void nvmet_execute_get_log_page_noop(struct nvmet_req *req)
 {
 	nvmet_req_complete(req, nvmet_zero_sgl(req, 0, req->transfer_len));
@@ -319,7 +301,7 @@ static void nvmet_execute_get_log_page_ana(struct nvmet_req *req)
 
 static void nvmet_execute_get_log_page(struct nvmet_req *req)
 {
-	if (!nvmet_check_transfer_len(req, nvmet_get_log_page_len(req->cmd)))
+	if (!nvmet_check_transfer_len(req, nvme_get_log_page_len(req->cmd)))
 		return;
 
 	switch (req->cmd->get_log_page.lid) {
drivers/nvme/target/discovery.c +2 -2
index 28843df5fa7c..71c94a54bcd8 100644
@@ -163,8 +163,8 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
 	const int entry_size = sizeof(struct nvmf_disc_rsp_page_entry);
 	struct nvmet_ctrl *ctrl = req->sq->ctrl;
 	struct nvmf_disc_rsp_page_hdr *hdr;
-	u64 offset = nvmet_get_log_page_offset(req->cmd);
-	size_t data_len = nvmet_get_log_page_len(req->cmd);
+	u64 offset = nvme_get_log_page_offset(req->cmd);
+	size_t data_len = nvme_get_log_page_len(req->cmd);
 	size_t alloc_len;
 	struct nvmet_subsys_link *p;
 	struct nvmet_port *r;
drivers/nvme/target/nvmet.h +0 -3
index 190f55e6d753..6e9499268c28 100644
@@ -541,9 +541,6 @@ u16 nvmet_copy_from_sgl(struct nvmet_req *req, off_t off, void *buf,
 		size_t len);
 u16 nvmet_zero_sgl(struct nvmet_req *req, off_t off, size_t len);
 
-u32 nvmet_get_log_page_len(struct nvme_command *cmd);
-u64 nvmet_get_log_page_offset(struct nvme_command *cmd);
-
 extern struct list_head *nvmet_ports;
 void nvmet_port_disc_changed(struct nvmet_port *port,
 		struct nvmet_subsys *subsys);
include/linux/nvme.h +19 -0
index b58d9405d65e..1f6d8cd0389a 100644
@@ -10,6 +10,7 @@
 #include <linux/bits.h>
 #include <linux/types.h>
 #include <linux/uuid.h>
+#include <asm/byteorder.h>
 
 /* NQN names in commands fields specified one size */
 #define NVMF_NQN_FIELD_LEN	256
@@ -1856,6 +1857,24 @@ static inline bool nvme_is_write(const struct nvme_command *cmd)
 	return cmd->common.opcode & 1;
 }
 
+static inline __u32 nvme_get_log_page_len(struct nvme_command *cmd)
+{
+	__u32 len = le16_to_cpu(cmd->get_log_page.numdu);
+
+	len <<= 16;
+	len += le16_to_cpu(cmd->get_log_page.numdl);
+	/* NUMD is a 0's based value */
+	len += 1;
+	len *= sizeof(__u32);
+
+	return len;
+}
+
+static inline __u64 nvme_get_log_page_offset(struct nvme_command *cmd)
+{
+	return le64_to_cpu(cmd->get_log_page.lpo);
+}
+
 enum {
 	/*
 	 * Generic Command Status:
Signature · 1 line
2.46.2
maintainer Reviewed-by on v1 1/5
#2 lore raw

Re: [PATCH v1 1/5] nvmet: rename and move nvmet_get_log_page_len()

Quoted 7 lines · Damien Le Moal

The code for nvmet_get_log_page_len() has no pedendency on nvme target code and only depends on struct nvme_command. Move this helper function out of drivers/nvme/target/admin-cmd.c and inline it as part of the generic definitions in include/linux/nvme.h. Apply the same modification to nvmet_get_log_page_offset().

Looks good:

#4 lore raw

PATCH 2/5 nvmef: export nvmef_create_ctrl()

In reply to Damien Le Moal

Export nvmef_create_ctrl() to allow drivers to directly call this function instead of forcing the creation of a fabrics host controller with a write to /dev/nvmef. The export is restricted to the NVME_FABRICS namespace.

---
 drivers/nvme/host/fabrics.c | 4 ++--
 drivers/nvme/host/fabrics.h | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fabrics.c +2 -2
index 432efcbf9e2f..e3c990d50704 100644
@@ -1276,8 +1276,7 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
 				 NVMF_OPT_DHCHAP_CTRL_SECRET)
 
-static struct nvme_ctrl *
-nvmf_create_ctrl(struct device *dev, const char *buf)
+struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf)
 {
 	struct nvmf_ctrl_options *opts;
 	struct nvmf_transport_ops *ops;
@@ -1346,6 +1345,7 @@ nvmf_create_ctrl(struct device *dev, const char *buf)
 	nvmf_free_options(opts);
 	return ERR_PTR(ret);
 }
+EXPORT_SYMBOL_NS_GPL(nvmf_create_ctrl, NVME_FABRICS);
 
 static const struct class nvmf_class = {
 	.name = "nvme-fabrics",
drivers/nvme/host/fabrics.h +1 -0
index 21d75dc4a3a0..2dd3aeb8c53a 100644
@@ -214,6 +214,7 @@ static inline unsigned int nvmf_nr_io_queues(struct nvmf_ctrl_options *opts)
 		min(opts->nr_poll_queues, num_online_cpus());
 }
 
+struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf);
 int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val);
 int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val);
 int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val);
Signature · 1 line
2.46.2
#5 lore raw

PATCH 3/5 nvmef: Introduce the NVME_OPT_HIDDEN_NS option

In reply to Damien Le Moal

Introduce the NVME fabrics option NVME_OPT_HIDDEN_NS to allow a host controller to be created without any user visible or internally usable namespace devices. That is, if set, this option will result in the controller having no character device and no block device for any of its namespaces.

This option should be used only when the nvme controller will be managed using passthrough commands using the controller character device, either by the user or by another device driver.

---
 drivers/nvme/host/core.c    | 17 ++++++++++++++---
 drivers/nvme/host/fabrics.c |  7 ++++++-
 drivers/nvme/host/fabrics.h |  4 ++++
 3 files changed, 24 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +14 -3
index ba6508455e18..c7f0be39a30a 100644
@@ -1714,11 +1714,17 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)
 	queue_work(nvme_wq, &ctrl->async_event_work);
 }
 
+static inline bool nvme_hidden_ns(struct nvme_ctrl *ctrl)
+{
+	return ctrl->opts && ctrl->opts->hidden_ns;
+}
+
 static int nvme_ns_open(struct nvme_ns *ns)
 {
 
 	/* should never be called due to GENHD_FL_HIDDEN */
-	if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head)))
+	if (WARN_ON_ONCE(nvme_ns_head_multipath(ns->head) ||
+			 nvme_hidden_ns(ns->ctrl)))
 		goto fail;
 	if (!nvme_get_ns(ns))
 		goto fail;
@@ -3828,6 +3834,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	disk->fops = &nvme_bdev_ops;
 	disk->private_data = ns;
 
+	if (nvme_hidden_ns(ctrl))
+		disk->flags |= GENHD_FL_HIDDEN;
+
 	ns->disk = disk;
 	ns->queue = disk->queue;
 	ns->ctrl = ctrl;
@@ -3879,7 +3888,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
 	if (device_add_disk(ctrl->device, ns->disk, nvme_ns_attr_groups))
 		goto out_cleanup_ns_from_list;
 
-	if (!nvme_ns_head_multipath(ns->head))
+	if (!nvme_ns_head_multipath(ns->head) &&
+	    !nvme_hidden_ns(ctrl))
 		nvme_add_ns_cdev(ns);
 
 	nvme_mpath_add_disk(ns, info->anagrpid);
@@ -3945,7 +3955,8 @@ static void nvme_ns_remove(struct nvme_ns *ns)
 	/* guarantee not available in head->list */
 	synchronize_srcu(&ns->head->srcu);
 
-	if (!nvme_ns_head_multipath(ns->head))
+	if (!nvme_ns_head_multipath(ns->head) &&
+	    !nvme_hidden_ns(ns->ctrl))
 		nvme_cdev_del(&ns->cdev, &ns->cdev_device);
 	del_gendisk(ns->disk);
 
drivers/nvme/host/fabrics.c +6 -1
index e3c990d50704..64e95727ae2a 100644
@@ -707,6 +707,7 @@ static const match_table_t opt_tokens = {
 #ifdef CONFIG_NVME_TCP_TLS
 	{ NVMF_OPT_TLS,			"tls"			},
 #endif
+	{ NVMF_OPT_HIDDEN_NS,		"hidden_ns"		},
 	{ NVMF_OPT_ERR,			NULL			}
 };
 
@@ -1053,6 +1054,9 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 			}
 			opts->tls = true;
 			break;
+		case NVMF_OPT_HIDDEN_NS:
+			opts->hidden_ns = true;
+			break;
 		default:
 			pr_warn("unknown parameter or missing value '%s' in ctrl creation request\n",
 				p);
@@ -1274,7 +1278,8 @@ EXPORT_SYMBOL_GPL(nvmf_free_options);
 				 NVMF_OPT_HOST_ID | NVMF_OPT_DUP_CONNECT |\
 				 NVMF_OPT_DISABLE_SQFLOW | NVMF_OPT_DISCOVERY |\
 				 NVMF_OPT_FAIL_FAST_TMO | NVMF_OPT_DHCHAP_SECRET |\
-				 NVMF_OPT_DHCHAP_CTRL_SECRET)
+				 NVMF_OPT_DHCHAP_CTRL_SECRET | \
+				 NVMF_OPT_HIDDEN_NS)
 
 struct nvme_ctrl *nvmf_create_ctrl(struct device *dev, const char *buf)
 {
drivers/nvme/host/fabrics.h +4 -0
index 2dd3aeb8c53a..5388610e475d 100644
@@ -66,6 +66,7 @@ enum {
 	NVMF_OPT_TLS		= 1 << 25,
 	NVMF_OPT_KEYRING	= 1 << 26,
 	NVMF_OPT_TLS_KEY	= 1 << 27,
+	NVMF_OPT_HIDDEN_NS	= 1 << 28,
 };
 
 /**
@@ -108,6 +109,8 @@ enum {
  * @nr_poll_queues: number of queues for polling I/O
  * @tos: type of service
  * @fast_io_fail_tmo: Fast I/O fail timeout in seconds
+ * @fast_io_fail_tmo: Fast I/O fail timeout in seconds
+ * @hide_dev: Hide block devices for the namesapces of the controller
  */
 struct nvmf_ctrl_options {
 	unsigned		mask;
@@ -133,6 +136,7 @@ struct nvmf_ctrl_options {
 	bool			disable_sqflow;
 	bool			hdr_digest;
 	bool			data_digest;
+	bool			hidden_ns;
 	unsigned int		nr_write_queues;
 	unsigned int		nr_poll_queues;
 	int			tos;
Signature · 1 line
2.46.2
#6 lore raw

PATCH 4/5 PCI: endpoint: Add NVMe endpoint function driver

In reply to Damien Le Moal

From: Damien Le Moal <[email protected]>

Add a Linux PCI Endpoint function driver that implements a PCIe NVMe controller for a local NVMe fabrics host controller. The NVMe endpoint function driver relies as most as possible on the NVMe fabrics driver for executing NVMe commands received from the host, to minimize NVMe command parsing. However, some admin commands must be modified to satisfy PCI transport specifications constraints (e.g. queue management commands support and the optional SGL support).

The NVMe endpoint function drivers is created as follows:
1) Upon binding of the endpoint function driver to the endpoint controller (pci_epf_nvme_bind()), the function driver sets up BAR 0 for the NVMe PCI controller with enough doorbell space to accommodate up to PCI_EPF_NVME_MAX_NR_QUEUES (16) queue pairs. The DMA channels that will be used to exchange data with the host over the PCI link are also initialized.
2) The endpoint function driver then creates the NVMe host fabrics controller using nvmef_create_ctrl() (called from pci_epf_nvme_create_ctrl()), which connects the host controller to its target (e.g. a loop target with a file or block device or a TCP remote target).
3) Once the PCI link status is detected to be up, the endpoint controller initializes IRQ management and BAR 0 content to advertize its capabilities. The capabilities of the fabrics controller are mostly used unmodified (pci_epf_nvme_init_ctrl_regs()). With that, the endpoint controller starts a delayed task to poll the BAR 0 register bar to detect changes to the CC register.
4) When the PCI host enables the controller, pci_epf_nvme_enable_ctrl() is called to create the admin submission and completion queues and start the fabrics controller with nvme_start_ctrl(). The endpoint controller then starts a delayed work to poll the admin submission queue doorbell to detect commands from the PCI host.
5) Admin commands received from the PCI host are retrieved from the admin queue by mapping the queue memory to PCI memory space, copying the command locally using a struct pci_epf_nvme_cmd, and proccess the command using pci_epf_nvme_process_admin_cmd().
6) I/O commands are similarly handled: each I/O submission queue uses a delayed work to poll the queue doorbell and upon detection of a command being issued by the host, the I/O command is copied locally and processed using pci_epf_nvme_process_io_cmd().

I/O and admin commands are processed as follows:
1) A minimal parsing of the command is done to determine the command buffer size and data transfer direction. The command processing then continues using a command work scheduled using a per queue-pair high-priority workqueue (pci_epf_nvme_exec_cmd_work()).
2) The command execution work calls pci_epf_nvme_exec_cmd() which will retrieve and parse the command PRPs to determine the PCI address location of the command buffer segments, and retrieve the command data if the command is a write command. The command is then executed using the host fabrics controller by calling __nvme_submit_sync_cmd(). Once done, pci_epf_nvme_complete_cmd() is called to complete the command, after having transferred the command data back to the PCI host in the case of a read command.
3) pci_epf_nvme_complete_cmd() queues the command in a completion list for the completion queue of the command and schedules the queue completion work which will batch CQ entry transfers to the PCI host with the completion queue memory mapped to the host PCI address of the completion queue.

With this processing, most of the command parsing and handling is left to the NVMe fabrics code. The only NVMe specific parsing implemented in the endpoint driver is the command PRP parsing. Of note is that the current code does not support SGL (this capability is thus not advertized).

For data transfers, the endpoint driver relies by default on the DMA RX and TX channels of the hardware endpoint PCI controller. If no DMA channels are available, the NVMe endpoint function driver falls back to using mmio, which degrades performance significantly but keeps the function working.

The BAR register polling work also monitors for controller-disable events (e.g. the PCI host reboots or shutdown). Such events trigger calls to pci_epf_nvme_disable_ctrl() which drains, cleanups and destroys the local queue pairs.

The configuration and enablement of this NVMe endpoint function driver can be fully controlled using configfs, once a NVMe fabrics target is also setup. The available configfs parameters are:
- ctrl_opts: Fabrics controller connection arguments, as formatted for the nvme cli "connect" command.
- dma_enable: Enable (default) or disable DMA data transfers.
- mdts_kb: Change the maximum data transfer size (default: 128 KB).

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 4 files changed, 2506 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c
MAINTAINERS +7 -0
index c27f3190737f..c9b40621dbc1 100644
@@ -16554,6 +16554,13 @@ S:	Supported
 F:	drivers/platform/x86/nvidia-wmi-ec-backlight.c
 F:	include/linux/platform_data/x86/nvidia-wmi-ec-backlight.h
 
+NVME ENDPOINT DRIVER
+M:	Damien Le Moal <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/pci/endpoint/functions/pci-epf-nvme.c
+
 NVM EXPRESS DRIVER
 M:	Keith Busch <[email protected]>
 M:	Jens Axboe <[email protected]>
drivers/pci/endpoint/functions/Kconfig +9 -0
index 0c9cea0698d7..ea641d558fb8 100644
@@ -47,3 +47,12 @@ config PCI_EPF_MHI
 	   devices such as SDX55.
 
 	   If in doubt, say "N" to disable Endpoint driver for MHI bus.
+
+config PCI_EPF_NVME
+	tristate "PCI Endpoint NVMe function driver"
+	depends on PCI_ENDPOINT && NVME_TARGET
+	help
+	   Enable this configuration option to enable the NVMe PCI endpoint
+	   function driver.
+
+	   If in doubt, say "N".
drivers/pci/endpoint/functions/Makefile +1 -0
index 696473fce50e..fe2d6cf8c502 100644
@@ -7,3 +7,4 @@ obj-$(CONFIG_PCI_EPF_TEST)		+= pci-epf-test.o
 obj-$(CONFIG_PCI_EPF_NTB)		+= pci-epf-ntb.o
 obj-$(CONFIG_PCI_EPF_VNTB) 		+= pci-epf-vntb.o
 obj-$(CONFIG_PCI_EPF_MHI)		+= pci-epf-mhi.o
+obj-$(CONFIG_PCI_EPF_NVME)		+= pci-epf-nvme.o
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +2489 -0

2,489 lines not on this page · raw message

Signature · 1 line
2.46.2
#7 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 1 line · Damien Le Moal

From: Damien Le Moal <[email protected]>

(snip)

Quoted 3 lines

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

Here you state that the code has been completely rewritten...

Quoted 11 lines
---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 4 files changed, 2506 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c

(snip)

drivers/pci/endpoint/functions/pci-epf-nvme.c line 5

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +5 -0
@@ -0,0 +1,2489 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe function driver for PCI Endpoint Framework
+ *
+ * Copyright (C) 2019 SiFive

...yet here you claim Copyright (C) SiFive.

*If* the code has been completely rewritten, then you probably should put yourself and/or your current employeer as the copyright holder.

Kind regards,
Niklas

#8 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 37 lines · Niklas Cassel
Quoted 1 line · Damien Le Moal

From: Damien Le Moal <[email protected]>

(snip)

Quoted 3 lines

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

Here you state that the code has been completely rewritten...

Quoted 11 lines
---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 4 files changed, 2506 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c

(snip)

drivers/pci/endpoint/functions/pci-epf-nvme.c line 5

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +5 -0
@@ -0,0 +1,2489 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe function driver for PCI Endpoint Framework
+ *
+ * Copyright (C) 2019 SiFive

...yet here you claim Copyright (C) SiFive.

Quoted 2 lines

*If* the code has been completely rewritten, then you probably should put yourself and/or your current employeer as the copyright holder.

Oops. One thing I forgot to rewrite :)
Will change the copyright in v2.

Quoted 4 lines

Kind regards,
Niklas

Signature · 2 lines
Damien Le Moal
Western Digital Research
#9 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 39 lines · Damien Le Moal
Quoted 37 lines · Niklas Cassel
Quoted 1 line · Damien Le Moal

From: Damien Le Moal <[email protected]>

(snip)

Quoted 3 lines

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

Here you state that the code has been completely rewritten...

Quoted 11 lines
---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 4 files changed, 2506 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c

(snip)

drivers/pci/endpoint/functions/pci-epf-nvme.c line 5

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +5 -0
@@ -0,0 +1,2489 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe function driver for PCI Endpoint Framework
+ *
+ * Copyright (C) 2019 SiFive

...yet here you claim Copyright (C) SiFive.

Quoted 5 lines
Quoted 2 lines

*If* the code has been completely rewritten, then you probably should put yourself and/or your current employeer as the copyright holder.

Oops. One thing I forgot to rewrite :)
Will change the copyright in v2.

Hello Damien.
Thank you for acknowledging my contributions in the commit message. I would appreciate it if you could add both our names in the copyright for the v2, I don't mind not being in the MODULE_AUTHOR() but I would appreciate a mention in the header comment of the file, thank you.

Best regards,
Rick

Quoted 10 lines
Quoted 5 lines

Kind regards,
Niklas

--
Damien Le Moal
Western Digital Research

#10 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

Quoted 43 lines · Rick Wertenbroek
Quoted 42 lines · Damien Le Moal
Quoted 39 lines · Niklas Cassel
Quoted 1 line · Damien Le Moal

From: Damien Le Moal <[email protected]>

(snip)

Quoted 3 lines

Early versions of this driver code were based on an RFC submission by Alan Mikhak <[email protected]> (https://lwn.net/Articles/804369/).
The code however has since been completely rewritten.

Here you state that the code has been completely rewritten...

Quoted 11 lines
---
 MAINTAINERS                                   |    7 +
 drivers/pci/endpoint/functions/Kconfig        |    9 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-nvme.c | 2489 +++++++++++++++++
 4 files changed, 2506 insertions(+)
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-nvme.c

(snip)

drivers/pci/endpoint/functions/pci-epf-nvme.c line 5

Quoted 8 lines
drivers/pci/endpoint/functions/pci-epf-nvme.c new file +5 -0
@@ -0,0 +1,2489 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe function driver for PCI Endpoint Framework
+ *
+ * Copyright (C) 2019 SiFive

...yet here you claim Copyright (C) SiFive.

*If* the code has been completely rewritten, then you probably should put yourself and/or your current employeer as the copyright holder.

Quoted 8 lines
Quoted 2 lines

Oops. One thing I forgot to rewrite :)
Will change the copyright in v2.

Hello Damien.
Thank you for acknowledging my contributions in the commit message. I would appreciate it if you could add both our names in the copyright for the v2, I don't mind not being in the MODULE_AUTHOR() but I would appreciate a mention in the header comment of the file, thank you.

Sure, no problem. Would a copyright with your name only (no affiliation) be OK ?

Quoted 14 lines

Best regards,
Rick

Quoted 10 lines
Quoted 5 lines

Kind regards,
Niklas

--
Damien Le Moal
Western Digital Research

Signature · 2 lines
Damien Le Moal
Western Digital Research
#11 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

In reply to Damien Le Moal
kernel test robot noticed the following build errors:

Hi Damien,

kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.12-rc2 next-20241009] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Damien-Le-Moal/nvmet-rename-and-move-nvmet_get_log_page_len/20241007-124428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20241007044351.157912-5-dlemoal%40kernel.org
patch subject: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver
config: microblaze-allmodconfig (https://download.01.org/0day-ci/archive/20241010/[email protected]/config)
compiler: microblaze-linux-gcc (GCC) 14.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241010/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All error/warnings (new ones prefixed by >>):

>> drivers/pci/endpoint/functions/pci-epf-nvme.c:81:33: error: field 'pci_map' has incomplete type
      81 |         struct pci_epc_map      pci_map;
         |                                 ^~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c: In function 'pci_epf_nvme_mmio_transfer':
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:415:28: error: storage size of 'map' isn't known
     415 |         struct pci_epc_map map;
         |                            ^~~
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:419:15: error: implicit declaration of function 'pci_epc_mem_map'; did you mean 'pci_epc_mem_exit'? [-Wimplicit-function-declaration]
     419 |         ret = pci_epc_mem_map(epf->epc, epf->func_no, epf->vfunc_no,
         |               ^~~~~~~~~~~~~~~
         |               pci_epc_mem_exit
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:438:9: error: implicit declaration of function 'pci_epc_mem_unmap'; did you mean 'pci_epc_mem_init'? [-Wimplicit-function-declaration]
     438 |         pci_epc_mem_unmap(epf->epc, epf->func_no, epf->vfunc_no, &map);
         |         ^~~~~~~~~~~~~~~~~
         |         pci_epc_mem_init
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:415:28: warning: unused variable 'map' [-Wunused-variable]
     415 |         struct pci_epc_map map;
         |                            ^~~
   In file included from include/linux/bits.h:7,
                    from include/linux/bitops.h:6,
                    from include/linux/log2.h:12,
                    from include/asm-generic/div64.h:55,
                    from ./arch/microblaze/include/generated/asm/div64.h:1,
                    from include/linux/math.h:6,
                    from include/linux/delay.h:22,
                    from drivers/pci/endpoint/functions/pci-epf-nvme.c:10:
   drivers/pci/endpoint/functions/pci-epf-nvme.c: In function 'pci_epf_nvme_init_ctrl_regs':
   include/uapi/linux/bits.h:8:31: warning: left shift count >= width of type [-Wshift-count-overflow]
       8 |         (((~_UL(0)) - (_UL(1) << (l)) + 1) & \
         |                               ^~
   include/linux/bits.h:35:38: note: in expansion of macro '__GENMASK'
      35 |         (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l))
         |                                      ^~~~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1391:23: note: in expansion of macro 'GENMASK'
    1391 |         ctrl->cap &= ~GENMASK(35, 32);
         |                       ^~~~~~~
   include/uapi/linux/bits.h:9:19: warning: right shift count is negative [-Wshift-count-negative]
       9 |          (~_UL(0) >> (__BITS_PER_LONG - 1 - (h))))
         |                   ^~
   include/linux/bits.h:35:38: note: in expansion of macro '__GENMASK'
      35 |         (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l))
         |                                      ^~~~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1391:23: note: in expansion of macro 'GENMASK'
    1391 |         ctrl->cap &= ~GENMASK(35, 32);
         |                       ^~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c: In function 'pci_epf_nvme_enable_ctrl':
   include/uapi/linux/bits.h:9:19: warning: right shift count is negative [-Wshift-count-negative]
       9 |          (~_UL(0) >> (__BITS_PER_LONG - 1 - (h))))
         |                   ^~
   include/linux/bits.h:35:38: note: in expansion of macro '__GENMASK'
      35 |         (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l))
         |                                      ^~~~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1467:45: note: in expansion of macro 'GENMASK'
    1467 |                                 ctrl->acq & GENMASK(63, 12));
         |                                             ^~~~~~~
   include/uapi/linux/bits.h:9:19: warning: right shift count is negative [-Wshift-count-negative]
       9 |          (~_UL(0) >> (__BITS_PER_LONG - 1 - (h))))
         |                   ^~
   include/linux/bits.h:35:38: note: in expansion of macro '__GENMASK'
      35 |         (GENMASK_INPUT_CHECK(h, l) + __GENMASK(h, l))
         |                                      ^~~~~~~~~
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1473:50: note: in expansion of macro 'GENMASK'
    1473 |                                      ctrl->asq & GENMASK(63, 12));
         |                                                  ^~~~~~~

Kconfig warnings: (for reference only)
WARNING: unmet direct dependencies detected for GET_FREE_REGION Depends on [n]: SPARSEMEM [=n]
Selected by [m]:
- RESOURCE_KUNIT_TEST [=m] && RUNTIME_TESTING_MENU [=y] && KUNIT [=m]

vim +/pci_map +81 drivers/pci/endpoint/functions/pci-epf-nvme.c

    69	
    70	/*
    71	 * Queue definition and mapping for the local PCI controller.
    72	 */
    73	struct pci_epf_nvme_queue {
    74		struct pci_epf_nvme	*epf_nvme;
    75	
    76		unsigned int		qflags;
    77		int			ref;
    78	
    79		phys_addr_t		pci_addr;
    80		size_t			pci_size;
  > 81		struct pci_epc_map	pci_map;
    82	
    83		u16			qid;
    84		u16			cqid;
    85		u16			size;
    86		u16			depth;
    87		u16			flags;
    88		u16			vector;
    89		u16			head;
    90		u16			tail;
    91		u16			phase;
    92		u32			db;
    93	
    94		size_t			qes;
    95	
    96		struct workqueue_struct	*cmd_wq;
    97		struct delayed_work	work;
    98		spinlock_t		lock;
    99		struct list_head	list;

100 };
101

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#12 lore raw

Re: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver

In reply to Damien Le Moal
kernel test robot noticed the following build errors:

Hi Damien,

kernel test robot noticed the following build errors:

[auto build test ERROR on pci/next]
[also build test ERROR on pci/for-linus linus/master v6.12-rc2 next-20241009] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Damien-Le-Moal/nvmet-rename-and-move-nvmet_get_log_page_len/20241007-124428
base:   https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link:    https://lore.kernel.org/r/20241007044351.157912-5-dlemoal%40kernel.org
patch subject: [PATCH v1 4/5] PCI: endpoint: Add NVMe endpoint function driver
config: x86_64-allyesconfig (https://download.01.org/0day-ci/archive/20241010/[email protected]/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241010/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> drivers/pci/endpoint/functions/pci-epf-nvme.c:81:21: error: field has incomplete type 'struct pci_epc_map'
      81 |         struct pci_epc_map      pci_map;
         |                                 ^
   drivers/pci/endpoint/functions/pci-epf-nvme.c:81:9: note: forward declaration of 'struct pci_epc_map'
      81 |         struct pci_epc_map      pci_map;
         |                ^
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:415:21: error: variable has incomplete type 'struct pci_epc_map'
     415 |         struct pci_epc_map map;
         |                            ^
   drivers/pci/endpoint/functions/pci-epf-nvme.c:81:9: note: forward declaration of 'struct pci_epc_map'
      81 |         struct pci_epc_map      pci_map;
         |                ^
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:419:8: error: call to undeclared function 'pci_epc_mem_map'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     419 |         ret = pci_epc_mem_map(epf->epc, epf->func_no, epf->vfunc_no,
         |               ^
>> drivers/pci/endpoint/functions/pci-epf-nvme.c:438:2: error: call to undeclared function 'pci_epc_mem_unmap'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     438 |         pci_epc_mem_unmap(epf->epc, epf->func_no, epf->vfunc_no, &map);
         |         ^
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1021:8: error: call to undeclared function 'pci_epc_mem_map'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1021 |         ret = pci_epc_mem_map(epf->epc, epf->func_no, epf->vfunc_no,
         |               ^
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1034:3: error: call to undeclared function 'pci_epc_mem_unmap'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1034 |                 pci_epc_mem_unmap(epf->epc, epf->func_no, epf->vfunc_no,
         |                 ^
   drivers/pci/endpoint/functions/pci-epf-nvme.c:1047:2: error: call to undeclared function 'pci_epc_mem_unmap'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    1047 |         pci_epc_mem_unmap(epf->epc, epf->func_no, epf->vfunc_no,
         |         ^

7 errors generated.

Kconfig warnings: (for reference only)
WARNING: unmet direct dependencies detected for MODVERSIONS
Depends on [n]: MODULES [=y] && !COMPILE_TEST [=y]
Selected by [y]:
- RANDSTRUCT_FULL [=y] && (CC_HAS_RANDSTRUCT [=y] || GCC_PLUGINS [=n]) && MODULES [=y]

vim +81 drivers/pci/endpoint/functions/pci-epf-nvme.c

    69	
    70	/*
    71	 * Queue definition and mapping for the local PCI controller.
    72	 */
    73	struct pci_epf_nvme_queue {
    74		struct pci_epf_nvme	*epf_nvme;
    75	
    76		unsigned int		qflags;
    77		int			ref;
    78	
    79		phys_addr_t		pci_addr;
    80		size_t			pci_size;
  > 81		struct pci_epc_map	pci_map;
    82	
    83		u16			qid;
    84		u16			cqid;
    85		u16			size;
    86		u16			depth;
    87		u16			flags;
    88		u16			vector;
    89		u16			head;
    90		u16			tail;
    91		u16			phase;
    92		u32			db;
    93	
    94		size_t			qes;
    95	
    96		struct workqueue_struct	*cmd_wq;
    97		struct delayed_work	work;
    98		spinlock_t		lock;
    99		struct list_head	list;

100 };
101

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#13 lore raw

PATCH 5/5 PCI: endpoint: Document the NVMe endpoint function driver

In reply to Damien Le Moal

Add the documentation files:
- Documentation/PCI/endpoint/pci-nvme-function.rst
- Documentation/PCI/endpoint/pci-nvme-howto.rst
- Documentation/PCI/endpoint/function/binding/pci-nvme.rst

To respectively document the NVMe PCI endpoint function driver internals, provide a user guide explaning how to setup an NVMe endpoint device and describe the NVMe endpoint function driver binding attributes.

---
 .../endpoint/function/binding/pci-nvme.rst    |  34 ++++
 Documentation/PCI/endpoint/index.rst          |   3 +
 .../PCI/endpoint/pci-nvme-function.rst        | 151 ++++++++++++++
 Documentation/PCI/endpoint/pci-nvme-howto.rst | 190 ++++++++++++++++++
 MAINTAINERS                                   |   2 +
 5 files changed, 380 insertions(+)
 create mode 100644 Documentation/PCI/endpoint/function/binding/pci-nvme.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-nvme-howto.rst
Documentation/PCI/endpoint/function/binding/pci-nvme.rst new file +34 -0
new file mode 100644
index 000000000000..d80293c08bcd
@@ -0,0 +1,34 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==========================
+PCI NVMe Endpoint Function
+==========================
+
+1) Create the function subdirectory pci_epf_nvme.0 in the
+pci_ep/functions/pci_epf_nvme directory of configfs.
+
+Standard EPF Configurable Fields:
+
+================   ===========================================================
+vendorid           Do not care (e.g. PCI_ANY_ID)
+deviceid           Do not care (e.g. PCI_ANY_ID)
+revid              Do not care
+progif_code	   Must be 0x02 (NVM Express)
+baseclass_code     Must be 0x1 (PCI_BASE_CLASS_STORAGE)
+subclass_code      Must be 0x08 (Non-Volatile Memory controller)
+cache_line_size    Do not care
+subsys_vendor_id   Do not care (e.g. PCI_ANY_ID)
+subsys_id          Do not care (e.g. PCI_ANY_ID)
+msi_interrupts     At least equal to the number of queue pairs desired
+msix_interrupts    At least equal to the number of queue pairs desired
+interrupt_pin      Interrupt PIN to use if MSI and MSI-X are not supported
+================   ===========================================================
+
+The NVMe EPF specific configurable fields are in the nvme subdirectory of the
+directory created in 1
+
+================   ===========================================================
+ctrl_opts          NVMe target connection parameters
+dma_enable         Enable (1) or disable (0) DMA transfers; default = 1
+mdts_kb            Maximum data transfer size in KiB; default = 128
+================   ===========================================================
Documentation/PCI/endpoint/index.rst +3 -0
index 4d2333e7ae06..764f1e8f81f2 100644
@@ -15,6 +15,9 @@ PCI Endpoint Framework
    pci-ntb-howto
    pci-vntb-function
    pci-vntb-howto
+   pci-nvme-function
+   pci-nvme-howto
 
    function/binding/pci-test
    function/binding/pci-ntb
+   function/binding/pci-nvme
Documentation/PCI/endpoint/pci-nvme-function.rst new file +151 -0

151 lines not on this page · raw message

Documentation/PCI/endpoint/pci-nvme-howto.rst new file +190 -0

190 lines not on this page · raw message

MAINTAINERS +2 -0
index c9b40621dbc1..18eeaba01996 100644
@@ -16559,6 +16559,8 @@ M:	Damien Le Moal <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Supported
+F:	Documentation/PCI/endpoint/function/binding/pci-nvme.rst
+F:	Documentation/PCI/endpoint/pci-nvme-*.rst
 F:	drivers/pci/endpoint/functions/pci-epf-nvme.c
 
 NVM EXPRESS DRIVER
Signature · 1 line
2.46.2