NVMe PCI endpoint target driver
Subsystems (8)
v4 cover NVMe PCI endpoint target driver
This patch series implements an NVMe target driver for the PCI transport using the PCI endpoint framework.
The first 4 patches of this series move and cleanup some nvme code that will be reused in following patches.
Patch 5 introduces the PCI transport type to allow setting up ports for the new PCI target controller driver. Patch 6 to 9 are improvements of the target core code to allow creating the PCI controller and processing its nvme commands without the need to rely on fabrics commands like the connect command to create the admin and I/O queues.
Patch 10 relaxes the SGL check in nvmet_req_init() to allow for PCI admin commands (which must use PRPs).
Patches 11 to 15 improve the set/get feature support of the target code to get closer to achieving NVMe specification compliance. These patches though do not implement support for some mandatory features.
Patch 16 is the main patch which introduces the NVMe PCI endpoint target driver. This patch commit message provides and overview of the driver design and operation.
Finally, patch 17 documents the NVMe PCI endpoint target driver and provides a user guide explaning how to setup an NVMe PCI endpoint device.
The patches are base on Linus 6.13-rc2 tree.
This driver has been extensively tested using a Radxa Rock5B board (RK3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board. However, this board does not support DMA channels for the PCI endpoint controller, leading to very poor performance.
Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:
+----------------------------------+------------------------+ | Workload | IOPS (BW) | +----------------------------------+------------------------+ | Rand read, 4KB, QD=1, 1 job | 14.3k IOPS | | Rand read, 4KB, QD=32, 1 job | 80.8k IOPS | | Rand read, 4KB, QD=32, 4 jobs | 131k IOPS | | Rand read, 128KB, QD=32, 1 job | 16.7k IOPS (2.18 GB/s) | | Rand read, 128KB, QD=32, 4 jobs | 17.4k IOPS (2.27 GB/s) | | Rand read, 512KB, QD=32, 1 job | 5380 IOPS (2.82 GB/s) | | Rand read, 512KB, QD=32, 4 jobs | 5206 IOPS (2.27 GB/s) | | Rand write, 128KB, QD=32, 1 job | 9617 IOPS (1.26 GB/s) | | Rand write, 128KB, QD=32, 4 jobs | 8405 IOPS (1.10 GB/s) | +----------------------------------+------------------------+
These results use the default MDTS of the NVMe enpoint driver of 512 KB.
This driver is not intended for production use but rather to be a playground for learning NVMe and exploring/testing new NVMe features while providing reasonably good performance.
Changes from v3:
- Added patch 1 which was missing from v3 and caused the 0day build failure
- Corrected a few typos in the documentation (patch 18)
- Added Christoph's review tag and Rick's tested tag
Changes from v2:
- Changed all preparatory patches before patch 16 to move more NVMe generic code out of the PCI endpoint target driver and into the target core.
- Changed patch 16 to use directly a target controller instead of a host controller. Many aspects of the command management and DMA transfer management have also been simplified, leading to higher performance.
- Change the documentation patch to match the above changes
Changes from v1:
- Added review tag to patch 1
- Modified patch 4 to:
- Add Rick's copyright notice
- Improve admin command handling (set_features command) to handle the number of queues feature (among others) to enable Windows host
- Improved SQ and CQ work items handling
Damien Le Moal (18):
nvme: Move opcode string helper functions declarations
nvmet: Add vendor_id and subsys_vendor_id subsystem attributes
nvmet: Export nvmet_update_cc() and nvmet_cc_xxx() helpers
nvmet: Introduce nvmet_get_cmd_effects_admin()
nvmet: Add drvdata field to struct nvmet_ctrl
nvme: Add PCI transport type
nvmet: Improve nvmet_alloc_ctrl() interface and implementation
nvmet: Introduce nvmet_req_transfer_len()
nvmet: Introduce nvmet_sq_create() and nvmet_cq_create()
nvmet: Add support for I/O queue management admin commands
nvmet: Do not require SGL for PCI target controller commands
nvmet: Introduce get/set_feature controller operations
nvmet: Implement host identifier set feature support
nvmet: Implement interrupt coalescing feature support
nvmet: Implement interrupt config feature support
nvmet: Implement arbitration feature support
nvmet: New NVMe PCI endpoint target driver
Documentation: Document the NVMe PCI endpoint target driver
Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 14 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 365 +++ Documentation/subsystem-apis.rst | 1 + drivers/nvme/host/nvme.h | 39 - drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/admin-cmd.c | 388 ++- drivers/nvme/target/configfs.c | 49 + drivers/nvme/target/core.c | 266 +- drivers/nvme/target/discovery.c | 17 + drivers/nvme/target/fabrics-cmd-auth.c | 14 +- drivers/nvme/target/fabrics-cmd.c | 101 +- drivers/nvme/target/nvmet.h | 110 +- drivers/nvme/target/pci-ep.c | 2626 +++++++++++++++++ include/linux/nvme.h | 42 + 17 files changed, 3897 insertions(+), 160 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst create mode 100644 drivers/nvme/target/pci-ep.c
base-commit: 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b
Signature · 1 line
2.47.1
PATCH v4 1/18 nvme: Move opcode string helper functions declarations
Move the declaration of all helper functions converting NVMe command opcodes and status codes into strings from drivers/nvme/host/nvme.h into include/linux/nvme.h, together with the commands definitions. This allows NVMe target drivers to call these functions without having to include a host header file.
--- drivers/nvme/host/nvme.h | 39 --------------------------------------- include/linux/nvme.h | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 39 deletions(-)
drivers/nvme/host/nvme.h +0 -39
@@ -1182,43 +1182,4 @@ static inline bool nvme_multi_css(struct nvme_ctrl *ctrl) return (ctrl->ctrl_config & NVME_CC_CSS_MASK) == NVME_CC_CSS_CSI; } -#ifdef CONFIG_NVME_VERBOSE_ERRORS -const char *nvme_get_error_status_str (u16 status); -const char *nvme_get_opcode_str (u8 opcode); -const char *nvme_get_admin_opcode_str (u8 opcode); -const char *nvme_get_fabrics_opcode_str (u8 opcode); -#else /* CONFIG_NVME_VERBOSE_ERRORS */ -static inline const char *nvme_get_error_status_str (u16 status) -{ -return "I/O Error" ; -} -static inline const char *nvme_get_opcode_str (u8 opcode) -{ -return "I/O Cmd" ; -} -static inline const char *nvme_get_admin_opcode_str (u8 opcode) -{ -return "Admin Cmd" ; -} - -static inline const char *nvme_get_fabrics_opcode_str (u8 opcode) -{ -return "Fabrics Cmd" ; -} -#endif /* CONFIG_NVME_VERBOSE_ERRORS */ - -static inline const char *nvme_opcode_str (int qid,u8 opcode) -{ -return qid ?nvme_get_opcode_str (opcode) : -nvme_get_admin_opcode_str (opcode); -} - -static inline const char *nvme_fabrics_opcode_str ( -int qid,const struct nvme_command *cmd) -{ -if (nvme_is_fabrics (cmd)) -return nvme_get_fabrics_opcode_str (cmd->fabrics.fctype); - -return nvme_opcode_str (qid, cmd->common.opcode); -}#endif /* _NVME_H */
include/linux/nvme.h +40 -0
@@ -1896,6 +1896,46 @@ static inline bool nvme_is_fabrics(const struct nvme_command *cmd)return cmd->common.opcode == nvme_fabrics_command; } +#ifdef CONFIG_NVME_VERBOSE_ERRORS +const char *nvme_get_error_status_str (u16 status); +const char *nvme_get_opcode_str (u8 opcode); +const char *nvme_get_admin_opcode_str (u8 opcode); +const char *nvme_get_fabrics_opcode_str (u8 opcode); +#else /* CONFIG_NVME_VERBOSE_ERRORS */ +static inline const char *nvme_get_error_status_str (u16 status) +{ +return "I/O Error" ; +} +static inline const char *nvme_get_opcode_str (u8 opcode) +{ +return "I/O Cmd" ; +} +static inline const char *nvme_get_admin_opcode_str (u8 opcode) +{ +return "Admin Cmd" ; +} + +static inline const char *nvme_get_fabrics_opcode_str (u8 opcode) +{ +return "Fabrics Cmd" ; +} +#endif /* CONFIG_NVME_VERBOSE_ERRORS */ + +static inline const char *nvme_opcode_str (int qid,u8 opcode) +{ +return qid ?nvme_get_opcode_str (opcode) : +nvme_get_admin_opcode_str (opcode); +} + +static inline const char *nvme_fabrics_opcode_str ( +int qid,const struct nvme_command *cmd) +{ +if (nvme_is_fabrics (cmd)) +return nvme_get_fabrics_opcode_str (cmd->fabrics.fctype); + +return nvme_opcode_str (qid, cmd->common.opcode); +} +struct nvme_error_slot {__le64 error_count;__le16 sqid;
Signature · 1 line
2.47.1
PATCH v4 2/18 nvmet: Add vendor_id and subsys_vendor_id subsystem attributes
In reply to Damien Le MoalDefine the new vendor_id and subsys_vendor_id configfs attribute for target subsystems. These attributes are respectively reported as the vid field and as the ssvid field of the identify controller data of a target controllers using the subsystem for which these attributes are set.
--- drivers/nvme/target/admin-cmd.c | 5 ++-- drivers/nvme/target/configfs.c | 45 +++++++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 2 ++ 3 files changed, 49 insertions(+), 3 deletions(-)
drivers/nvme/target/admin-cmd.c +2 -3
@@ -522,9 +522,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) goto out; } -/* XXX: figure out how to assign real vendors IDs. */ - id->vid =0 ; - id->ssvid =0 ; + id->vid =cpu_to_le16 (subsys->vendor_id); + id->ssvid =cpu_to_le16 (subsys->subsys_vendor_id);memcpy (id->sn, ctrl->subsys->serial,NVMET_SN_MAX_SIZE ); memcpy_and_pad(id->mn, sizeof(id->mn), subsys->model_number,
drivers/nvme/target/configfs.c +45 -0
57 lines not on this page · raw message
drivers/nvme/target/nvmet.h +2 -0
@@ -324,6 +324,8 @@ struct nvmet_subsys {struct config_group namespaces_group;struct config_group allowed_hosts_group; +u16 vendor_id; +u16 subsys_vendor_id;char *model_number;u32 ieee_oui;char *firmware_rev;
Signature · 1 line
2.47.1
PATCH v4 3/18 nvmet: Export nvmet_update_cc() and nvmet_cc_xxx() helpers
In reply to Damien Le MoalMake the function nvmet_update_cc() available to target drivers by exporting it. To also facilitate the manipulation of the cc register bits, move the inline helper functions nvmet_cc_en(), nvmet_cc_css(), nvmet_cc_mps(), nvmet_cc_ams(), nvmet_cc_shn(), nvmet_cc_iosqes(), and nvmet_cc_iocqes() from core.c to nvmet.h so that these functions can be reused in target controller drivers.
--- drivers/nvme/target/core.c | 36 +----------------------------------- drivers/nvme/target/nvmet.h | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 35 deletions(-)
drivers/nvme/target/core.c +1 -35
@@ -1166,41 +1166,6 @@ void nvmet_req_free_sgls(struct nvmet_req *req) }EXPORT_SYMBOL_GPL (nvmet_req_free_sgls); -static inline bool nvmet_cc_en (u32 cc) -{ -return (cc >>NVME_CC_EN_SHIFT ) &0x1 ; -} - -static inline u8 nvmet_cc_css (u32 cc) -{ -return (cc >>NVME_CC_CSS_SHIFT ) &0x7 ; -} - -static inline u8 nvmet_cc_mps (u32 cc) -{ -return (cc >>NVME_CC_MPS_SHIFT ) &0xf ; -} - -static inline u8 nvmet_cc_ams (u32 cc) -{ -return (cc >>NVME_CC_AMS_SHIFT ) &0x7 ; -} - -static inline u8 nvmet_cc_shn (u32 cc) -{ -return (cc >>NVME_CC_SHN_SHIFT ) &0x3 ; -} - -static inline u8 nvmet_cc_iosqes (u32 cc) -{ -return (cc >>NVME_CC_IOSQES_SHIFT ) &0xf ; -} - -static inline u8 nvmet_cc_iocqes (u32 cc) -{ -return (cc >>NVME_CC_IOCQES_SHIFT ) &0xf ; -} -static inline bool nvmet_css_supported (u8 cc_css) {switch (cc_css <<NVME_CC_CSS_SHIFT ) {
@@ -1277,6 +1242,7 @@ void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new) ctrl->csts &= ~NVME_CSTS_SHST_CMPLT ;mutex_unlock (&ctrl->lock); } +EXPORT_SYMBOL_GPL (nvmet_update_cc);static void nvmet_init_cap (struct nvmet_ctrl *ctrl) {
drivers/nvme/target/nvmet.h +35 -0
@@ -732,6 +732,41 @@ void nvmet_passthrough_override_cap(struct nvmet_ctrl *ctrl);u16 errno_to_nvme_status (struct nvmet_req *req,int errno);u16 nvmet_report_invalid_opcode (struct nvmet_req *req); +static inline bool nvmet_cc_en (u32 cc) +{ +return (cc >>NVME_CC_EN_SHIFT ) &0x1 ; +} + +static inline u8 nvmet_cc_css (u32 cc) +{ +return (cc >>NVME_CC_CSS_SHIFT ) &0x7 ; +} + +static inline u8 nvmet_cc_mps (u32 cc) +{ +return (cc >>NVME_CC_MPS_SHIFT ) &0xf ; +} + +static inline u8 nvmet_cc_ams (u32 cc) +{ +return (cc >>NVME_CC_AMS_SHIFT ) &0x7 ; +} + +static inline u8 nvmet_cc_shn (u32 cc) +{ +return (cc >>NVME_CC_SHN_SHIFT ) &0x3 ; +} + +static inline u8 nvmet_cc_iosqes (u32 cc) +{ +return (cc >>NVME_CC_IOSQES_SHIFT ) &0xf ; +} + +static inline u8 nvmet_cc_iocqes (u32 cc) +{ +return (cc >>NVME_CC_IOCQES_SHIFT ) &0xf ; +} +/* Convert a 32-bit number to a 16-bit 0's based number */ static inline __le16 to0based (u32 a) {
Signature · 1 line
2.47.1
PATCH v4 4/18 nvmet: Introduce nvmet_get_cmd_effects_admin()
In reply to Damien Le MoalIn order to have a logically better organized implementation of the effects log page, split out reporting the supported admin commands from nvmet_get_cmd_effects_nvm() into the new function nvmet_get_cmd_effects_admin().
--- drivers/nvme/target/admin-cmd.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/target/admin-cmd.c +6 -1
@@ -230,7 +230,7 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)nvmet_req_complete (req, status); } -static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log) +static void nvmet_get_cmd_effects_admin (struct nvme_effects_log *log) { log->acs[nvme_admin_get_log_page] = log->acs[nvme_admin_identify] =
@@ -240,7 +240,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) log->acs[nvme_admin_async_event] = log->acs[nvme_admin_keep_alive] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); +} +static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log) +{ log->iocs[nvme_cmd_read] = log->iocs[nvme_cmd_flush] = log->iocs[nvme_cmd_dsm] =
@@ -276,6 +279,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)switch (req->cmd->get_log_page.csi) {case NVME_CSI_NVM : +nvmet_get_cmd_effects_admin (log);nvmet_get_cmd_effects_nvm (log);break ;case NVME_CSI_ZNS :
@@ -283,6 +287,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req) status =NVME_SC_INVALID_IO_CMD_SET ; goto free; } +nvmet_get_cmd_effects_admin (log);nvmet_get_cmd_effects_nvm (log);nvmet_get_cmd_effects_zns (log);break ;
Signature · 1 line
2.47.1
PATCH v4 5/18 nvmet: Add drvdata field to struct nvmet_ctrl
In reply to Damien Le MoalAllow a target driver to attach private data to a target controller by adding the new field drvdata to struct nvmet_ctrl.
--- drivers/nvme/target/nvmet.h | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/target/nvmet.h +2 -0
@@ -238,6 +238,8 @@ struct nvmet_ctrl {struct nvmet_subsys *subsys;struct nvmet_sq **sqs; +void *drvdata; +bool reset_tbkas;struct mutex lock;
Signature · 1 line
2.47.1
PATCH v4 6/18 nvme: Add PCI transport type
In reply to Damien Le MoalDefine the transport type NVMF_TRTYPE_PCI for PCI endpoint targets. This transport type is defined using the value 0 which is reserved in the NVMe base specifications v2.1 (Figure 294). Given that struct nvmet_port are zeroed out on creation, to avoid having this transsport type becoming the new default, nvmet_referral_make() and nvmet_ports_make() are modified to initialize a port discovery address transport type field (disc_addr.trtype) to NVMF_TRTYPE_MAX.
Any port using this transport type is also skipped and not reported in the discovery log page (nvmet_execute_disc_get_log_page()).
The helper function nvmet_is_pci_ctrl() is also introduced to check if a target controller uses the PCI transport.
--- drivers/nvme/target/configfs.c | 4 ++++ drivers/nvme/target/discovery.c | 3 +++ drivers/nvme/target/nvmet.h | 5 +++++ include/linux/nvme.h | 1 + 4 files changed, 13 insertions(+)
drivers/nvme/target/configfs.c +4 -0
@@ -37,6 +37,7 @@ static struct nvmet_type_name_map nvmet_transport[] = { { NVMF_TRTYPE_RDMA, "rdma" }, { NVMF_TRTYPE_FC, "fc" }, { NVMF_TRTYPE_TCP, "tcp" }, + { NVMF_TRTYPE_PCI, "pci" }, { NVMF_TRTYPE_LOOP, "loop" }, };
@@ -46,6 +47,7 @@ static const struct nvmet_type_name_map nvmet_addr_family[] = { { NVMF_ADDR_FAMILY_IP6, "ipv6" }, { NVMF_ADDR_FAMILY_IB, "ib" }, { NVMF_ADDR_FAMILY_FC, "fc" }, + { NVMF_ADDR_FAMILY_PCI, "pci" }, { NVMF_ADDR_FAMILY_LOOP, "loop" }, };
@@ -1839,6 +1841,7 @@ static struct config_group *nvmet_referral_make(return ERR_PTR (-ENOMEM );INIT_LIST_HEAD (&port->entry); + port->disc_addr.trtype =NVMF_TRTYPE_MAX ;config_group_init_type_name (&port->group, name, &nvmet_referral_type);return &port->group;
@@ -2064,6 +2067,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group, port->inline_data_size =-1 ;/* < 0 == let the transport choose */ port->max_queue_size =-1 ;/* < 0 == let the transport choose */ + port->disc_addr.trtype =NVMF_TRTYPE_MAX ; port->disc_addr.portid =cpu_to_le16 (portid); port->disc_addr.adrfam =NVMF_ADDR_FAMILY_MAX ; port->disc_addr.treq =NVMF_TREQ_DISABLE_SQFLOW ;
drivers/nvme/target/discovery.c +3 -0
@@ -224,6 +224,9 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) }list_for_each_entry (r, &req->port->referrals, entry) { +if (r->disc_addr.trtype ==NVMF_TRTYPE_PCI ) +continue ; + nvmet_format_discovery_entry(hdr, r, NVME_DISC_SUBSYS_NAME, r->disc_addr.traddr,
drivers/nvme/target/nvmet.h +5 -0
@@ -693,6 +693,11 @@ static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)return subsys->type !=NVME_NQN_NVME ; } +static inline bool nvmet_is_pci_ctrl (struct nvmet_ctrl *ctrl) +{ +return ctrl->port->disc_addr.trtype ==NVMF_TRTYPE_PCI ; +} +#ifdef CONFIG_NVME_TARGET_PASSTHRU void nvmet_passthru_subsys_free (struct nvmet_subsys *subsys);int nvmet_passthru_ctrl_enable (struct nvmet_subsys *subsys);
include/linux/nvme.h +1 -0
@@ -64,6 +64,7 @@ enum {/* Transport Type codes for Discovery Log Page entry TRTYPE field */ enum { +NVMF_TRTYPE_PCI =0 ,/* PCI */ NVMF_TRTYPE_RDMA =1 ,/* RDMA */ NVMF_TRTYPE_FC =2 ,/* Fibre Channel */ NVMF_TRTYPE_TCP =3 ,/* TCP/IP */
Signature · 1 line
2.47.1
PATCH v4 7/18 nvmet: Improve nvmet_alloc_ctrl() interface and implementation
In reply to Damien Le MoalIntroduce struct nvmet_alloc_ctrl_args to define the arguments for the function nvmet_alloc_ctrl() to avoid the need for passing a pointer to a struct nvmet_req as an argument. This new data structure aggregates together the arguments that were passed to nvmet_alloc_ctrl() (subsysnqn, hostnqn and kato), together with the struct nvmet_req fields used by nvmet_alloc_ctrl(), that is, the fields port, p2p_client, and ops as input and the result and error_loc fields as output, as well as a status field. nvmet_alloc_ctrl() is also changed to return a pointer to the allocated and initialized controller structure instead of a status code, as the status is now returned through the status field of struct nvmet_alloc_ctrl_args.
The function nvmet_setup_p2p_ns_map() is changed to not take a pointer to a struct nvmet_req as argument, instead, directly specify the p2p_client device pointer needed as argument.
The code in nvmet_execute_admin_connect() that initializes a new target controller after allocating it is moved into nvmet_alloc_ctrl(). The code that sets up an admin queue for the controller (and the call to nvmet_install_queue()) remains in nvmet_execute_admin_connect().
Finally, nvmet_alloc_ctrl() is also exported to allow target drivers to use this function directly to allocate and initialize a new controller structure without the need to rely on a fabrics connect command request.
--- drivers/nvme/target/core.c | 83 ++++++++++++++++++++----------- drivers/nvme/target/fabrics-cmd.c | 58 ++++++++++----------- drivers/nvme/target/nvmet.h | 18 +++++-- 3 files changed, 94 insertions(+), 65 deletions(-)
drivers/nvme/target/core.c +53 -30
@@ -1350,15 +1350,15 @@ bool nvmet_host_allowed(struct nvmet_subsys *subsys, const char *hostnqn)* Note: ctrl->subsys->lock should be held when calling this function */ static void nvmet_setup_p2p_ns_map (struct nvmet_ctrl *ctrl, -struct nvmet_req *req) +struct device *p2p_client) {struct nvmet_ns *ns;unsigned long idx; -if (!req->p2p_client) +if (!p2p_client)return ; - ctrl->p2p_client =get_device (req->p2p_client); + ctrl->p2p_client =get_device (p2p_client);xa_for_each (&ctrl->subsys->namespaces, idx, ns)nvmet_p2pmem_ns_add_p2p (ctrl, ns);
@@ -1387,45 +1387,44 @@ static void nvmet_fatal_error_handler(struct work_struct *work) ctrl->ops->delete_ctrl (ctrl); } -u16 nvmet_alloc_ctrl (const char *subsysnqn,const char *hostnqn, -struct nvmet_req *req,u32 kato,struct nvmet_ctrl **ctrlp, -uuid_t *hostid) +struct nvmet_ctrl *nvmet_alloc_ctrl (struct nvmet_alloc_ctrl_args *args) {struct nvmet_subsys *subsys;struct nvmet_ctrl *ctrl; +u32 kato = args->kato; +u8 dhchap_status;int ret; -u16 status; - status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; - subsys =nvmet_find_get_subsys (req->port, subsysnqn); + args->status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; + subsys =nvmet_find_get_subsys (args->port, args->subsysnqn);if (!subsys) {pr_warn ("connect request for invalid subsystem %s!\n" , - subsysnqn); - req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (subsysnqn); - req->error_loc = offsetof(struct nvme_common_command , dptr); - goto out; + args->subsysnqn); + args->result =IPO_IATTR_CONNECT_DATA (subsysnqn); + args->error_loc = offsetof(struct nvme_common_command , dptr); +return NULL ; }down_read (&nvmet_config_sem); -if (!nvmet_host_allowed (subsys, hostnqn)) { +if (!nvmet_host_allowed (subsys, args->hostnqn)) {pr_info ("connect by host %s for subsystem %s not allowed\n" , - hostnqn, subsysnqn); - req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (hostnqn); + args->hostnqn, args->subsysnqn); + args->result =IPO_IATTR_CONNECT_DATA (hostnqn);up_read (&nvmet_config_sem); - status =NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; - req->error_loc = offsetof(struct nvme_common_command , dptr); + args->status =NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; + args->error_loc = offsetof(struct nvme_common_command , dptr); goto out_put_subsystem; }up_read (&nvmet_config_sem); - status =NVME_SC_INTERNAL ; + args->status =NVME_SC_INTERNAL ; ctrl =kzalloc (sizeof (*ctrl),GFP_KERNEL );if (!ctrl) goto out_put_subsystem;mutex_init (&ctrl->lock); - ctrl->port = req->port; - ctrl->ops = req->ops; + ctrl->port = args->port; + ctrl->ops = args->ops;#ifdef CONFIG_NVME_TARGET_PASSTHRU /* By default, set loop targets to clear IDS by default */
@@ -1439,8 +1438,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,INIT_WORK (&ctrl->fatal_err_work, nvmet_fatal_error_handler);INIT_DELAYED_WORK (&ctrl->ka_work, nvmet_keep_alive_timer); -memcpy (ctrl->subsysnqn, subsysnqn,NVMF_NQN_SIZE ); -memcpy (ctrl->hostnqn, hostnqn,NVMF_NQN_SIZE ); +memcpy (ctrl->subsysnqn, args->subsysnqn,NVMF_NQN_SIZE ); +memcpy (ctrl->hostnqn, args->hostnqn,NVMF_NQN_SIZE );kref_init (&ctrl->ref); ctrl->subsys = subsys;
@@ -1463,12 +1462,12 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, subsys->cntlid_min, subsys->cntlid_max, GFP_KERNEL);if (ret <0 ) { - status =NVME_SC_CONNECT_CTRL_BUSY |NVME_STATUS_DNR ; + args->status =NVME_SC_CONNECT_CTRL_BUSY |NVME_STATUS_DNR ; goto out_free_sqs; } ctrl->cntlid = ret; -uuid_copy (&ctrl->hostid, hostid); +uuid_copy (&ctrl->hostid, args->hostid);/* * Discovery controllers may use some arbitrary high value
@@ -1490,12 +1489,35 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,if (ret) goto init_pr_fail;list_add_tail (&ctrl->subsys_entry, &subsys->ctrls); -nvmet_setup_p2p_ns_map (ctrl, req); +nvmet_setup_p2p_ns_map (ctrl, args->p2p_client);nvmet_debugfs_ctrl_setup (ctrl);mutex_unlock (&subsys->lock); - *ctrlp = ctrl; -return 0 ; +if (args->hostid) +uuid_copy (&ctrl->hostid, args->hostid); + + dhchap_status =nvmet_setup_auth (ctrl); +if (dhchap_status) { +pr_err ("Failed to setup authentication, dhchap status %u\n" , + dhchap_status); +nvmet_ctrl_put (ctrl); +if (dhchap_status ==NVME_AUTH_DHCHAP_FAILURE_FAILED ) + args->status = +NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; +else + args->status =NVME_SC_INTERNAL ; +return NULL ; + } + + args->status =NVME_SC_SUCCESS ; + +pr_info ("Created %s controller %d for subsystem %s for NQN %s%s%s.\n" , +nvmet_is_disc_subsys (ctrl->subsys) ?"discovery" :"nvm" , + ctrl->cntlid, ctrl->subsys->subsysnqn, ctrl->hostnqn, + ctrl->pi_support ?" T10-PI is enabled" :"" , +nvmet_has_auth (ctrl) ?" with DH-HMAC-CHAP" :"" ); + +return ctrl; init_pr_fail:mutex_unlock (&subsys->lock);
@@ -1509,9 +1531,9 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,kfree (ctrl); out_put_subsystem:nvmet_subsys_put (subsys); -out: -return status; +return NULL ; } +EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl);static void nvmet_ctrl_free (struct kref *ref) {
@@ -1547,6 +1569,7 @@ void nvmet_ctrl_put(struct nvmet_ctrl *ctrl) {kref_put (&ctrl->ref, nvmet_ctrl_free); } +EXPORT_SYMBOL_GPL (nvmet_ctrl_put);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl) {
drivers/nvme/target/fabrics-cmd.c +26 -32
99 lines not on this page · raw message
drivers/nvme/target/nvmet.h +15 -3
@@ -549,9 +549,21 @@ int nvmet_sq_init(struct nvmet_sq *sq);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl);void nvmet_update_cc (struct nvmet_ctrl *ctrl,u32 new); -u16 nvmet_alloc_ctrl (const char *subsysnqn,const char *hostnqn, -struct nvmet_req *req,u32 kato,struct nvmet_ctrl **ctrlp, -uuid_t *hostid); + +struct nvmet_alloc_ctrl_args { +struct nvmet_port *port; +char *subsysnqn; +char *hostnqn; +uuid_t *hostid; +const struct nvmet_fabrics_ops *ops; +struct device *p2p_client; +u32 kato; +u32 result; +u16 error_loc; +u16 status; +}; + +struct nvmet_ctrl *nvmet_alloc_ctrl (struct nvmet_alloc_ctrl_args *args);struct nvmet_ctrl *nvmet_ctrl_find_get (const char *subsysnqn,const char *hostnqn,u16 cntlid,struct nvmet_req *req);
Signature · 1 line
2.47.1
PATCH v4 8/18 nvmet: Introduce nvmet_req_transfer_len()
In reply to Damien Le MoalAdd the new function nvmet_req_transfer_len() to parse a request command to extract the transfer length of the command. This function implementation relies on multiple helper functions for parsing I/O commands (nvmet_io_cmd_transfer_len()), admin commands (nvmet_admin_cmd_data_len()) and fabrics connect commands (nvmet_connect_cmd_data_len).
--- drivers/nvme/target/admin-cmd.c | 21 +++++++++++++ drivers/nvme/target/core.c | 37 ++++++++++++++++++++++ drivers/nvme/target/discovery.c | 14 +++++++++ drivers/nvme/target/fabrics-cmd-auth.c | 14 +++++++-- drivers/nvme/target/fabrics-cmd.c | 43 ++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 8 +++++ 6 files changed, 135 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +21 -0
@@ -1296,6 +1296,27 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)nvmet_req_complete (req, status); } +u32 nvmet_admin_cmd_data_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; + +if (nvme_is_fabrics (cmd)) +return nvmet_fabrics_admin_cmd_data_len (req); +if (nvmet_is_disc_subsys (nvmet_req_subsys (req))) +return nvmet_discovery_cmd_data_len (req); + +switch (cmd->common.opcode) { +case nvme_admin_get_log_page: +return nvmet_get_log_page_len (cmd); +case nvme_admin_identify: +return NVME_IDENTIFY_DATA_SIZE ; +case nvme_admin_get_features: +return nvmet_feat_data_len (req,le32_to_cpu (cmd->common.cdw10)); +default : +return 0 ; + } +} +u16 nvmet_parse_admin_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
drivers/nvme/target/core.c +37 -0
@@ -911,6 +911,33 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req)return 0 ; } +static u32 nvmet_io_cmd_transfer_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; +u32 metadata_len =0 ; + +if (nvme_is_fabrics (cmd)) +return nvmet_fabrics_io_cmd_data_len (req); + +if (!req->ns) +return 0 ; + +switch (req->cmd->common.opcode) { +case nvme_cmd_read: +case nvme_cmd_write: +case nvme_cmd_zone_append: +if (req->sq->ctrl->pi_support &&nvmet_ns_has_pi (req->ns)) + metadata_len =nvmet_rw_metadata_len (req); +return nvmet_rw_data_len (req) + metadata_len; +case nvme_cmd_dsm: +return nvmet_dsm_len (req); +case nvme_cmd_zone_mgmt_recv: +return (le32_to_cpu (req->cmd->zmr.numd) +1 ) <<2 ; +default : +return 0 ; + } +} +static u16 nvmet_parse_io_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
@@ -1059,6 +1086,16 @@ void nvmet_req_uninit(struct nvmet_req *req) }EXPORT_SYMBOL_GPL (nvmet_req_uninit); +size_t nvmet_req_transfer_len (struct nvmet_req *req) +{ +if (likely (req->sq->qid !=0 )) +return nvmet_io_cmd_transfer_len (req); +if (unlikely (!req->sq->ctrl)) +return nvmet_connect_cmd_data_len (req); +return nvmet_admin_cmd_data_len (req); +} +EXPORT_SYMBOL_GPL (nvmet_req_transfer_len); +bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len) {if (unlikely (len != req->transfer_len)) {
drivers/nvme/target/discovery.c +14 -0
@@ -355,6 +355,20 @@ static void nvmet_execute_disc_get_features(struct nvmet_req *req)nvmet_req_complete (req, stat); } +u32 nvmet_discovery_cmd_data_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; + +switch (cmd->common.opcode) { +case nvme_admin_get_log_page: +return nvmet_get_log_page_len (req->cmd); +case nvme_admin_identify: +return NVME_IDENTIFY_DATA_SIZE ; +default : +return 0 ; + } +} +u16 nvmet_parse_discovery_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
drivers/nvme/target/fabrics-cmd-auth.c +12 -2
@@ -179,6 +179,11 @@ static u8 nvmet_auth_failure2(void *d)return data->rescode_exp; } +u32 nvmet_auth_send_data_len (struct nvmet_req *req) +{ +return le32_to_cpu (req->cmd->auth_send.tl); +} +void nvmet_execute_auth_send (struct nvmet_req *req) {struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -206,7 +211,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) offsetof(struct nvmf_auth_send_command , spsp1); goto done; } - tl =le32_to_cpu (req->cmd->auth_send.tl); + tl =nvmet_auth_send_data_len (req);if (!tl) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
@@ -429,6 +434,11 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al) data->rescode_exp = req->sq->dhchap_status; } +u32 nvmet_auth_receive_data_len (struct nvmet_req *req) +{ +return le32_to_cpu (req->cmd->auth_receive.al); +} +void nvmet_execute_auth_receive (struct nvmet_req *req) {struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -454,7 +464,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) offsetof(struct nvmf_auth_receive_command , spsp1); goto done; } - al =le32_to_cpu (req->cmd->auth_receive.al); + al =nvmet_auth_receive_data_len (req);if (!al) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
drivers/nvme/target/fabrics-cmd.c +43 -0
61 lines not on this page · raw message
drivers/nvme/target/nvmet.h +8 -0
@@ -517,18 +517,24 @@ void nvmet_start_keep_alive_timer(struct nvmet_ctrl *ctrl);void nvmet_stop_keep_alive_timer (struct nvmet_ctrl *ctrl);u16 nvmet_parse_connect_cmd (struct nvmet_req *req); +u32 nvmet_connect_cmd_data_len (struct nvmet_req *req);void nvmet_bdev_set_limits (struct block_device *bdev,struct nvme_id_ns *id);u16 nvmet_bdev_parse_io_cmd (struct nvmet_req *req);u16 nvmet_file_parse_io_cmd (struct nvmet_req *req);u16 nvmet_bdev_zns_parse_io_cmd (struct nvmet_req *req); +u32 nvmet_admin_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_admin_cmd (struct nvmet_req *req); +u32 nvmet_discovery_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_discovery_cmd (struct nvmet_req *req);u16 nvmet_parse_fabrics_admin_cmd (struct nvmet_req *req); +u32 nvmet_fabrics_admin_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_fabrics_io_cmd (struct nvmet_req *req); +u32 nvmet_fabrics_io_cmd_data_len (struct nvmet_req *req);bool nvmet_req_init (struct nvmet_req *req,struct nvmet_cq *cq,struct nvmet_sq *sq,const struct nvmet_fabrics_ops *ops);void nvmet_req_uninit (struct nvmet_req *req); +size_t nvmet_req_transfer_len (struct nvmet_req *req);bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len);bool nvmet_check_data_len_lte (struct nvmet_req *req,size_t data_len);void nvmet_req_complete (struct nvmet_req *req,u16 status);
@@ -822,7 +828,9 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio) }#ifdef CONFIG_NVME_TARGET_AUTH +u32 nvmet_auth_send_data_len (struct nvmet_req *req);void nvmet_execute_auth_send (struct nvmet_req *req); +u32 nvmet_auth_receive_data_len (struct nvmet_req *req);void nvmet_execute_auth_receive (struct nvmet_req *req);int nvmet_auth_set_key (struct nvmet_host *host,const char *secret,bool set_ctrl);
Signature · 1 line
2.47.1
PATCH v4 9/18 nvmet: Introduce nvmet_sq_create() and nvmet_cq_create()
In reply to Damien Le MoalIntroduce the new functions nvmet_sq_create() and nvmet_cq_create() to allow a target driver to initialize and setup admin and IO queues directly, without needing to execute connect fabrics commands. The helper functions nvmet_check_cqid() and nvmet_check_sqid() are implemented to check the correctness of SQ and CQ IDs when nvmet_sq_create() and nvmet_cq_create() are called.
nvmet_sq_create() and nvmet_cq_create() are primarily intended for use with PCI target controller drivers and thus are not well integrated with the current queue creation of fabrics controllers using the connect command. These fabrices drivers are not modified to use these functions. This simple implementation of SQ and CQ management for PCI target controller drivers does not allow multiple SQs to share the same CQ, similarly to other fabrics transports. This is a specification violation. A more involved set of changes will follow to add support for this required completion queue sharing feature.
--- drivers/nvme/target/core.c | 83 +++++++++++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 6 +++ 2 files changed, 89 insertions(+)
drivers/nvme/target/core.c +83 -0
@@ -818,6 +818,89 @@ static void nvmet_confirm_sq(struct percpu_ref *ref)complete (&sq->confirm_done); } +u16 nvmet_check_cqid (struct nvmet_ctrl *ctrl,u16 cqid) +{ +if (!ctrl->sqs) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + +if (cqid > ctrl->subsys->max_qid) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +/* +* Note: For PCI controllers, the NVMe specifications allows multiple +* SQs to share a single CQ. However, we do not support this yet, so +* check that there is no SQ defined for a CQ. If one exist, then the +* CQ ID is invalid for creation as well as when the CQ is being +* deleted (as that would mean that the SQ was not deleted before the +* CQ). +*/ +if (ctrl->sqs[cqid]) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +return NVME_SC_SUCCESS ; +} + +u16 nvmet_cq_create (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq, +u16 qid,u16 size) +{ +u16 status; + + status =nvmet_check_cqid (ctrl, qid); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_cq_setup (ctrl, cq, qid, size); + +return NVME_SC_SUCCESS ; +} +EXPORT_SYMBOL_GPL (nvmet_cq_create); + +u16 nvmet_check_sqid (struct nvmet_ctrl *ctrl,u16 sqid, +bool create) +{ +if (!ctrl->sqs) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + +if (sqid > ctrl->subsys->max_qid) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +if ((create && ctrl->sqs[sqid]) || + (!create && !ctrl->sqs[sqid])) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +return NVME_SC_SUCCESS ; +} + +u16 nvmet_sq_create (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq, +u16 sqid,u16 size) +{ +u16 status; +int ret; + +if (!kref_get_unless_zero (&ctrl->ref)) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + + status =nvmet_check_sqid (ctrl, sqid, true); +if (status !=NVME_SC_SUCCESS ) +return status; + + ret =nvmet_sq_init (sq); +if (ret) { + status =NVME_SC_INTERNAL |NVME_STATUS_DNR ; + goto ctrl_put; + } + +nvmet_sq_setup (ctrl, sq, sqid, size); + sq->ctrl = ctrl; + +return NVME_SC_SUCCESS ; + +ctrl_put: +nvmet_ctrl_put (ctrl); +return status; +} +EXPORT_SYMBOL_GPL (nvmet_sq_create); +void nvmet_sq_destroy (struct nvmet_sq *sq) {struct nvmet_ctrl *ctrl = sq->ctrl;
drivers/nvme/target/nvmet.h +6 -0
@@ -545,10 +545,16 @@ void nvmet_execute_set_features(struct nvmet_req *req);void nvmet_execute_get_features (struct nvmet_req *req);void nvmet_execute_keep_alive (struct nvmet_req *req); +u16 nvmet_check_cqid (struct nvmet_ctrl *ctrl,u16 cqid);void nvmet_cq_setup (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq,u16 qid,u16 size); +u16 nvmet_cq_create (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq,u16 qid, +u16 size); +u16 nvmet_check_sqid (struct nvmet_ctrl *ctrl,u16 sqid,bool create);void nvmet_sq_setup (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq,u16 qid,u16 size); +u16 nvmet_sq_create (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq,u16 qid, +u16 size);void nvmet_sq_destroy (struct nvmet_sq *sq);int nvmet_sq_init (struct nvmet_sq *sq);
Signature · 1 line
2.47.1
PATCH v4 10/18 nvmet: Add support for I/O queue management admin commands
In reply to Damien Le MoalThe I/O submission queue management admin commands (nvme_admin_delete_sq, nvme_admin_create_sq, nvme_admin_delete_cq, and nvme_admin_create_cq) are mandatory admin commands for I/O controllers using the PCI transport, that is, support for these commands is mandatory for a a PCI target I/O controller.
Implement support for these commands by adding the functions nvmet_execute_delete_sq(), nvmet_execute_create_sq(), nvmet_execute_delete_cq() and nvmet_execute_create_cq() to set as the execute method of requests for these commands. These functions will return an invalid opcode error for any controller that is not a PCI target controller. Support for the I/O queue management commands is also reported in the command effect log of PCI target controllers (using nvmet_get_cmd_effects_admin()).
Each management command is backed by a controller fabric operation that can be defined by a PCI target controller driver to setup I/O queues using nvmet_sq_create() and nvmet_cq_create() or delete I/O queues using nvmet_sq_destroy().
As noted in a comment in nvmet_execute_create_sq(), we do not yet support sharing a single CQ between multiple SQs.
--- drivers/nvme/target/admin-cmd.c | 165 +++++++++++++++++++++++++++++++- drivers/nvme/target/nvmet.h | 8 ++ 2 files changed, 170 insertions(+), 3 deletions(-)
drivers/nvme/target/admin-cmd.c +162 -3
@@ -12,6 +12,142 @@#include <linux/unaligned.h> #include "nvmet.h" +static void nvmet_execute_delete_sq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u16 sqid =le16_to_cpu (req->cmd->delete_queue.qid); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!sqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_sqid (ctrl, sqid, false); +if (status !=NVME_SC_SUCCESS ) + goto complete; + + status = ctrl->ops->delete_sq (ctrl, sqid); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_create_sq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +u16 sqid =le16_to_cpu (cmd->create_sq.sqid); +u16 cqid =le16_to_cpu (cmd->create_sq.cqid); +u16 sq_flags =le16_to_cpu (cmd->create_sq.sq_flags); +u16 qsize =le16_to_cpu (cmd->create_sq.qsize); +u64 prp1 =le64_to_cpu (cmd->create_sq.prp1); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!sqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_sqid (ctrl, sqid, true); +if (status !=NVME_SC_SUCCESS ) + goto complete; + +/* +* Note: The NVMe specification allows multiple SQs to use the same CQ. +* However, the target code does not really support that. So for now, +* prevent this and fail the command if sqid and cqid are different. +*/ +if (!cqid || cqid != sqid) { +pr_err ("SQ %u: Unsupported CQID %u\n" , sqid, cqid); + status =NVME_SC_CQ_INVALID |NVME_STATUS_DNR ; + goto complete; + } + +if (!qsize || qsize >NVME_CAP_MQES (ctrl->cap)) { + status =NVME_SC_QUEUE_SIZE |NVME_STATUS_DNR ; + goto complete; + } + + status = ctrl->ops->create_sq (ctrl, sqid, sq_flags, qsize, prp1); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_delete_cq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u16 cqid =le16_to_cpu (req->cmd->delete_queue.qid); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!cqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_cqid (ctrl, cqid); +if (status !=NVME_SC_SUCCESS ) + goto complete; + + status = ctrl->ops->delete_cq (ctrl, cqid); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_create_cq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +u16 cqid =le16_to_cpu (cmd->create_cq.cqid); +u16 cq_flags =le16_to_cpu (cmd->create_cq.cq_flags); +u16 qsize =le16_to_cpu (cmd->create_cq.qsize); +u16 irq_vector =le16_to_cpu (cmd->create_cq.irq_vector); +u64 prp1 =le64_to_cpu (cmd->create_cq.prp1); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!cqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_cqid (ctrl, cqid); +if (status !=NVME_SC_SUCCESS ) + goto complete; + +if (!qsize || qsize >NVME_CAP_MQES (ctrl->cap)) { + status =NVME_SC_QUEUE_SIZE |NVME_STATUS_DNR ; + goto complete; + } + + status = ctrl->ops->create_cq (ctrl, cqid, cq_flags, qsize, + prp1, irq_vector); + +complete: +nvmet_req_complete (req, status); +} +u32 nvmet_get_log_page_len (struct nvme_command *cmd) {u32 len =le16_to_cpu (cmd->get_log_page.numdu);
@@ -230,8 +366,18 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)nvmet_req_complete (req, status); } -static void nvmet_get_cmd_effects_admin (struct nvme_effects_log *log) +static void nvmet_get_cmd_effects_admin (struct nvmet_ctrl *ctrl, +struct nvme_effects_log *log) { +/* For a PCI target controller, advertize support for the . */ +if (nvmet_is_pci_ctrl (ctrl)) { + log->acs[nvme_admin_delete_sq] = + log->acs[nvme_admin_create_sq] = + log->acs[nvme_admin_delete_cq] = + log->acs[nvme_admin_create_cq] = +cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); + } + log->acs[nvme_admin_get_log_page] = log->acs[nvme_admin_identify] = log->acs[nvme_admin_abort_cmd] =
@@ -268,6 +414,7 @@ static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)static void nvmet_execute_get_log_cmd_effects_ns (struct nvmet_req *req) { +struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvme_effects_log *log;u16 status =NVME_SC_SUCCESS ;
@@ -279,7 +426,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)switch (req->cmd->get_log_page.csi) {case NVME_CSI_NVM : -nvmet_get_cmd_effects_admin (log); +nvmet_get_cmd_effects_admin (ctrl, log);nvmet_get_cmd_effects_nvm (log);break ;case NVME_CSI_ZNS :
@@ -287,7 +434,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req) status =NVME_SC_INVALID_IO_CMD_SET ; goto free; } -nvmet_get_cmd_effects_admin (log); +nvmet_get_cmd_effects_admin (ctrl, log);nvmet_get_cmd_effects_nvm (log);nvmet_get_cmd_effects_zns (log);break ;
@@ -1335,9 +1482,21 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)return nvmet_parse_passthru_admin_cmd (req);switch (cmd->common.opcode) { +case nvme_admin_delete_sq: + req->execute = nvmet_execute_delete_sq; +return 0 ; +case nvme_admin_create_sq: + req->execute = nvmet_execute_create_sq; +return 0 ;case nvme_admin_get_log_page: req->execute = nvmet_execute_get_log_page;return 0 ; +case nvme_admin_delete_cq: + req->execute = nvmet_execute_delete_cq; +return 0 ; +case nvme_admin_create_cq: + req->execute = nvmet_execute_create_cq; +return 0 ;case nvme_admin_identify: req->execute = nvmet_execute_identify;return 0 ;
drivers/nvme/target/nvmet.h +8 -0
@@ -408,6 +408,14 @@ struct nvmet_fabrics_ops {void (*discovery_chg)(struct nvmet_port *port);u8 (*get_mdts)(const struct nvmet_ctrl *ctrl);u16 (*get_max_queue_size)(const struct nvmet_ctrl *ctrl); + +/* Operations mandatory for PCI target controllers */ +u16 (*create_sq)(struct nvmet_ctrl *ctrl,u16 sqid,u16 flags, +u16 qsize,u64 prp1); +u16 (*delete_sq)(struct nvmet_ctrl *ctrl,u16 sqid); +u16 (*create_cq)(struct nvmet_ctrl *ctrl,u16 cqid,u16 flags, +u16 qsize,u64 prp1,u16 irq_vector); +u16 (*delete_cq)(struct nvmet_ctrl *ctrl,u16 cqid); };#define NVMET_MAX_INLINE_BIOVEC 8
Signature · 1 line
2.47.1
PATCH v4 11/18 nvmet: Do not require SGL for PCI target controller commands
In reply to Damien Le MoalSupport for SGL is optional for the PCI transport. Modify nvmet_req_init() to not require the NVME_CMD_SGL_METABUF command flag to be set if the target controller transport type is NVMF_TRTYPE_PCI. In addition to this, the NVMe base specification v2.1 mandate that all admin commands use PRP, that is, have CDW0.PSDT cleared to 0. Modify nvmet_parse_admin_cmd() to check this.
Finally, modify nvmet_check_transfer_len() and nvmet_check_data_len_lte() to return the appropriate error status depending on the command using SGL or PRP. Since for fabrics nvmet_req_init() checks that a command uses SGL, always, this change affects only PCI target controllers.
--- drivers/nvme/target/admin-cmd.c | 5 +++++ drivers/nvme/target/core.c | 27 +++++++++++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
drivers/nvme/target/admin-cmd.c +5 -0
@@ -1478,6 +1478,11 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)if (unlikely (ret))return ret; +/* For PCI controllers, admin commands shall not use SGL. */ +if (nvmet_is_pci_ctrl (req->sq->ctrl) && !req->sq->qid && + cmd->common.flags &NVME_CMD_SGL_ALL ) +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; +if (nvmet_is_passthru_req (req))return nvmet_parse_passthru_admin_cmd (req);
drivers/nvme/target/core.c +21 -6
@@ -1122,12 +1122,15 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,/* * For fabrics, PSDT field shall describe metadata pointer (MPTR) that * contains an address of a single contiguous physical buffer that is -* byte aligned. +* byte aligned. For PCI controllers, this is optional so not enforced. */ if (unlikely ((flags &NVME_CMD_SGL_ALL ) !=NVME_CMD_SGL_METABUF )) { - req->error_loc = offsetof(struct nvme_common_command , flags); - status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; - goto fail; +if (!req->sq->ctrl || !nvmet_is_pci_ctrl (req->sq->ctrl)) { + req->error_loc = + offsetof(struct nvme_common_command , flags); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto fail; + } }if (unlikely (!req->sq->ctrl))
@@ -1182,8 +1185,14 @@ EXPORT_SYMBOL_GPL(nvmet_req_transfer_len);bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len) {if (unlikely (len != req->transfer_len)) { +u16 status; + req->error_loc = offsetof(struct nvme_common_command , dptr); -nvmet_req_complete (req,NVME_SC_SGL_INVALID_DATA |NVME_STATUS_DNR ); +if (req->cmd->common.flags &NVME_CMD_SGL_ALL ) + status =NVME_SC_SGL_INVALID_DATA ; +else + status =NVME_SC_INVALID_FIELD ; +nvmet_req_complete (req, status |NVME_STATUS_DNR );return false; }
@@ -1194,8 +1203,14 @@ EXPORT_SYMBOL_GPL(nvmet_check_transfer_len);bool nvmet_check_data_len_lte (struct nvmet_req *req,size_t data_len) {if (unlikely (data_len > req->transfer_len)) { +u16 status; + req->error_loc = offsetof(struct nvme_common_command , dptr); -nvmet_req_complete (req,NVME_SC_SGL_INVALID_DATA |NVME_STATUS_DNR ); +if (req->cmd->common.flags &NVME_CMD_SGL_ALL ) + status =NVME_SC_SGL_INVALID_DATA ; +else + status =NVME_SC_INVALID_FIELD ; +nvmet_req_complete (req, status |NVME_STATUS_DNR );return false; }
Signature · 1 line
2.47.1
PATCH v4 12/18 nvmet: Introduce get/set_feature controller operations
In reply to Damien Le MoalThe implementation of some features cannot always be done generically by the target core code. Arbitraion and IRQ coalescing features are examples of such features: their implementation must be provided (at least partially) by the target controller driver.
Introduce the set_feature() and get_feature() controller fabrics operations (in struct nvmet_fabrics_ops) to allow supporting such features.
--- drivers/nvme/target/nvmet.h | 4 ++++ 1 file changed, 4 insertions(+)
drivers/nvme/target/nvmet.h +4 -0
@@ -416,6 +416,10 @@ struct nvmet_fabrics_ops {u16 (*create_cq)(struct nvmet_ctrl *ctrl,u16 cqid,u16 flags,u16 qsize,u64 prp1,u16 irq_vector);u16 (*delete_cq)(struct nvmet_ctrl *ctrl,u16 cqid); +u16 (*set_feature)(const struct nvmet_ctrl *ctrl,u8 feat, +void *feat_data); +u16 (*get_feature)(const struct nvmet_ctrl *ctrl,u8 feat, +void *feat_data); };#define NVMET_MAX_INLINE_BIOVEC 8
Signature · 1 line
2.47.1
PATCH v4 13/18 nvmet: Implement host identifier set feature support
In reply to Damien Le MoalThe NVMe specifications mandate support for the host identifier set_features for controllers that also supports reservations. Satisfy this requirement by implementing handling of the NVME_FEAT_HOST_ID feature for the nvme_set_features command. This implementation is for now effective only for PCI target controllers. For other controller types, the set features command is failed with a NVME_SC_CMD_SEQ_ERROR status as before.
As noted in the code, 128 bits host identifiers are supported since the NVMe bass specifications version 2.1 indicate in section 5.1.25.1.28.1 that "The controller may support a 64-bit Host Identifier...".
The RHII (Reservations and Host Identifier Interaction) bit of the controller attribute (ctratt) field of the identify controller data is also set to indicate that a host ID of "0" is supported but that the host ID must be a non-zero value to use reservations.
--- drivers/nvme/target/admin-cmd.c | 35 +++++++++++++++++++++++++++++---- include/linux/nvme.h | 1 + 2 files changed, 32 insertions(+), 4 deletions(-)
drivers/nvme/target/admin-cmd.c +31 -4
@@ -659,7 +659,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvmet_subsys *subsys = ctrl->subsys;struct nvme_id_ctrl *id; -u32 cmd_capsule_size; +u32 cmd_capsule_size, ctratt;u16 status =0 ;if (!subsys->subsys_discovered) {
@@ -707,8 +707,10 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)/* XXX: figure out what to do about RTD3R/RTD3 */ id->oaes =cpu_to_le32 (NVMET_AEN_CFG_OPTIONAL ); - id->ctratt =cpu_to_le32 (NVME_CTRL_ATTR_HID_128_BIT | -NVME_CTRL_ATTR_TBKAS ); + ctratt =NVME_CTRL_ATTR_HID_128_BIT |NVME_CTRL_ATTR_TBKAS ; +if (nvmet_is_pci_ctrl (ctrl)) + ctratt |=NVME_CTRL_ATTR_RHII ; + id->ctratt =cpu_to_le32 (ctratt); id->oacs =0 ;
@@ -1255,6 +1257,31 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)return 0 ; } +static u16 nvmet_set_feat_host_id (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; + +if (!nvmet_is_pci_ctrl (ctrl)) +return NVME_SC_CMD_SEQ_ERROR |NVME_STATUS_DNR ; + +/* +* The NVMe base specifications v2.1 recommends supporting 128-bits host +* IDs (section 5.1.25.1.28.1). However, that same section also says +* that "The controller may support a 64-bit Host Identifier and/or an +* extended 128-bit Host Identifier". So simplify this support and do +* not support 64-bits host IDs to avoid needing to check that all +* controllers associated with the same subsystem all use the same host +* ID size. +*/ +if (!(req->cmd->common.cdw11 &cpu_to_le32 (1 <<0 ))) { + req->error_loc = offsetof(struct nvme_common_command , cdw11); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return nvmet_copy_from_sgl (req,0 , &req->sq->ctrl->hostid, +sizeof (req->sq->ctrl->hostid)); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1285,7 +1312,7 @@ void nvmet_execute_set_features(struct nvmet_req *req) status =nvmet_set_feat_async_event (req,NVMET_AEN_CFG_ALL );break ;case NVME_FEAT_HOST_ID : - status =NVME_SC_CMD_SEQ_ERROR |NVME_STATUS_DNR ; + status =nvmet_set_feat_host_id (req);break ;case NVME_FEAT_WRITE_PROTECT : status =nvmet_set_feat_write_protect (req);
include/linux/nvme.h +1 -0
@@ -276,6 +276,7 @@ enum nvme_ctrl_attr {NVME_CTRL_ATTR_HID_128_BIT = (1 <<0 ),NVME_CTRL_ATTR_TBKAS = (1 <<6 ),NVME_CTRL_ATTR_ELBAS = (1 <<15 ), + NVME_CTRL_ATTR_RHII = (1 << 18), }; struct nvme_id_ctrl {
Signature · 1 line
2.47.1
Re: [PATCH v4 13/18] nvmet: Implement host identifier set feature support
Quoted 11 lines · Damien Le Moal
The NVMe specifications mandate support for the host identifier
set_featuresfor controllers that also supports reservations. Satisfy this requirement by implementing handling of theNVME_FEAT_HOST_IDfeature for thenvme_set_featurescommand. This implementation is for now effective only for PCI target controllers. For other controller types, the set features command is failed with aNVME_SC_CMD_SEQ_ERRORstatus as before.As noted in the code, 128 bits host identifiers are supported since the NVMe bass specifications version 2.1 indicate in section 5.1.25.1.28.1 that "The controller may support a 64-bit Host Identifier...".
s/bass/base/
PATCH v4 14/18 nvmet: Implement interrupt coalescing feature support
In reply to Damien Le MoalThe NVMe base specifications v2.1 mandate Supporting the interrupt coalescing feature (NVME_FEAT_IRQ_COALESCE) for PCI controllers. Introduce the data structure struct nvmet_feat_irq_coalesce to define the time and threshold (thr) fields of this feature and implement the functions nvmet_get_feat_irq_coalesce() and nvmet_set_feat_irq_coalesce() to get and set this feature. These functions respectively use the controller get_feature() and set_feature() operations to fill and handle the fields of struct nvmet_feat_irq_coalesce.
While the Linux kernel nvme driver does not use this feature and thus will not complain if it is not implemented, other major OSes fail initializing the NVMe device if this feature support is missing.
Support for this feature is prohibited for fabrics controllers. If a get feature or set feature command for this feature is received for a fabrics controller, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 53 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 10 +++++++ 2 files changed, 61 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +51 -2
@@ -1282,6 +1282,27 @@ static u16 nvmet_set_feat_host_id(struct nvmet_req *req) sizeof(req->sq->ctrl->hostid)); } +static u16 nvmet_set_feat_irq_coalesce (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_irq_coalesce irqc = { + .time = (cdw11 >>8 ) &0xff , + .thr = cdw11 &0xff , + }; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1305,6 +1326,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)nvmet_set_result (req, (subsys->max_qid -1 ) | ((subsys->max_qid -1 ) <<16 ));break ; +case NVME_FEAT_IRQ_COALESCE : + status =nvmet_set_feat_irq_coalesce (req); +break ;case NVME_FEAT_KATO : status =nvmet_set_feat_kato (req);break ;
@@ -1349,6 +1373,30 @@ static u16 nvmet_get_feat_write_protect(struct nvmet_req *req)return 0 ; } +static u16 nvmet_get_feat_irq_coalesce (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvmet_feat_irq_coalesce irqc = { }; +u16 status; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, ((u32 )irqc.time <<8 ) | (u32 )irqc.thr); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1383,13 +1431,14 @@ void nvmet_execute_get_features(struct nvmet_req *req)break ;case NVME_FEAT_ERR_RECOVERY :break ; -case NVME_FEAT_IRQ_COALESCE : -break ;case NVME_FEAT_IRQ_CONFIG :break ;case NVME_FEAT_WRITE_ATOMIC :break ;#endif +case NVME_FEAT_IRQ_COALESCE : + status =nvmet_get_feat_irq_coalesce (req); +break ;case NVME_FEAT_ASYNC_EVENT :nvmet_get_feat_async_event (req);break ;
drivers/nvme/target/nvmet.h +10 -0
@@ -906,4 +906,14 @@ static inline void nvmet_pr_put_ns_pc_ref(struct nvmet_pr_per_ctrl_ref *pc_ref) {percpu_ref_put (&pc_ref->ref); } + +/* +* Data for the get_feature() and set_feature() operations of PCI target +* controllers. +*/ +struct nvmet_feat_irq_coalesce { +u8 thr; +u8 time; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v4 15/18 nvmet: Implement interrupt config feature support
In reply to Damien Le MoalThe NVMe base specifications v2.1 mandate supporting the interrupt config feature (NVME_FEAT_IRQ_CONFIG) for PCI controllers. Introduce the data structure struct nvmet_feat_irq_config to define the coalescing disabled (cd) and interrupt vector (iv) fields of this feature and implement the functions nvmet_get_feat_irq_config() and nvmet_set_feat_irq_config() functions to get and set these fields. These functions respectively use the controller get_feature() and set_feature() operations to fill and handle the fields of struct nvmet_feat_irq_config.
Support for this feature is prohibited for fabrics controllers. If a get feature command or a set feature command for this feature is received for a fabrics controller, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 54 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 5 +++ 2 files changed, 57 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +52 -2
@@ -1303,6 +1303,27 @@ static u16 nvmet_set_feat_irq_coalesce(struct nvmet_req *req)return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); } +static u16 nvmet_set_feat_irq_config (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_irq_config irqcfg = { + .iv = cdw11 &0xffff , + .cd = (cdw11 >>16 ) &0x1 , + }; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1329,6 +1350,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)case NVME_FEAT_IRQ_COALESCE : status =nvmet_set_feat_irq_coalesce (req);break ; +case NVME_FEAT_IRQ_CONFIG : + status =nvmet_set_feat_irq_config (req); +break ;case NVME_FEAT_KATO : status =nvmet_set_feat_kato (req);break ;
@@ -1397,6 +1421,31 @@ static u16 nvmet_get_feat_irq_coalesce(struct nvmet_req *req)return NVME_SC_SUCCESS ; } +static u16 nvmet_get_feat_irq_config (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 iv =le32_to_cpu (req->cmd->common.cdw11) &0xffff ; +struct nvmet_feat_irq_config irqcfg = { .iv = iv }; +u16 status; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, ((u32 )irqcfg.cd <<16 ) | iv); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1431,14 +1480,15 @@ void nvmet_execute_get_features(struct nvmet_req *req)break ;case NVME_FEAT_ERR_RECOVERY :break ; -case NVME_FEAT_IRQ_CONFIG : -break ;case NVME_FEAT_WRITE_ATOMIC :break ;#endif case NVME_FEAT_IRQ_COALESCE : status =nvmet_get_feat_irq_coalesce (req);break ; +case NVME_FEAT_IRQ_CONFIG : + status =nvmet_get_feat_irq_config (req); +break ;case NVME_FEAT_ASYNC_EVENT :nvmet_get_feat_async_event (req);break ;
drivers/nvme/target/nvmet.h +5 -0
@@ -916,4 +916,9 @@ struct nvmet_feat_irq_coalesce { u8 time; }; +struct nvmet_feat_irq_config { +u16 iv; +bool cd; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v4 16/18 nvmet: Implement arbitration feature support
In reply to Damien Le MoalNVMe base specification v2.1 mandates support for the arbitration feature (NVME_FEAT_ARBITRATION). Introduce the data structure struct nvmet_feat_arbitration to define the high, medium and low priority weight fields and the arbitration burst field of this feature and implement the functions nvmet_get_feat_arbitration() and nvmet_set_feat_arbitration() functions to get and set these fields.
Since there is no generic way to implement support for the arbitration feature, these functions respectively use the controller get_feature() and set_feature() operations to process the feature with the help of the controller driver. If the controller driver does not implement these operations and a get feature command or a set feature command for this feature is received, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 51 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 7 +++++ 2 files changed, 56 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +49 -2
@@ -1324,6 +1324,25 @@ static u16 nvmet_set_feat_irq_config(struct nvmet_req *req)return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); } +static u16 nvmet_set_feat_arbitration (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_arbitration arb = { + .hpw = (cdw11 >>24 ) &0xff , + .mpw = (cdw11 >>16 ) &0xff , + .lpw = (cdw11 >>8 ) &0xff , + .ab = cdw11 &0x3 , + }; + +if (!ctrl->ops->set_feature) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_ARBITRATION , &arb); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1337,6 +1356,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)return ;switch (cdw10 &0xff ) { +case NVME_FEAT_ARBITRATION : + status =nvmet_set_feat_arbitration (req); +break ;case NVME_FEAT_NUM_QUEUES : ncqr = (cdw11 >>16 ) &0xffff ; nsqr = cdw11 &0xffff ;
@@ -1446,6 +1468,30 @@ static u16 nvmet_get_feat_irq_config(struct nvmet_req *req)return NVME_SC_SUCCESS ; } +static u16 nvmet_get_feat_arbitration (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvmet_feat_arbitration arb = { }; +u16 status; + +if (!ctrl->ops->get_feature) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_ARBITRATION , &arb); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, + ((u32 )arb.hpw <<24 ) | + ((u32 )arb.mpw <<16 ) | + ((u32 )arb.lpw <<8 ) | + (arb.ab &0x3 )); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1472,8 +1518,6 @@ void nvmet_execute_get_features(struct nvmet_req *req)* need to come up with some fake values for these. */ #if 0 -case NVME_FEAT_ARBITRATION : -break ;case NVME_FEAT_POWER_MGMT :break ;case NVME_FEAT_TEMP_THRESH :
@@ -1483,6 +1527,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)case NVME_FEAT_WRITE_ATOMIC :break ;#endif +case NVME_FEAT_ARBITRATION : + status =nvmet_get_feat_arbitration (req); +break ;case NVME_FEAT_IRQ_COALESCE : status =nvmet_get_feat_irq_coalesce (req);break ;
drivers/nvme/target/nvmet.h +7 -0
@@ -921,4 +921,11 @@ struct nvmet_feat_irq_config { bool cd; }; +struct nvmet_feat_arbitration { +u8 hpw; +u8 mpw; +u8 lpw; +u8 ab; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v4 17/18 nvmet: New NVMe PCI endpoint target driver
In reply to Damien Le MoalImplement a PCI target driver using the PCI endpoint framework. This requires hardware with a PCI controller capable of executing in endpoint mode.
The PCI endpoint framework is used to set up a PCI endpoint device and its BAR compatible with a NVMe PCI controller. The framework is also used to map local memory to the PCI address space to execute MMIO accesses for retrieving NVMe commands from submission queues and posting completion entries to completion queues. If supported, DMA is used for command data transfers, based on the PCI address segments indicated by the command using either PRPs or SGLs.
The NVMe target driver relies on the NVMe target core code to execute all commands isssued by the host. The PCI target driver is mainly responsible for the following:
- Initialization and teardown of the endpoint device and its backend PCI target controller. The PCI target controller is created using a subsystem and a port defined through configfs. The port used must be initialized with the "pci" transport type. The target controller is allocated and initialized when the PCI endpoint is started by binding it to the endpoint PCI device (nvmet_pciep_epf_epc_init() function).
- Manage the endpoint controller state according to the PCI link state and the actions of the host (e.g. checking the CC.EN register) and propagate these actions to the PCI target controller. Polling of the controller enable/disable is done using a delayed work scheduled every 5ms (nvmet_pciep_poll_cc() function). This work is started whenever the PCI link comes up (nvmet_pciep_epf_link_up() notifier function) and stopped when the PCI link comes down (nvmet_pciep_epf_link_down() notifier function). nvmet_pciep_poll_cc() enables and disables the PCI controller using the functions nvmet_pciep_enable_ctrl() and nvmet_pciep_disable_ctrl(). The controller admin queue is created using nvmet_pciep_create_cq(), which calls nvmet_cq_create(), and nvmet_pciep_create_sq() which uses nvmet_sq_create(). nvmet_pciep_disable_ctrl() always resets the PCI controller to its initial state so that nvmet_pciep_enable_ctrl() can be called again. This ensure correct operation if, for instance, the host reboots causing the PCI link to be temporarily down.
- Manage the controller admin and I/O submission queues using local memory. Commands are obtained from submission queues using a work item that constantly polls the doorbells of all submissions queues (nvmet_pciep_poll_sqs() function). This work is started whenever the controller is enabled (nvmet_pciep_enable_ctrl() function) and stopped when the controller is disabled (nvmet_pciep_disable_ctrl() function). When new commands are submitted by the host, DMA transfers are used to retrieve the commands.
- Initiate the execution of all admin and I/O commands using the target core code, by calling a requests execute() function. All commands are individually handled using a per-command work item (nvmet_pciep_iod_work() function). A command overall execution
includes: initializing a struct nvmet_req request for the command, using nvmet_req_transfer_len() to get a command data transfer length, parse the command PRPs or SGLs to get the PCI address segments of the command data buffer, retrieve data from the host (if the command is a write command), call req->execute() to execute the command and transfer data to the host (for read commands).
- Handle the completions of commands as notified by the ->queue_response() operation of the PCI target controller (nvmet_pciep_queue_response() function). Completed commands are added to a list of completed command for their CQ. Each CQ list of completed command is processed using a work item (nvmet_pciep_cq_work() function) which posts entries for the completed commands in the CQ memory and raise an IRQ to the host to signal the completion. IRQ coalescing is supported as mandated by the NVMe base specification for PCI controllers. Of note is that completion entries are transmitted to the host using MMIO, after mapping the completion queue memory to the host PCI address space. Unlike for retrieving commands from SQs, DMA is not used as it degrades performance due to the transfer serialization needed (which delays completion entries transmission).
The configuration of a NVMe PCI endpoint controller is done using configfgs. First the NVMe PCI target controller configuration must be done to set up a subsystem and a port with the "pci" addr_trtype attribute. The subsystem can be setup using a file or block device backed namespace or using a passthrough NVMe device. After this, the PCI endpoint can be configured and bound to the PCI endpoint controller to start the NVMe endpoint controller.
In order to not overcomplicate this initial implementation of an endpoint PCI target controller driver, protection information is not for now supported. If the PCI controller port and namespace are configured with protection information support, an error will be returned when the controller is created and initialized when the endpoint function is started. Protection information support will be added in a follow-up patch series.
Using a Rock5B board (Rockchip RK3588 SoC, PCI Gen3x4 endpoint controller) with a target PCI controller setup with 4 I/O queues and a null_blk block device as a namespace, the maximum performance using fio was measured at 131 KIOPS for random 4K reads and up to 2.8 GB/S throughput. Some data points are:
Rnd read, 4KB, QD=1, 1 job : IOPS=16.9k, BW=66.2MiB/s (69.4MB/s) Rnd read, 4KB, QD=32, 1 job : IOPS=78.5k, BW=307MiB/s (322MB/s) Rnd read, 4KB, QD=32, 4 jobs: IOPS=131k, BW=511MiB/s (536MB/s)
Seq read, 512KB, QD=32, 1 job : IOPS=5381, BW=2691MiB/s (2821MB/s)
The NVMe PCI endpoint target driver is not intended for production use. It is a tool for learning NVMe, exploring existing features and testing implementations of new NVMe features.
--- drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/pci-ep.c | 2626 ++++++++++++++++++++++++++++++++++ 3 files changed, 2638 insertions(+) create mode 100644 drivers/nvme/target/pci-ep.c
drivers/nvme/target/Kconfig +10 -0
@@ -115,3 +115,13 @@ config NVME_TARGET_AUTH target side. If unsure, say N. + +config NVME_TARGET_PCI_EP + tristate "NVMe PCI Endpoint Target support" + depends on PCI_ENDPOINT && NVME_TARGET + help + This enables the NVMe PCI endpoint target support which allows to + create an NVMe PCI controller using a PCI endpoint capable PCI + controller. + + If unsure, say N.
drivers/nvme/target/Makefile +2 -0
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o obj-$(CONFIG_NVME_TARGET_FC) += nvmet-fc.o obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o +obj-$(CONFIG_NVME_TARGET_PCI_EP) += nvmet-pciep.o nvmet-y += core.o configfs.o admin-cmd.o fabrics-cmd.o \ discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
@@ -20,4 +21,5 @@ nvmet-rdma-y += rdma.o nvmet-fc-y += fc.o nvme-fcloop-y += fcloop.o nvmet-tcp-y += tcp.o +nvmet-pciep-y += pci-ep.o nvmet-$(CONFIG_TRACING) += trace.o
drivers/nvme/target/pci-ep.c new file +2626 -0
2,626 lines not on this page · raw message
Signature · 1 line
2.47.1
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 2 lines · Damien Le Moal
This ensure correct operation if, for instance, the host reboots causing the PCI link to be temporarily down.
s/ensure/ensures/
Quoted 7 lines
The configuration of a NVMe PCI endpoint controller is done using configfgs. First the NVMe PCI target controller configuration must be done to set up a subsystem and a port with the "pci"
addr_trtypeattribute. The subsystem can be setup using a file or block device backed namespace or using a passthrough NVMe device. After this, the PCI endpoint can be configured and bound to the PCI endpoint controller to start the NVMe endpoint controller.
s/addr_trtype/addr_type/ ?
s/configfgs/configfs/
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 15 lines · Bjorn Helgaas
Quoted 2 lines · Damien Le Moal
This ensure correct operation if, for instance, the host reboots causing the PCI link to be temporarily down.
s/ensure/ensures/
Quoted 7 lines
The configuration of a NVMe PCI endpoint controller is done using configfgs. First the NVMe PCI target controller configuration must be done to set up a subsystem and a port with the "pci"
addr_trtypeattribute. The subsystem can be setup using a file or block device backed namespace or using a passthrough NVMe device. After this, the PCI endpoint can be configured and bound to the PCI endpoint controller to start the NVMe endpoint controller.
Quoted 1 line
s/addr_trtype/addr_type/ ?
Nope, addr_trtype is correct. "address transport type" is the meaning for an nvme target port.
Quoted 1 line
s/configfgs/configfs/
Good catch. Thanks.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Damien Le MoalHello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
If you look at the nvmet_pciep_epf_dma_transfer() function below, it takes a mutex around the dmaengine_slave_config(), dmaengine_prep_slave_single(), dmaengine_submit(), dma_sync_wait(), and dmaengine_terminate_sync() calls.
I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the drivers/dma/dw-edma/dw-edma-core.c driver, I can see that dmaengine_prep_slave_single() (which will call device_prep_slave_sg(.., .., 1, .., .., ..)) allocates a new dma_async_tx_descriptor for each function call.
I can see that device_prep_slave_sg() (dw_edma_device_prep_slave_sg()) will call dw_edma_device_transfer() which will call vchan_tx_prep(), which adds the descriptor to the tail of a list.
I can also see that dw_edma_done_interrupt() will automatically start the transfer of the next descriptor (using vchan_next_desc()).
So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because dmaengine_prep_slave_single() really requires dmaengine_slave_config() for each transfer. (Since we are supplying a src_addr in the sconf that we are supplying to dmaengine_slave_config().)
(i.e. we can't call dmaengine_slave_config() while a DMA transfer is active.)
So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
If the dmaengine_prep_slave_single() instead took a sconfig as a parameter, then it seems like it would be possible to use this API asynchronously.
There does seem to be other drivers holding a mutex around dmaengine_slave_config() + dmaengine_prep_slave_single()... but it feels like this should be avoidable (at least for dw-edma), if dmaengine_prep_slave_single() simply took an sconf.
What am I missing? :)
Kind regards,
Niklas
Quoted 75 lines · Damien Le Moal
drivers/nvme/target/pci-ep.c new file +75
-0+static int nvmet_pciep_epf_dma_transfer (struct nvmet_pciep_epf *nvme_epf, +struct nvmet_pciep_segment *seg,enum dma_data_direction dir) +{ +struct pci_epf *epf = nvme_epf->epf; +struct dma_async_tx_descriptor *desc; +struct dma_slave_config sconf = {}; +struct device *dev = &epf->dev; +struct device *dma_dev; +struct dma_chan *chan; +dma_cookie_t cookie; +dma_addr_t dma_addr; +struct mutex *lock; +int ret; + +switch (dir) { +case DMA_FROM_DEVICE : + lock = &nvme_epf->dma_rx_lock; + chan = nvme_epf->dma_rx_chan; + sconf.direction =DMA_DEV_TO_MEM ; + sconf.src_addr = seg->pci_addr; +break ; +case DMA_TO_DEVICE : + lock = &nvme_epf->dma_tx_lock; + chan = nvme_epf->dma_tx_chan; + sconf.direction =DMA_MEM_TO_DEV ; + sconf.dst_addr = seg->pci_addr; +break ; +default : +return -EINVAL ; + } + +mutex_lock (lock); + + dma_dev =dmaengine_get_dma_device (chan); + dma_addr =dma_map_single (dma_dev, seg->buf, seg->length, dir); + ret =dma_mapping_error (dma_dev, dma_addr); +if (ret) + goto unlock; + + ret =dmaengine_slave_config (chan, &sconf); +if (ret) { +dev_err (dev,"Failed to configure DMA channel\n" ); + goto unmap; + } + + desc =dmaengine_prep_slave_single (chan, dma_addr, seg->length, + sconf.direction,DMA_CTRL_ACK ); +if (!desc) { +dev_err (dev,"Failed to prepare DMA\n" ); + ret = -EIO ; + goto unmap; + } + + cookie =dmaengine_submit (desc); + ret =dma_submit_error (cookie); +if (ret) { +dev_err (dev,"DMA submit failed %d\n" , ret); + goto unmap; + } + +if (dma_sync_wait (chan, cookie) !=DMA_COMPLETE ) { +dev_err (dev,"DMA transfer failed\n" ); + ret = -EIO ; + } + +dmaengine_terminate_sync (chan); + +unmap: +dma_unmap_single (dma_dev, dma_addr, seg->length, dir); + +unlock: +mutex_unlock (lock); + +return ret; +}
Vinod Koul
maintainerRe: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)
Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the dma_slave_config would pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...
I suspect since you are passing an address which keep changing in the dma_slave_config, you need to guard that and prep_slave_single() call, as while preparing the descriptor driver would lookup what was setup for the configuration.
I suggest then use the prep_memcpy() API instead and pass on source and destination, no need to lock the calls...
Quoted 91 lines
If the
dmaengine_prep_slave_single()instead took a sconfig as a parameter, then it seems like it would be possible to use this API asynchronously.There does seem to be other drivers holding a mutex around
dmaengine_slave_config()+dmaengine_prep_slave_single()... but it feels like this should be avoidable (at least for dw-edma), ifdmaengine_prep_slave_single()simply took an sconf.What am I missing? :)
Kind regards,
NiklasQuoted 75 lines · Damien Le Moal
drivers/nvme/target/pci-ep.c new file +75
-0+static int nvmet_pciep_epf_dma_transfer (struct nvmet_pciep_epf *nvme_epf, +struct nvmet_pciep_segment *seg,enum dma_data_direction dir) +{ +struct pci_epf *epf = nvme_epf->epf; +struct dma_async_tx_descriptor *desc; +struct dma_slave_config sconf = {}; +struct device *dev = &epf->dev; +struct device *dma_dev; +struct dma_chan *chan; +dma_cookie_t cookie; +dma_addr_t dma_addr; +struct mutex *lock; +int ret; + +switch (dir) { +case DMA_FROM_DEVICE : + lock = &nvme_epf->dma_rx_lock; + chan = nvme_epf->dma_rx_chan; + sconf.direction =DMA_DEV_TO_MEM ; + sconf.src_addr = seg->pci_addr; +break ; +case DMA_TO_DEVICE : + lock = &nvme_epf->dma_tx_lock; + chan = nvme_epf->dma_tx_chan; + sconf.direction =DMA_MEM_TO_DEV ; + sconf.dst_addr = seg->pci_addr; +break ; +default : +return -EINVAL ; + } + +mutex_lock (lock); + + dma_dev =dmaengine_get_dma_device (chan); + dma_addr =dma_map_single (dma_dev, seg->buf, seg->length, dir); + ret =dma_mapping_error (dma_dev, dma_addr); +if (ret) + goto unlock; + + ret =dmaengine_slave_config (chan, &sconf); +if (ret) { +dev_err (dev,"Failed to configure DMA channel\n" ); + goto unmap; + } + + desc =dmaengine_prep_slave_single (chan, dma_addr, seg->length, + sconf.direction,DMA_CTRL_ACK ); +if (!desc) { +dev_err (dev,"Failed to prepare DMA\n" ); + ret = -EIO ; + goto unmap; + } + + cookie =dmaengine_submit (desc); + ret =dma_submit_error (cookie); +if (ret) { +dev_err (dev,"DMA submit failed %d\n" , ret); + goto unmap; + } + +if (dma_sync_wait (chan, cookie) !=DMA_COMPLETE ) { +dev_err (dev,"DMA transfer failed\n" ); + ret = -EIO ; + } + +dmaengine_terminate_sync (chan); + +unmap: +dma_unmap_single (dma_dev, dma_addr, seg->length, dir); + +unlock: +mutex_unlock (lock); + +return ret; +}
Signature · 1 line
~Vinod
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 51 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...
Quoted 7 lines
I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...
Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the device_prep_dma_memcpy operation, no ? In our case, the DWC EDMA driver does not seem to implement this.
Signature · 2 lines
Damien Le Moal Western Digital Research
Vinod Koul
maintainerRe: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 57 lines · Damien Le Moal
Quoted 56 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.
Quoted 8 lines
Quoted 2 lines
I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the
device_prep_dma_memcpyoperation, no ? In our case, the DWC EDMA driver does not seem to implement this.
It should be added in that case.
Before that, the bigger question is, should nvme be slave transfer or memcpy.. Was driver support the reason why the slave transfer was used here...?
As i said, slave is for peripherals which have a static fifo to send/receive data from, nvme sounds like a memory transfer to me, is that a right assumption?
Signature · 1 line
~Vinod
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 67 lines · Vinod Koul
Quoted 57 lines · Damien Le Moal
Quoted 56 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.Quoted 8 lines
Quoted 2 lines
I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the
device_prep_dma_memcpyoperation, no ? In our case, the DWC EDMA driver does not seem to implement this.
Quoted 9 lines
It should be added in that case.
Before that, the bigger question is, should nvme be slave transfer or memcpy.. Was driver support the reason why the slave transfer was used here...?
As i said, slave is for peripherals which have a static fifo to send/receive data from, nvme sounds like a memory transfer to me, is that a right assumption?
My understanding is that DMA_MEMCPY is for local DDR transfer i.e., src and dst are local addresses. And DMA_SLAVE is for transfer between remote and local addresses. I haven't looked into the NVMe EPF driver yet, but it should do the transfer between remote and local addresses. This is similar to MHI EPF driver as well.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 78 lines · Manivannan Sadhasivam
Quoted 67 lines · Vinod Koul
Quoted 57 lines · Damien Le Moal
Quoted 56 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.Quoted 8 lines
Quoted 2 lines
I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the
device_prep_dma_memcpyoperation, no ? In our case, the DWC EDMA driver does not seem to implement this.Quoted 9 lines
It should be added in that case.
Before that, the bigger question is, should nvme be slave transfer or memcpy.. Was driver support the reason why the slave transfer was used here...?
As i said, slave is for peripherals which have a static fifo to send/receive data from, nvme sounds like a memory transfer to me, is that a right assumption?
Quoted 6 lines
My understanding is that
DMA_MEMCPYis for local DDR transfer i.e., src and dst are local addresses. AndDMA_SLAVEis for transfer between remote and local addresses. I haven't looked into the NVMe EPF driver yet, but it should do the transfer between remote and local addresses. This is similar to MHI EPF driver as well.
I had an offline discussion with Vinod and he clarified that the DMA_SLAVE implementation is supposed to be used by clients with fixed FIFO. That's why 'struct dma_slave_config' has options to configure maxburst, addr_width, port_window_size etc... So these are not applicable for our usecase where we would be just carrying out transfer between remote and local DDR.
So we should be implementing prep_memcpy() in dw-edma driver and use DMA_MEMCPY in client drivers.
@Niklas: Since you asked this question, are you volunteering to implement prep_memcpy() in dw-edma driver? ;)
If not, I will work with Qcom to make it happen.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 92 lines · Manivannan Sadhasivam
Quoted 78 lines · Manivannan Sadhasivam
Quoted 67 lines · Vinod Koul
Quoted 57 lines · Damien Le Moal
Quoted 56 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.Quoted 8 lines
Quoted 2 lines
I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the
device_prep_dma_memcpyoperation, no ? In our case, the DWC EDMA driver does not seem to implement this.Quoted 9 lines
It should be added in that case.
Before that, the bigger question is, should nvme be slave transfer or memcpy.. Was driver support the reason why the slave transfer was used here...?
As i said, slave is for peripherals which have a static fifo to send/receive data from, nvme sounds like a memory transfer to me, is that a right assumption?
Quoted 6 lines
My understanding is that
DMA_MEMCPYis for local DDR transfer i.e., src and dst are local addresses. AndDMA_SLAVEis for transfer between remote and local addresses. I haven't looked into the NVMe EPF driver yet, but it should do the transfer between remote and local addresses. This is similar to MHI EPF driver as well.I had an offline discussion with Vinod and he clarified that the
DMA_SLAVEimplementation is supposed to be used by clients with fixed FIFO. That's why 'structdma_slave_config' has options to configure maxburst,addr_width,port_window_sizeetc... So these are not applicable for our usecase where we would be just carrying out transfer between remote and local DDR.
Quoted 5 lines
So we should be implementing
prep_memcpy()in dw-edma driver and useDMA_MEMCPYin client drivers.@Niklas: Since you asked this question, are you volunteering to implement
prep_memcpy()in dw-edma driver? ;)
I did implement something that seems to be working.
It would be nice if you could help testing it using your MHI EPF driver. (You need to convert it to use _prep_memcpy() in order to be able to test.)
Kind regards,
Niklas
PATCH 1/3 dmaengine: dw-edma: Add support for DMA_MEMCPY
--- drivers/dma/dw-edma/dw-edma-core.c | 52 ++++++++++++++++++++++++++++-- drivers/dma/dw-edma/dw-edma-core.h | 10 +++++- 2 files changed, 59 insertions(+), 3 deletions(-)
drivers/dma/dw-edma/dw-edma-core.c +50 -2
@@ -422,6 +422,9 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer)return NULL ;if (!xfer->xfer.il->src_inc || !xfer->xfer.il->dst_inc)return NULL ; + }else if (xfer->type ==EDMA_XFER_MEMCPY ) { +if (!xfer->xfer.memcpy.len) +return NULL ; }else {return NULL ; }
@@ -437,6 +440,9 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer)if (xfer->type ==EDMA_XFER_INTERLEAVED ) { src_addr = xfer->xfer.il->src_start; dst_addr = xfer->xfer.il->dst_start; + }else if (xfer->type ==EDMA_XFER_MEMCPY ) { + src_addr = xfer->xfer.memcpy.src; + dst_addr = xfer->xfer.memcpy.dst; }else { src_addr = chan->config.src_addr; dst_addr = chan->config.dst_addr;
@@ -455,6 +461,8 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer) }else if (xfer->type ==EDMA_XFER_INTERLEAVED ) { cnt = xfer->xfer.il->numf * xfer->xfer.il->frame_size; fsz = xfer->xfer.il->frame_size; + }else if (xfer->type ==EDMA_XFER_MEMCPY ) { + cnt =1 ; }for (i =0 ; i < cnt; i++) {
@@ -477,6 +485,8 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer) burst->sz =sg_dma_len (sg); else if (xfer->type == EDMA_XFER_INTERLEAVED) burst->sz = xfer->xfer.il->sgl[i % fsz].size; + else if (xfer->type == EDMA_XFER_MEMCPY) + burst->sz = xfer->xfer.memcpy.len; chunk->ll_region.sz += burst->sz; desc->alloc_sz += burst->sz;
@@ -495,7 +505,8 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer)* and destination addresses are increased * by the same portion (data length) */ - }else if (xfer->type ==EDMA_XFER_INTERLEAVED ) { + }else if (xfer->type ==EDMA_XFER_INTERLEAVED || + xfer->type ==EDMA_XFER_MEMCPY ) { burst->dar = dst_addr; } }else {
@@ -512,7 +523,8 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer)* and destination addresses are increased * by the same portion (data length) */ - }else if (xfer->type ==EDMA_XFER_INTERLEAVED ) { + }else if (xfer->type ==EDMA_XFER_INTERLEAVED || + xfer->type ==EDMA_XFER_MEMCPY ) { burst->sar = src_addr; } }
@@ -595,6 +607,40 @@ dw_edma_device_prep_interleaved_dma(struct dma_chan *dchan,return dw_edma_device_transfer (&xfer); } +static struct dma_async_tx_descriptor * +dw_edma_device_prep_dma_memcpy (struct dma_chan *dchan,dma_addr_t dst, +dma_addr_t src,size_t len,unsigned long flags) +{ +struct dw_edma_chan *chan =dchan2dw_edma_chan (dchan); +enum dma_transfer_direction direction; +struct dw_edma_transfer xfer; + +if (chan->dw->chip->flags &DW_EDMA_CHIP_LOCAL ) { +if (chan->dir ==EDMA_DIR_READ ) + direction =DMA_DEV_TO_MEM ; +else + direction =DMA_MEM_TO_DEV ; + }else { +if (chan->dir ==EDMA_DIR_WRITE ) + direction =DMA_DEV_TO_MEM ; +else + direction =DMA_MEM_TO_DEV ; + } + + xfer.dchan = dchan; + xfer.direction = direction; + xfer.xfer.memcpy.dst = dst; + xfer.xfer.memcpy.src = src; + xfer.xfer.memcpy.len = len; + xfer.flags = flags; + xfer.type =EDMA_XFER_MEMCPY ; + +/* DMA_MEMCPY does not need an initial dmaengine_slave_config() call */ + chan->configured = true; + +return dw_edma_device_transfer (&xfer); +} +static void dw_edma_done_interrupt (struct dw_edma_chan *chan) {struct dw_edma_desc *desc;
@@ -787,6 +833,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc)dma_cap_set (DMA_CYCLIC , dma->cap_mask);dma_cap_set (DMA_PRIVATE , dma->cap_mask);dma_cap_set (DMA_INTERLEAVE , dma->cap_mask); +dma_cap_set (DMA_MEMCPY , dma->cap_mask); dma->directions =BIT (DMA_DEV_TO_MEM ) |BIT (DMA_MEM_TO_DEV ); dma->src_addr_widths =BIT (DMA_SLAVE_BUSWIDTH_4_BYTES ); dma->dst_addr_widths =BIT (DMA_SLAVE_BUSWIDTH_4_BYTES );
@@ -806,6 +853,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, u32 wr_alloc, u32 rd_alloc) dma->device_prep_slave_sg = dw_edma_device_prep_slave_sg; dma->device_prep_dma_cyclic = dw_edma_device_prep_dma_cyclic; dma->device_prep_interleaved_dma = dw_edma_device_prep_interleaved_dma; + dma->device_prep_dma_memcpy = dw_edma_device_prep_dma_memcpy;dma_set_max_seg_size (dma->dev,U32_MAX );
drivers/dma/dw-edma/dw-edma-core.h +9 -1
@@ -36,7 +36,8 @@ enum dw_edma_status {enum dw_edma_xfer_type {EDMA_XFER_SCATTER_GATHER =0 ,EDMA_XFER_CYCLIC , -EDMA_XFER_INTERLEAVED +EDMA_XFER_INTERLEAVED , +EDMA_XFER_MEMCPY };struct dw_edma_chan ;
@@ -139,12 +140,19 @@ struct dw_edma_cyclic {size_t cnt; }; +struct dw_edma_memcpy { +dma_addr_t dst; +dma_addr_t src; +size_t len; +}; +struct dw_edma_transfer {struct dma_chan *dchan;union dw_edma_xfer {struct dw_edma_sg sg;struct dw_edma_cyclic cyclic;struct dma_interleaved_template *il; +struct dw_edma_memcpy memcpy; } xfer;enum dma_transfer_direction direction;unsigned long flags;
Signature · 1 line
2.47.1
PATCH 2/3 PCI: endpoint: pci-epf-test: Use private DMA_MEMCPY channel
First try to use a DMA channel with DMA_PRIVATE and DMA_MEMCPY. If there is no such channel, fall back to a DMA_MEMCPY channel (without DMA_PRIVATE).
--- drivers/pci/endpoint/functions/pci-epf-test.c | 24 +++---------------- 1 file changed, 3 insertions(+), 21 deletions(-)
drivers/pci/endpoint/functions/pci-epf-test.c +3 -21
@@ -123,11 +123,9 @@ static int pci_epf_test_data_transfer(struct pci_epf_test *epf_test, {struct dma_chan *chan = (dir ==DMA_MEM_TO_DEV ) ? epf_test->dma_chan_tx : epf_test->dma_chan_rx; -dma_addr_t dma_local = (dir ==DMA_MEM_TO_DEV ) ? dma_src : dma_dst;enum dma_ctrl_flags flags =DMA_CTRL_ACK |DMA_PREP_INTERRUPT ;struct pci_epf *epf = epf_test->epf;struct dma_async_tx_descriptor *tx; -struct dma_slave_config sconf = {};struct device *dev = &epf->dev;int ret;
@@ -136,24 +134,7 @@ static int pci_epf_test_data_transfer(struct pci_epf_test *epf_test,return -EINVAL ; } -if (epf_test->dma_private) { - sconf.direction = dir; -if (dir ==DMA_MEM_TO_DEV ) - sconf.dst_addr = dma_remote; -else - sconf.src_addr = dma_remote; - -if (dmaengine_slave_config (chan, &sconf)) { -dev_err (dev,"DMA slave config fail\n" ); -return -EIO ; - } - tx =dmaengine_prep_slave_single (chan, dma_local, len, dir, - flags); - }else { - tx =dmaengine_prep_dma_memcpy (chan, dma_dst, dma_src, len, - flags); - } - + tx =dmaengine_prep_dma_memcpy (chan, dma_dst, dma_src, len, flags);if (!tx) {dev_err (dev,"Failed to prepare DMA memcpy\n" );return -EIO ;
@@ -225,7 +206,8 @@ static int pci_epf_test_init_dma_chan(struct pci_epf_test *epf_test) filter.dma_mask =BIT (DMA_DEV_TO_MEM );dma_cap_zero (mask); -dma_cap_set (DMA_SLAVE , mask); +dma_cap_set (DMA_PRIVATE , mask); +dma_cap_set (DMA_MEMCPY , mask); dma_chan =dma_request_channel (mask, epf_dma_filter_fn, &filter);if (!dma_chan) {dev_info (dev,"Failed to get private DMA rx channel. Falling back to generic one\n" );
Signature · 1 line
2.47.1
PATCH 3/3 debug prints - DO NOT MERGE
In reply to Niklas Cassel--- drivers/dma/dw-edma/dw-edma-core.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/dma/dw-edma/dw-edma-core.c +7 -0
@@ -197,6 +197,7 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)if (!desc)return 0 ; +trace_printk ("start transfer for desc: %px\n" , desc); child =list_first_entry_or_null (&desc->chunk->list, struct dw_edma_chunk, list);if (!child)
@@ -545,6 +546,9 @@ dw_edma_device_transfer(struct dw_edma_transfer *xfer) } } +if (xfer->type ==EDMA_XFER_MEMCPY ) +trace_printk ("DMA_MEMCPY tx_prep desc: %px\n" , desc); +return vchan_tx_prep (&chan->vc, &desc->vd, xfer->flags); err_alloc:
@@ -653,6 +657,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)switch (chan->request) {case EDMA_REQ_NONE : desc =vd2dw_edma_desc (vd); +trace_printk ("REQ_NONE desc: %px\n" , desc);if (!desc->chunks_alloc) {list_del (&vd->node);vchan_cookie_complete (vd);
@@ -664,6 +669,8 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)break ;case EDMA_REQ_STOP : + desc =vd2dw_edma_desc (vd); +trace_printk ("REQ_STOP desc: %px\n" , desc);list_del (&vd->node);vchan_cookie_complete (vd); chan->request =EDMA_REQ_NONE ;
Signature · 1 line
2.47.1
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Niklas CasselQuoted 96 lines · Niklas Cassel
Quoted 95 lines · Manivannan Sadhasivam
Quoted 78 lines · Manivannan Sadhasivam
Quoted 67 lines · Vinod Koul
Quoted 57 lines · Damien Le Moal
Quoted 56 lines · Vinod Koul
Hi Niklas,
Quoted 4 lines · Niklas Cassel
Hello Vinod,
I am a bit confused about the usage of the dmaengine API, and I hope that you could help make me slightly less confused :)
Sure thing!
Quoted 27 lines
If you look at the
nvmet_pciep_epf_dma_transfer()function below, it takes a mutex around thedmaengine_slave_config(),dmaengine_prep_slave_single(),dmaengine_submit(),dma_sync_wait(), anddmaengine_terminate_sync()calls.I really wish that we would remove this mutex, to get better performance.
If I look at e.g. the
drivers/dma/dw-edma/dw-edma-core.cdriver, I can see thatdmaengine_prep_slave_single()(which will calldevice_prep_slave_sg(.., .., 1, .., .., ..)) allocates a newdma_async_tx_descriptorfor each function call.I can see that
device_prep_slave_sg()(dw_edma_device_prep_slave_sg()) will calldw_edma_device_transfer()which will callvchan_tx_prep(), which adds the descriptor to the tail of a list.I can also see that
dw_edma_done_interrupt()will automatically start the transfer of the next descriptor (usingvchan_next_desc()).So this looks like it is supposed to be asynchronous... however, if we simply remove the mutex, we get IOMMU errors, most likely because the DMA writes to an incorrect address.
It looks like this is because
dmaengine_prep_slave_single()really requiresdmaengine_slave_config()for each transfer. (Since we are supplying asrc_addrin the sconf that we are supplying todmaengine_slave_config().)Quoted 6 lines
(i.e. we can't call
dmaengine_slave_config()while a DMA transfer is active.)So while this API is supposed to be async, to me it looks like it can only be used in a synchronous manner... But that seems like a really weird design.
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.Quoted 8 lines
Quoted 2 lines
I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...Vinod,
Thank you for the information. However, I think we can use this only if the DMA controller driver implements the
device_prep_dma_memcpyoperation, no ? In our case, the DWC EDMA driver does not seem to implement this.Quoted 9 lines
It should be added in that case.
Before that, the bigger question is, should nvme be slave transfer or memcpy.. Was driver support the reason why the slave transfer was used here...?
As i said, slave is for peripherals which have a static fifo to send/receive data from, nvme sounds like a memory transfer to me, is that a right assumption?
Quoted 6 lines
My understanding is that
DMA_MEMCPYis for local DDR transfer i.e., src and dst are local addresses. AndDMA_SLAVEis for transfer between remote and local addresses. I haven't looked into the NVMe EPF driver yet, but it should do the transfer between remote and local addresses. This is similar to MHI EPF driver as well.I had an offline discussion with Vinod and he clarified that the
DMA_SLAVEimplementation is supposed to be used by clients with fixed FIFO. That's why 'structdma_slave_config' has options to configure maxburst,addr_width,port_window_sizeetc... So these are not applicable for our usecase where we would be just carrying out transfer between remote and local DDR.So we should be implementing
prep_memcpy()in dw-edma driver and useDMA_MEMCPYin client drivers.
Quoted 8 lines
Quoted 2 lines
@Niklas: Since you asked this question, are you volunteering to implement
prep_memcpy()in dw-edma driver? ;)I did implement something that seems to be working.
It would be nice if you could help testing it using your MHI EPF driver. (You need to convert it to use
_prep_memcpy()in order to be able to test.)
Sure, will do it tomorrow and let you know. Thanks for sending the patches quickly :)
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Vinod KoulOn Mon, Dec 16, 2024 at 10:05:44PM +0530, Vinod Koul wrote:
[...]
Quoted 10 lines
Quoted 2 lines
Am I missing something obvious here?
Yes, I feel nvme being treated as slave transfer, which it might not be. This API was designed for peripherals like i2c/spi etc where we have a hardware address to read/write to. So the
dma_slave_configwould pass on the transfer details for the peripheral like address, width of fifo, depth etc and these are setup config, so call once for a channel and then prepare the descriptor, submit... and repeat of prepare and submit ...
Quoted 7 lines
I suspect since you are passing an address which keep changing in the
dma_slave_config, you need to guard that andprep_slave_single()call, as while preparing the descriptor driver would lookup what was setup for the configuration.I suggest then use the
prep_memcpy()API instead and pass on source and destination, no need to lock the calls...
Vinod, as always, thank you very much for your suggestion, it is appreciated!
Kind regards,
Niklas
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Damien Le MoalQuoted 121 lines · Damien Le Moal
Implement a PCI target driver using the PCI endpoint framework. This requires hardware with a PCI controller capable of executing in endpoint mode.
The PCI endpoint framework is used to set up a PCI endpoint device and its BAR compatible with a NVMe PCI controller. The framework is also used to map local memory to the PCI address space to execute MMIO accesses for retrieving NVMe commands from submission queues and posting completion entries to completion queues. If supported, DMA is used for command data transfers, based on the PCI address segments indicated by the command using either PRPs or SGLs.
The NVMe target driver relies on the NVMe target core code to execute all commands isssued by the host. The PCI target driver is mainly responsible for the following:
- Initialization and teardown of the endpoint device and its backend PCI target controller. The PCI target controller is created using a subsystem and a port defined through configfs. The port used must be initialized with the "pci" transport type. The target controller is allocated and initialized when the PCI endpoint is started by binding it to the endpoint PCI device (nvmet_pciep_epf_epc_init()function).- Manage the endpoint controller state according to the PCI link state and the actions of the host (e.g. checking the CC.EN register) and propagate these actions to the PCI target controller. Polling of the controller enable/disable is done using a delayed work scheduled every 5ms (
nvmet_pciep_poll_cc()function). This work is started whenever the PCI link comes up (nvmet_pciep_epf_link_up()notifier function) and stopped when the PCI link comes down (nvmet_pciep_epf_link_down()notifier function).nvmet_pciep_poll_cc()enables and disables the PCI controller using the functionsnvmet_pciep_enable_ctrl()andnvmet_pciep_disable_ctrl(). The controller admin queue is created usingnvmet_pciep_create_cq(), which callsnvmet_cq_create(), andnvmet_pciep_create_sq()which usesnvmet_sq_create().nvmet_pciep_disable_ctrl()always resets the PCI controller to its initial state so thatnvmet_pciep_enable_ctrl()can be called again. This ensure correct operation if, for instance, the host reboots causing the PCI link to be temporarily down.- Manage the controller admin and I/O submission queues using local memory. Commands are obtained from submission queues using a work item that constantly polls the doorbells of all submissions queues (
nvmet_pciep_poll_sqs()function). This work is started whenever the controller is enabled (nvmet_pciep_enable_ctrl()function) and stopped when the controller is disabled (nvmet_pciep_disable_ctrl()function). When new commands are submitted by the host, DMA transfers are used to retrieve the commands.- Initiate the execution of all admin and I/O commands using the target core code, by calling a requests
execute()function. All commands are individually handled using a per-command work item (nvmet_pciep_iod_work()function). A command overall execution
includes: initializing a structnvmet_reqrequest for the command, usingnvmet_req_transfer_len()to get a command data transfer length, parse the command PRPs or SGLs to get the PCI address segments of the command data buffer, retrieve data from the host (if the command is a write command), callreq->execute()to execute the command and transfer data to the host (for read commands).- Handle the completions of commands as notified by the ->queue_response() operation of the PCI target controller (
nvmet_pciep_queue_response()function). Completed commands are added to a list of completed command for their CQ. Each CQ list of completed command is processed using a work item (nvmet_pciep_cq_work()function) which posts entries for the completed commands in the CQ memory and raise an IRQ to the host to signal the completion. IRQ coalescing is supported as mandated by the NVMe base specification for PCI controllers. Of note is that completion entries are transmitted to the host using MMIO, after mapping the completion queue memory to the host PCI address space. Unlike for retrieving commands from SQs, DMA is not used as it degrades performance due to the transfer serialization needed (which delays completion entries transmission).The configuration of a NVMe PCI endpoint controller is done using configfgs. First the NVMe PCI target controller configuration must be done to set up a subsystem and a port with the "pci"
addr_trtypeattribute. The subsystem can be setup using a file or block device backed namespace or using a passthrough NVMe device. After this, the PCI endpoint can be configured and bound to the PCI endpoint controller to start the NVMe endpoint controller.In order to not overcomplicate this initial implementation of an endpoint PCI target controller driver, protection information is not for now supported. If the PCI controller port and namespace are configured with protection information support, an error will be returned when the controller is created and initialized when the endpoint function is started. Protection information support will be added in a follow-up patch series.
Using a Rock5B board (Rockchip RK3588 SoC, PCI Gen3x4 endpoint controller) with a target PCI controller setup with 4 I/O queues and a
null_blkblock device as a namespace, the maximum performance using fio was measured at 131 KIOPS for random 4K reads and up to 2.8 GB/S throughput. Some data points are:Rnd read, 4KB, QD=1, 1 job : IOPS=16.9k, BW=66.2MiB/s (69.4MB/s) Rnd read, 4KB, QD=32, 1 job : IOPS=78.5k, BW=307MiB/s (322MB/s) Rnd read, 4KB, QD=32, 4 jobs: IOPS=131k, BW=511MiB/s (536MB/s)Seq read, 512KB, QD=32, 1 job : IOPS=5381, BW=2691MiB/s (2821MB/s)
The NVMe PCI endpoint target driver is not intended for production use. It is a tool for learning NVMe, exploring existing features and testing implementations of new NVMe features.
Co-developed-by Rick Wertenbroek <[email protected]>Signed-off-by Damien Le Moal <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>--- drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/pci-ep.c | 2626 ++++++++++++++++++++++++++++++++++ 3 files changed, 2638 insertions(+) create mode 100644 drivers/nvme/target/pci-ep.cdrivers/nvme/target/Kconfig +0
-0
drivers/nvme/target/Kconfig config NVME_TARGET_AUTH · line 119
Quoted 5 lines
drivers/nvme/target/Kconfig +2
-0@@ -115,3 +115,13 @@ config NVME_TARGET_AUTH target side. If unsure, say N. + +config NVME_TARGET_PCI_EP
Could you please use NVME_TARGET_PCI_EPF for consistency with other EPF drivers?
drivers/nvme/target/Kconfig config NVME_TARGET_AUTH · line 125
Quoted 6 lines
drivers/nvme/target/Kconfig +6
-0+ tristate "NVMe PCI Endpoint Target support" + depends on PCI_ENDPOINT && NVME_TARGET + help + This enables the NVMe PCI endpoint target support which allows to + create an NVMe PCI controller using a PCI endpoint capable PCI + controller.
'This allows creating a NVMe PCI endpoint target using endpoint capable PCI controller.'
drivers/nvme/target/Makefile obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o · line 11
Quoted 11 lines
drivers/nvme/target/Kconfig +2
-0+ + If unsure, say N.drivers/nvme/target/Makefile +1
-0@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o obj-$(CONFIG_NVME_TARGET_FC) += nvmet-fc.o obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o +obj-$(CONFIG_NVME_TARGET_PCI_EP) += nvmet-pciep.o
Same as above: nvmet-pci-epf
Quoted 15 lines
drivers/nvme/target/Makefile +1
-0nvmet-y += core.o configfs.o admin-cmd.o fabrics-cmd.o \ discovery.o io-cmd-file.o io-cmd-bdev.o pr.o@@ -20,4 +21,5 @@ nvmet-rdma-y += rdma.o nvmet-fc-y += fc.o nvme-fcloop-y += fcloop.o nvmet-tcp-y += tcp.o +nvmet-pciep-y += pci-ep.o nvmet-$(CONFIG_TRACING) += trace.odrivers/nvme/target/pci-ep.c new file +0
-0
drivers/nvme/target/pci-ep.c line 3
Quoted 3 lines
drivers/nvme/target/pci-ep.c new file +3
-0@@ -0,0 +1,2626 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* NVMe PCI endpoint device.
'NVMe PCI Endpoint Function driver'
Quoted 12 lines
drivers/nvme/target/pci-ep.c new file +12
-0+* Copyright (c) 2024, Western Digital Corporation or its affiliates. +* Copyright (c) 2024, Rick Wertenbroek <[email protected]> +* REDS Institute, HEIG-VD, HES-SO, Switzerland +*/ +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt + +#include <linux/delay.h> +#include <linux/dmaengine.h> +#include <linux/io.h> +#include <linux/mempool.h> +#include <linux/module.h> +#include <linux/mutex.h>
drivers/nvme/target/pci-ep.c line 25
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+#include <linux/nvme.h> +#include <linux/pci_ids.h> +#include <linux/pci-epc.h> +#include <linux/pci-epf.h> +#include <linux/pci_regs.h> +#include <linux/slab.h> + +#include "nvmet.h" + +static LIST_HEAD (nvmet_pciep_ports);
Please name all variables/functions as 'pci_epf' instead of 'pciep'.
Quoted 1 line
drivers/nvme/target/pci-ep.c new file +1
-0+static DEFINE_MUTEX (nvmet_pciep_ports_mutex);
[...]
drivers/nvme/target/pci-ep.c line 199
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* +* PCI EPF driver private data. +*/ +struct nvmet_pciep_epf {
nvmet_pci_epf?
Quoted 22 lines
drivers/nvme/target/pci-ep.c new file +22
-0+struct pci_epf *epf; + +const struct pci_epc_features *epc_features; + +void *reg_bar; +size_t msix_table_offset; + +unsigned int irq_type; +unsigned int nr_vectors; + +struct nvmet_pciep_ctrl ctrl; + +struct dma_chan *dma_tx_chan; +struct mutex dma_tx_lock; +struct dma_chan *dma_rx_chan; +struct mutex dma_rx_lock; + +struct mutex mmio_lock; + +/* PCI endpoint function configfs attributes */ +struct config_group group; +bool dma_enable;
drivers/nvme/target/pci-ep.c line 231
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+__le16 portid; +char subsysnqn[NVMF_NQN_SIZE ]; +unsigned int mdts_kb; +}; + +static inline u32 nvmet_pciep_bar_read32 (struct nvmet_pciep_ctrl *ctrl,u32 off) +{ +__le32 *bar_reg = ctrl->bar + off; + +return le32_to_cpu (READ_ONCE (*bar_reg));
Looks like you can use readl/writel variants here. Any reason to not use them?
Quoted 2 lines
drivers/nvme/target/pci-ep.c new file +2
-0+} +
[...]
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+static bool nvmet_pciep_epf_init_dma (struct nvmet_pciep_epf *nvme_epf) +{ +struct pci_epf *epf = nvme_epf->epf; +struct device *dev = &epf->dev; +struct nvmet_pciep_epf_dma_filter filter; +struct dma_chan *chan; +dma_cap_mask_t mask; + +mutex_init (&nvme_epf->dma_rx_lock); +mutex_init (&nvme_epf->dma_tx_lock);
drivers/nvme/target/pci-ep.c line 309
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ +dma_cap_zero (mask); +dma_cap_set (DMA_SLAVE , mask); + + filter.dev = epf->epc->dev.parent; + filter.dma_mask =BIT (DMA_DEV_TO_MEM ); + + chan =dma_request_channel (mask, nvmet_pciep_epf_dma_filter, &filter); +if (!chan) +return false;
You should also destroy mutexes in error path.
drivers/nvme/target/pci-ep.c line 316
Quoted 7 lines
drivers/nvme/target/pci-ep.c new file +7
-0+ + nvme_epf->dma_rx_chan = chan; + +dev_dbg (dev,"Using DMA RX channel %s, maximum segment size %u B\n" , +dma_chan_name (chan), +dma_get_max_seg_size (dmaengine_get_dma_device (chan))); +
You should print this message at the end of the function. Otherwise, if there is a problem in acquiring TX, this and 'DMA not supported...' will be printed.
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ filter.dma_mask =BIT (DMA_MEM_TO_DEV ); + chan =dma_request_channel (mask, nvmet_pciep_epf_dma_filter, &filter); +if (!chan) { +dma_release_channel (nvme_epf->dma_rx_chan); + nvme_epf->dma_rx_chan =NULL ; +return false; + } + + nvme_epf->dma_tx_chan = chan; +
drivers/nvme/target/pci-ep.c line 336
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+dev_dbg (dev,"Using DMA TX channel %s, maximum segment size %u B\n" , +dma_chan_name (chan), +dma_get_max_seg_size (dmaengine_get_dma_device (chan))); + +return true; +} + +static void nvmet_pciep_epf_deinit_dma (struct nvmet_pciep_epf *nvme_epf) +{ +if (nvme_epf->dma_tx_chan) {
If you destroy mutexes in error path as I suggested above, you could just bail out if !nvme_epf->dma_enable.
Quoted 64 lines
drivers/nvme/target/pci-ep.c new file +64
-0+dma_release_channel (nvme_epf->dma_tx_chan); + nvme_epf->dma_tx_chan =NULL ; + } + +if (nvme_epf->dma_rx_chan) { +dma_release_channel (nvme_epf->dma_rx_chan); + nvme_epf->dma_rx_chan =NULL ; + } + +mutex_destroy (&nvme_epf->dma_rx_lock); +mutex_destroy (&nvme_epf->dma_tx_lock); +} + +static int nvmet_pciep_epf_dma_transfer (struct nvmet_pciep_epf *nvme_epf, +struct nvmet_pciep_segment *seg,enum dma_data_direction dir) +{ +struct pci_epf *epf = nvme_epf->epf; +struct dma_async_tx_descriptor *desc; +struct dma_slave_config sconf = {}; +struct device *dev = &epf->dev; +struct device *dma_dev; +struct dma_chan *chan; +dma_cookie_t cookie; +dma_addr_t dma_addr; +struct mutex *lock; +int ret; + +switch (dir) { +case DMA_FROM_DEVICE : + lock = &nvme_epf->dma_rx_lock; + chan = nvme_epf->dma_rx_chan; + sconf.direction =DMA_DEV_TO_MEM ; + sconf.src_addr = seg->pci_addr; +break ; +case DMA_TO_DEVICE : + lock = &nvme_epf->dma_tx_lock; + chan = nvme_epf->dma_tx_chan; + sconf.direction =DMA_MEM_TO_DEV ; + sconf.dst_addr = seg->pci_addr; +break ; +default : +return -EINVAL ; + } + +mutex_lock (lock); + + dma_dev =dmaengine_get_dma_device (chan); + dma_addr =dma_map_single (dma_dev, seg->buf, seg->length, dir); + ret =dma_mapping_error (dma_dev, dma_addr); +if (ret) + goto unlock; + + ret =dmaengine_slave_config (chan, &sconf); +if (ret) { +dev_err (dev,"Failed to configure DMA channel\n" ); + goto unmap; + } + + desc =dmaengine_prep_slave_single (chan, dma_addr, seg->length, + sconf.direction,DMA_CTRL_ACK ); +if (!desc) { +dev_err (dev,"Failed to prepare DMA\n" ); + ret = -EIO ; + goto unmap;
drivers/nvme/target/pci-ep.c line 410
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ } + + cookie =dmaengine_submit (desc); + ret =dma_submit_error (cookie); +if (ret) { +dev_err (dev,"DMA submit failed %d\n" , ret); + goto unmap; + } + +if (dma_sync_wait (chan, cookie) !=DMA_COMPLETE ) {
Why do you need to do sync tranfer all the time? This defeats the purpose of using DMA.
Quoted 16 lines
drivers/nvme/target/pci-ep.c new file +16
-0+dev_err (dev,"DMA transfer failed\n" ); + ret = -EIO ; + } + +dmaengine_terminate_sync (chan); + +unmap: +dma_unmap_single (dma_dev, dma_addr, seg->length, dir); + +unlock: +mutex_unlock (lock); + +return ret; +} + +static int nvmet_pciep_epf_mmio_transfer(struct nvmet_pciep_epf *nvme_epf,
drivers/nvme/target/pci-ep.c line 436
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ struct nvmet_pciep_segment *seg, enum dma_data_direction dir) +{ +u64 pci_addr = seg->pci_addr; +u32 length = seg->length; +void *buf = seg->buf; +struct pci_epc_map map; +int ret = -EINVAL ; + +/* +* Note: mmio transfers do not need serialization but this is a
MMIO
Quoted 25 lines
drivers/nvme/target/pci-ep.c new file +25
-0+* simple way to avoid using too many mapping windows. +*/ +mutex_lock (&nvme_epf->mmio_lock); + +while (length) { + ret =nvmet_pciep_epf_mem_map (nvme_epf, pci_addr, length, &map); +if (ret) +break ; + +switch (dir) { +case DMA_FROM_DEVICE : +memcpy_fromio (buf, map.virt_addr, map.pci_size); +break ; +case DMA_TO_DEVICE : +memcpy_toio (map.virt_addr, buf, map.pci_size); +break ; +default : + ret = -EINVAL ; + goto unlock; + } + + pci_addr += map.pci_size; + buf += map.pci_size; + length -= map.pci_size; +
drivers/nvme/target/pci-ep.c line 471
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+nvmet_pciep_epf_mem_unmap (nvme_epf, &map); + } + +unlock: +mutex_unlock (&nvme_epf->mmio_lock); + +return ret; +} + +static inline int nvmet_pciep_epf_transfer(struct nvmet_pciep_epf *nvme_epf,
No need to add 'inline' keyword in .c files.
Quoted 8 lines
drivers/nvme/target/pci-ep.c new file +8
-0+ struct nvmet_pciep_segment *seg, enum dma_data_direction dir) +{ +if (nvme_epf->dma_enable) +return nvmet_pciep_epf_dma_transfer (nvme_epf, seg, dir); + +return nvmet_pciep_epf_mmio_transfer (nvme_epf, seg, dir); +} +
[...]
Quoted 2 lines
+/*
+ * Transfer a prp list from the host and return the number of prps.
PRP (everywhere in comments)
Quoted 28 lines
+ */
drivers/nvme/target/pci-ep.c new file +27
-0+static int nvmet_pciep_get_prp_list (struct nvmet_pciep_ctrl *ctrl,u64 prp, +size_t xfer_len,__le64 *prps) +{ +size_t nr_prps = (xfer_len + ctrl->mps_mask) >> ctrl->mps_shift; +u32 length; +int ret; + +/* +* Compute the number of PRPs required for the number of bytes to +* transfer (xfer_len). If this number overflows the memory page size +* with the PRP list pointer specified, only return the space available +* in the memory page, the last PRP in there will be a PRP list pointer +* to the remaining PRPs. +*/ + length =min (nvmet_pciep_prp_size (ctrl, prp), nr_prps <<3 ); + ret =nvmet_pciep_transfer (ctrl, prps, prp, length,DMA_FROM_DEVICE ); +if (ret) +return ret; + +return length >>3 ; +} + +static int nvmet_pciep_iod_parse_prp_list (struct nvmet_pciep_ctrl *ctrl, +struct nvmet_pciep_iod *iod) +{ +struct nvme_command *cmd = &iod->cmd; +struct nvmet_pciep_segment *seg;
drivers/nvme/target/pci-ep.c line 796
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+size_t size =0 , ofst, prp_size, xfer_len; +size_t transfer_len = iod->data_len; +int nr_segs, nr_prps =0 ; +u64 pci_addr, prp; +int i =0 , ret; +__le64 *prps; + + prps =kzalloc (ctrl->mps,GFP_KERNEL ); +if (!prps) + goto internal;
Can you prefix 'err_' to the label?
Quoted 105 lines
drivers/nvme/target/pci-ep.c new file +105
-0+ +/* +* Allocate PCI segments for the command: this considers the worst case +* scenario where all prps are discontiguous, so get as many segments +* as we can have prps. In practice, most of the time, we will have +* far less PCI segments than prps. +*/ + prp =le64_to_cpu (cmd->common.dptr.prp1); +if (!prp) + goto invalid_field; + + ofst =nvmet_pciep_prp_ofst (ctrl, prp); + nr_segs = (transfer_len + ofst + ctrl->mps -1 ) >> ctrl->mps_shift; + + ret =nvmet_pciep_alloc_iod_data_segs (iod, nr_segs); +if (ret) + goto internal; + +/* Set the first segment using prp1 */ + seg = &iod->data_segs[0 ]; + seg->pci_addr = prp; + seg->length =nvmet_pciep_prp_size (ctrl, prp); + + size = seg->length; + pci_addr = prp + size; + nr_segs =1 ; + +/* +* Now build the PCI address segments using the prp lists, starting +* from prp2. +*/ + prp =le64_to_cpu (cmd->common.dptr.prp2); +if (!prp) + goto invalid_field; + +while (size < transfer_len) { + xfer_len = transfer_len - size; + +if (!nr_prps) { +/* Get the prp list */ + nr_prps =nvmet_pciep_get_prp_list (ctrl, prp, + xfer_len, prps); +if (nr_prps <0 ) + goto internal; + + i =0 ; + ofst =0 ; + } + +/* Current entry */ + prp =le64_to_cpu (prps[i]); +if (!prp) + goto invalid_field; + +/* Did we reach the last prp entry of the list ? */ +if (xfer_len > ctrl->mps && i == nr_prps -1 ) { +/* We need more PRPs: prp is a list pointer */ + nr_prps =0 ; +continue ; + } + +/* Only the first prp is allowed to have an offset */ +if (nvmet_pciep_prp_ofst (ctrl, prp)) + goto invalid_offset; + +if (prp != pci_addr) { +/* Discontiguous prp: new segment */ + nr_segs++; +if (WARN_ON_ONCE (nr_segs > iod->nr_data_segs)) + goto internal; + + seg++; + seg->pci_addr = prp; + seg->length =0 ; + pci_addr = prp; + } + + prp_size =min_t (size_t, ctrl->mps, xfer_len); + seg->length += prp_size; + pci_addr += prp_size; + size += prp_size; + + i++; + } + + iod->nr_data_segs = nr_segs; + ret =0 ; + +if (size != transfer_len) { +dev_err (ctrl->dev,"PRPs transfer length mismatch %zu / %zu\n" , + size, transfer_len); + goto internal; + } + +kfree (prps); + +return 0 ; + +invalid_offset: +dev_err (ctrl->dev,"PRPs list invalid offset\n" ); +kfree (prps); + iod->status =NVME_SC_PRP_INVALID_OFFSET |NVME_STATUS_DNR ; +return -EINVAL ; + +invalid_field:
drivers/nvme/target/pci-ep.c line 911
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+dev_err (ctrl->dev,"PRPs list invalid field\n" ); +kfree (prps); + iod->status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; +return -EINVAL ; + +internal: +dev_err (ctrl->dev,"PRPs list internal error\n" ); +kfree (prps); + iod->status =NVME_SC_INTERNAL |NVME_STATUS_DNR ; +return -EINVAL ;
Can't you organize the labels in such a way that there is only one return path? Current code makes it difficult to read and also would confuse the static checkers.
Quoted 2 lines
drivers/nvme/target/pci-ep.c new file +2
-0+} +
[...]
Quoted 16 lines
drivers/nvme/target/pci-ep.c new file +16
-0+static void nvmet_pciep_init_bar (struct nvmet_pciep_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + + ctrl->bar = ctrl->nvme_epf->reg_bar; + +/* Copy the target controller capabilities as a base */ + ctrl->cap = tctrl->cap; + +/* Contiguous Queues Required (CQR) */ + ctrl->cap |=0x1ULL <<16 ; + +/* Set Doorbell stride to 4B (DSTRB) */ + ctrl->cap &= ~GENMASK (35 ,32 ); + +/* Clear NVM Subsystem Reset Supported (NSSRS) */
drivers/nvme/target/pci-ep.c line 1960
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ ctrl->cap &= ~(0x1ULL <<36 ); + +/* Clear Boot Partition Support (BPS) */ + ctrl->cap &= ~(0x1ULL <<45 ); + +/* Clear Persistent Memory Region Supported (PMRS) */ + ctrl->cap &= ~(0x1ULL <<56 ); + +/* Clear Controller Memory Buffer Supported (CMBS) */ + ctrl->cap &= ~(0x1ULL <<57 );
Can you use macros for these?
Quoted 13 lines
drivers/nvme/target/pci-ep.c new file +13
-0+ +/* Controller configuration */ + ctrl->cc = tctrl->cc & (~NVME_CC_ENABLE ); + +/* Controller status */ + ctrl->csts = ctrl->tctrl->csts; + +nvmet_pciep_bar_write64 (ctrl,NVME_REG_CAP , ctrl->cap); +nvmet_pciep_bar_write32 (ctrl,NVME_REG_VS , tctrl->subsys->ver); +nvmet_pciep_bar_write32 (ctrl,NVME_REG_CSTS , ctrl->csts); +nvmet_pciep_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); +} +
[...]
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+static int nvmet_pciep_epf_configure_bar (struct nvmet_pciep_epf *nvme_epf) +{ +struct pci_epf *epf = nvme_epf->epf; +const struct pci_epc_features *epc_features = nvme_epf->epc_features;
drivers/nvme/target/pci-ep.c line 2110
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+size_t reg_size, reg_bar_size; +size_t msix_table_size =0 ; + +/* +* The first free BAR will be our register BAR and per NVMe +* specifications, it must be BAR 0. +*/ +if (pci_epc_get_first_free_bar (epc_features) !=BAR_0 ) { +dev_err (&epf->dev,"BAR 0 is not free\n" ); +return -EINVAL ;
-ENOMEM or -ENODEV?
Quoted 15 lines
drivers/nvme/target/pci-ep.c new file +15
-0+ } + +/* Initialize BAR flags */ +if (epc_features->bar[BAR_0 ].only_64bit) + epf->bar[BAR_0 ].flags |=PCI_BASE_ADDRESS_MEM_TYPE_64 ; + +/* +* Calculate the size of the register bar: NVMe registers first with +* enough space for the doorbells, followed by the MSI-X table +* if supported. +*/ + reg_size =NVME_REG_DBS + (NVMET_NR_QUEUES *2 *sizeof (u32)); + reg_size =ALIGN (reg_size,8 ); + +if (epc_features->msix_capable) {
drivers/nvme/target/pci-ep.c line 2135
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+size_t pba_size; + + msix_table_size =PCI_MSIX_ENTRY_SIZE * epf->msix_interrupts; + nvme_epf->msix_table_offset = reg_size; + pba_size =ALIGN (DIV_ROUND_UP (epf->msix_interrupts,8 ),8 ); + + reg_size += msix_table_size + pba_size; + } + + reg_bar_size =ALIGN (reg_size,max (epc_features->align,4096 ));
From where does this 4k alignment comes from? NVMe spec? If so, is it OK to use fixed_size BAR?
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ +if (epc_features->bar[BAR_0 ].type ==BAR_FIXED ) { +if (reg_bar_size > epc_features->bar[BAR_0 ].fixed_size) { + dev_err(&epf->dev, + "Reg BAR 0 size %llu B too small, need %zu B\n", + epc_features->bar[BAR_0].fixed_size,
drivers/nvme/target/pci-ep.c line 2151
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ reg_bar_size); +return -ENOMEM ; + } + reg_bar_size = epc_features->bar[BAR_0 ].fixed_size; + } + + nvme_epf->reg_bar =pci_epf_alloc_space (epf, reg_bar_size,BAR_0 , + epc_features,PRIMARY_INTERFACE ); +if (!nvme_epf->reg_bar) { +dev_err (&epf->dev,"Allocate BAR 0 failed\n" );
'Failed to allocate memory for BAR 0'
Quoted 85 lines
drivers/nvme/target/pci-ep.c new file +85
-0+return -ENOMEM ; + } +memset (nvme_epf->reg_bar,0 , reg_bar_size); + +return 0 ; +} + +static void nvmet_pciep_epf_clear_bar (struct nvmet_pciep_epf *nvme_epf) +{ +struct pci_epf *epf = nvme_epf->epf; + +pci_epc_clear_bar (epf->epc, epf->func_no, epf->vfunc_no, + &epf->bar[BAR_0 ]); +pci_epf_free_space (epf, nvme_epf->reg_bar,BAR_0 ,PRIMARY_INTERFACE ); + nvme_epf->reg_bar =NULL ; +} + +static int nvmet_pciep_epf_init_irq (struct nvmet_pciep_epf *nvme_epf) +{ +const struct pci_epc_features *epc_features = nvme_epf->epc_features; +struct pci_epf *epf = nvme_epf->epf; +int ret; + +/* Enable MSI-X if supported, otherwise, use MSI */ +if (epc_features->msix_capable && epf->msix_interrupts) { + ret =pci_epc_set_msix (epf->epc, epf->func_no, epf->vfunc_no, + epf->msix_interrupts,BAR_0 , + nvme_epf->msix_table_offset); +if (ret) { +dev_err (&epf->dev,"MSI-X configuration failed\n" ); +return ret; + } + + nvme_epf->nr_vectors = epf->msix_interrupts; + nvme_epf->irq_type =PCI_IRQ_MSIX ; + +return 0 ; + } + +if (epc_features->msi_capable && epf->msi_interrupts) { + ret =pci_epc_set_msi (epf->epc, epf->func_no, epf->vfunc_no, + epf->msi_interrupts); +if (ret) { +dev_err (&epf->dev,"MSI configuration failed\n" ); +return ret; + } + + nvme_epf->nr_vectors = epf->msi_interrupts; + nvme_epf->irq_type =PCI_IRQ_MSI ; + +return 0 ; + } + +/* MSI and MSI-X are not supported: fall back to INTX */ + nvme_epf->nr_vectors =1 ; + nvme_epf->irq_type =PCI_IRQ_INTX ; + +return 0 ; +} + +static int nvmet_pciep_epf_epc_init (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +const struct pci_epc_features *epc_features = nvme_epf->epc_features; +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; +unsigned int max_nr_queues =NVMET_NR_QUEUES ; +int ret; + +/* +* Cap the maximum number of queues we can support on the controller +* with the number of IRQs we can use. +*/ +if (epc_features->msix_capable && epf->msix_interrupts) { +dev_info (&epf->dev, +"PCI endpoint controller supports MSI-X, %u vectors\n" , + epf->msix_interrupts); + max_nr_queues =min (max_nr_queues, epf->msix_interrupts); + }else if (epc_features->msi_capable && epf->msi_interrupts) { +dev_info (&epf->dev, +"PCI endpoint controller supports MSI, %u vectors\n" , + epf->msi_interrupts); + max_nr_queues =min (max_nr_queues, epf->msi_interrupts); + } + +if (max_nr_queues <2 ) {
drivers/nvme/target/pci-ep.c line 2246
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+dev_err (&epf->dev,"Invalid maximum number of queues %u\n" , + max_nr_queues); +return -EINVAL ; + } + +/* Create the target controller. */ + ret =nvmet_pciep_create_ctrl (nvme_epf, max_nr_queues); +if (ret) { +dev_err (&epf->dev, +"Create NVMe PCI target controller failed\n" );
Failed to create NVMe PCI target controller
drivers/nvme/target/pci-ep.c line 2250
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+return ret; + } + +if (epf->vfunc_no <=1 ) {
Are you really supporting virtual functions? If supported, 'vfunc_no < 1' is not possible.
drivers/nvme/target/pci-ep.c line 2254
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* Set device ID, class, etc */ + epf->header->vendorid = ctrl->tctrl->subsys->vendor_id; + epf->header->subsys_vendor_id = + ctrl->tctrl->subsys->subsys_vendor_id;
Why these are coming from somewhere else and not configured within the EPF driver?
Quoted 49 lines
drivers/nvme/target/pci-ep.c new file +49
-0+ ret =pci_epc_write_header (epf->epc, epf->func_no, epf->vfunc_no, + epf->header); +if (ret) { +dev_err (&epf->dev, +"Write configuration header failed %d\n" , ret); + goto out_destroy_ctrl; + } + } + +/* Setup the PCIe BAR and create the controller */ + ret =pci_epc_set_bar (epf->epc, epf->func_no, epf->vfunc_no, + &epf->bar[BAR_0 ]); +if (ret) { +dev_err (&epf->dev,"Set BAR 0 failed\n" ); + goto out_destroy_ctrl; + } + +/* +* Enable interrupts and start polling the controller BAR if we do not +* have any link up notifier. +*/ + ret =nvmet_pciep_epf_init_irq (nvme_epf); +if (ret) + goto out_clear_bar; + +if (!epc_features->linkup_notifier) { + ctrl->link_up = true; +nvmet_pciep_start_ctrl (&nvme_epf->ctrl); + } + +return 0 ; + +out_clear_bar: +nvmet_pciep_epf_clear_bar (nvme_epf); +out_destroy_ctrl: +nvmet_pciep_destroy_ctrl (&nvme_epf->ctrl); +return ret; +} + +static void nvmet_pciep_epf_epc_deinit (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + + ctrl->link_up = false; +nvmet_pciep_destroy_ctrl (ctrl); + +nvmet_pciep_epf_deinit_dma (nvme_epf); +nvmet_pciep_epf_clear_bar (nvme_epf);
drivers/nvme/target/pci-ep.c line 2313
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+ +mutex_destroy (&nvme_epf->mmio_lock); +} + +static int nvmet_pciep_epf_link_up (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + +dev_info (nvme_epf->ctrl.dev,"PCI link up\n" );
These prints are supposed to come from the controller drivers. So no need to have them here also.
Quoted 7 lines
drivers/nvme/target/pci-ep.c new file +7
-0+ + ctrl->link_up = true; +nvmet_pciep_start_ctrl (ctrl); + +return 0 ; +} +
[...]
Quoted 17 lines
drivers/nvme/target/pci-ep.c new file +17
-0+static ssize_t nvmet_pciep_epf_dma_enable_show (struct config_item *item, +char *page) +{ +struct config_group *group =to_config_group (item); +struct nvmet_pciep_epf *nvme_epf =to_nvme_epf (group); + +return sysfs_emit (page,"%d\n" , nvme_epf->dma_enable); +} + +static ssize_t nvmet_pciep_epf_dma_enable_store (struct config_item *item, +const char *page,size_t len) +{ +struct config_group *group =to_config_group (item); +struct nvmet_pciep_epf *nvme_epf =to_nvme_epf (group); +int ret; + +if (nvme_epf->ctrl.tctrl)
drivers/nvme/target/pci-ep.c line 2453
Quoted 10 lines
drivers/nvme/target/pci-ep.c new file +10
-0+return -EBUSY ; + + ret =kstrtobool (page, &nvme_epf->dma_enable); +if (ret) +return ret; + +return len; +} + +CONFIGFS_ATTR (nvmet_pciep_epf_, dma_enable);
What is the use of making this option user configurable? It is purely a hardware capability and I don't think users would want to have their NVMe device working without DMA voluntarily.
Quoted 9 lines
drivers/nvme/target/pci-ep.c new file +9
-0+ +static ssize_t nvmet_pciep_epf_portid_show (struct config_item *item,char *page) +{ +struct config_group *group =to_config_group (item); +struct nvmet_pciep_epf *nvme_epf =to_nvme_epf (group); + +return sysfs_emit (page,"%u\n" ,le16_to_cpu (nvme_epf->portid)); +} +
[...]
drivers/nvme/target/pci-ep.c line 2606
Quoted 9 lines
drivers/nvme/target/pci-ep.c new file +9
-0+static int __init nvmet_pciep_init_module (void ) +{ +int ret; + + ret =pci_epf_register_driver (&nvmet_pciep_epf_driver); +if (ret) +return ret; + + ret =nvmet_register_transport (&nvmet_pciep_fabrics_ops);
What is the need to register the transport so early? You should consider moving the registration to bind() so that the transport can be removed once the driver is unbind with the controller.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 35 lines · Manivannan Sadhasivam
[...]
drivers/nvme/target/pci-ep.c line 199
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* +* PCI EPF driver private data. +*/ +struct nvmet_pciep_epf {
nvmet_pci_epf?Quoted 26 lines
drivers/nvme/target/pci-ep.c new file +26
-0+struct pci_epf *epf; + +const struct pci_epc_features *epc_features; + +void *reg_bar; +size_t msix_table_offset; + +unsigned int irq_type; +unsigned int nr_vectors; + +struct nvmet_pciep_ctrl ctrl; + +struct dma_chan *dma_tx_chan; +struct mutex dma_tx_lock; +struct dma_chan *dma_rx_chan; +struct mutex dma_rx_lock; + +struct mutex mmio_lock; + +/* PCI endpoint function configfs attributes */ +struct config_group group; +bool dma_enable; +__le16 portid; +char subsysnqn[NVMF_NQN_SIZE ]; +unsigned int mdts_kb; +};
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 231
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ +static inline u32 nvmet_pciep_bar_read32 (struct nvmet_pciep_ctrl *ctrl,u32 off) +{ +__le32 *bar_reg = ctrl->bar + off; + +return le32_to_cpu (READ_ONCE (*bar_reg));Looks like you can use readl/writel variants here. Any reason to not use them?
A bar memory comes from dma_alloc_coherent(), which is a "void *" pointer and *not* iomem. So using readl/writel for accessing that memory seems awefully wrong to me.
Quoted 14 lines
Quoted 14 lines
drivers/nvme/target/pci-ep.c new file +14
-0+static bool nvmet_pciep_epf_init_dma (struct nvmet_pciep_epf *nvme_epf) +{ +struct pci_epf *epf = nvme_epf->epf; +struct device *dev = &epf->dev; +struct nvmet_pciep_epf_dma_filter filter; +struct dma_chan *chan; +dma_cap_mask_t mask; + +mutex_init (&nvme_epf->dma_rx_lock); +mutex_init (&nvme_epf->dma_tx_lock); + +dma_cap_zero (mask); +dma_cap_set (DMA_SLAVE , mask); +
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 309
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ filter.dev = epf->epc->dev.parent; + filter.dma_mask =BIT (DMA_DEV_TO_MEM ); + + chan =dma_request_channel (mask, nvmet_pciep_epf_dma_filter, &filter); +if (!chan) +return false;You should also destroy mutexes in error path.
Good catch. Will add that.
[...]
Quoted 56 lines
Quoted 56 lines
drivers/nvme/target/pci-ep.c new file +56
-0+static int nvmet_pciep_epf_dma_transfer (struct nvmet_pciep_epf *nvme_epf, +struct nvmet_pciep_segment *seg,enum dma_data_direction dir) +{ +struct pci_epf *epf = nvme_epf->epf; +struct dma_async_tx_descriptor *desc; +struct dma_slave_config sconf = {}; +struct device *dev = &epf->dev; +struct device *dma_dev; +struct dma_chan *chan; +dma_cookie_t cookie; +dma_addr_t dma_addr; +struct mutex *lock; +int ret; + +switch (dir) { +case DMA_FROM_DEVICE : + lock = &nvme_epf->dma_rx_lock; + chan = nvme_epf->dma_rx_chan; + sconf.direction =DMA_DEV_TO_MEM ; + sconf.src_addr = seg->pci_addr; +break ; +case DMA_TO_DEVICE : + lock = &nvme_epf->dma_tx_lock; + chan = nvme_epf->dma_tx_chan; + sconf.direction =DMA_MEM_TO_DEV ; + sconf.dst_addr = seg->pci_addr; +break ; +default : +return -EINVAL ; + } + +mutex_lock (lock); + + dma_dev =dmaengine_get_dma_device (chan); + dma_addr =dma_map_single (dma_dev, seg->buf, seg->length, dir); + ret =dma_mapping_error (dma_dev, dma_addr); +if (ret) + goto unlock; + + ret =dmaengine_slave_config (chan, &sconf); +if (ret) { +dev_err (dev,"Failed to configure DMA channel\n" ); + goto unmap; + } + + desc =dmaengine_prep_slave_single (chan, dma_addr, seg->length, + sconf.direction,DMA_CTRL_ACK ); +if (!desc) { +dev_err (dev,"Failed to prepare DMA\n" ); + ret = -EIO ; + goto unmap; + } + + cookie =dmaengine_submit (desc); + ret =dma_submit_error (cookie); +if (ret) {
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 410
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+dev_err (dev,"DMA submit failed %d\n" , ret); + goto unmap; + } + +if (dma_sync_wait (chan, cookie) !=DMA_COMPLETE ) {Why do you need to do sync tranfer all the time? This defeats the purpose of using DMA.
You may be getting confused about the meaning of sync here. This simply means "spin wait for the completion of the transfer". I initially was using the completion callback like in pci-epf-tets and othe EPF drivers doing DMA, but that causes a context switch for every transfer, even small ones, which really hurts performance. Switching to using dma_sync_wait() avoids this context switch while achieving what we need, which is to wait for the end of the transfer. I nearly doubled the max IOPS for small IOs with this.
Note that we cannot easily do asynchronous DMA transfers with the current DMA slave API, unless we create some special work item responsible for DMA transfers.
I did not try to micro optimize this driver too much for this first drop. Any improvement around how data transfers are done can come later. The preformance I am getting with this code is decent enough as it is.
[...]
Quoted 3 lines
Quoted 1 line
+static inline int nvmet_pciep_epf_transfer(struct
nvmet_pciep_epf*nvme_epf,No need to add 'inline' keyword in .c files.
Yes there is. Because that funtion is tiny and I really want it to be forced to be inlined.
[...]
Quoted 13 lines
Quoted 13 lines
drivers/nvme/target/pci-ep.c new file +13
-0+invalid_offset: +dev_err (ctrl->dev,"PRPs list invalid offset\n" ); +kfree (prps); + iod->status =NVME_SC_PRP_INVALID_OFFSET |NVME_STATUS_DNR ; +return -EINVAL ; + +invalid_field: +dev_err (ctrl->dev,"PRPs list invalid field\n" ); +kfree (prps); + iod->status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; +return -EINVAL ; + +internal:
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 911
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+dev_err (ctrl->dev,"PRPs list internal error\n" ); +kfree (prps); + iod->status =NVME_SC_INTERNAL |NVME_STATUS_DNR ; +return -EINVAL ;Can't you organize the labels in such a way that there is only one return path? Current code makes it difficult to read and also would confuse the static checkers.
I do not see how this is difficult to read... But sure, I can try to simplify this.
Quoted 22 lines
[...]
Quoted 20 lines
drivers/nvme/target/pci-ep.c new file +20
-0+static void nvmet_pciep_init_bar (struct nvmet_pciep_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + + ctrl->bar = ctrl->nvme_epf->reg_bar; + +/* Copy the target controller capabilities as a base */ + ctrl->cap = tctrl->cap; + +/* Contiguous Queues Required (CQR) */ + ctrl->cap |=0x1ULL <<16 ; + +/* Set Doorbell stride to 4B (DSTRB) */ + ctrl->cap &= ~GENMASK (35 ,32 ); + +/* Clear NVM Subsystem Reset Supported (NSSRS) */ + ctrl->cap &= ~(0x1ULL <<36 ); + +/* Clear Boot Partition Support (BPS) */ + ctrl->cap &= ~(0x1ULL <<45 );
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 1960
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ +/* Clear Persistent Memory Region Supported (PMRS) */ + ctrl->cap &= ~(0x1ULL <<56 ); + +/* Clear Controller Memory Buffer Supported (CMBS) */ + ctrl->cap &= ~(0x1ULL <<57 );Can you use macros for these?
I could. But that would mean defining *all* bits of the cap register, which is a lot... The NVMe code does not have all these defined now. That is a cleanup that can come later I think.
[...]
Quoted 6 lines
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+if (epc_features->msix_capable) { +size_t pba_size; + + msix_table_size =PCI_MSIX_ENTRY_SIZE * epf->msix_interrupts; + nvme_epf->msix_table_offset = reg_size; + pba_size =ALIGN (DIV_ROUND_UP (epf->msix_interrupts,8 ),8 );
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2135
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+ + reg_size += msix_table_size + pba_size; + } + + reg_bar_size =ALIGN (reg_size,max (epc_features->align,4096 ));From where does this 4k alignment comes from? NVMe spec? If so, is it OK to use
fixed_sizeBAR?
NVMe BAR cannot be fixed size as its size depends on the number of queue pairs the controller can support. Will check if the 4K alignment is mandated by the specs. But it sure does not hurt...
[...]
Quoted 7 lines
drivers/nvme/target/pci-ep.c line 2246
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+/* Create the target controller. */ + ret =nvmet_pciep_create_ctrl (nvme_epf, max_nr_queues); +if (ret) { +dev_err (&epf->dev, +"Create NVMe PCI target controller failed\n" );Failed to create NVMe PCI target controller
How is that better ?
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2250
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +4
-0+return ret; + } + +if (epf->vfunc_no <=1 ) {Are you really supporting virtual functions? If supported, '
vfunc_no< 1' is not possible.
NVMe does support virtual functions and nothing in the configuration path prevents the user from setting one such function. So yes, this is supposed to work if the endpoint controller supports it. Though I must say that I have not tried/tested yet.
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2254
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* Set device ID, class, etc */ + epf->header->vendorid = ctrl->tctrl->subsys->vendor_id; + epf->header->subsys_vendor_id = + ctrl->tctrl->subsys->subsys_vendor_id;Why these are coming from somewhere else and not configured within the EPF driver?
They are set through the nvme target configfs. So there is no need to have these again setup through the epf configfs. We just grab the values set for the NVME target subsystem config.
Quoted 9 lines
drivers/nvme/target/pci-ep.c line 2313
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+static int nvmet_pciep_epf_link_up (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + +dev_info (nvme_epf->ctrl.dev,"PCI link up\n" );These prints are supposed to come from the controller drivers. So no need to have them here also.
Nope, the controller driver does not print anything. At least the DWC driver does not print anything.
Quoted 14 lines
Quoted 14 lines
drivers/nvme/target/pci-ep.c new file +14
-0+static ssize_t nvmet_pciep_epf_dma_enable_store (struct config_item *item, +const char *page,size_t len) +{ +struct config_group *group =to_config_group (item); +struct nvmet_pciep_epf *nvme_epf =to_nvme_epf (group); +int ret; + +if (nvme_epf->ctrl.tctrl) +return -EBUSY ; + + ret =kstrtobool (page, &nvme_epf->dma_enable); +if (ret) +return ret; +
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2453
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+return len; +} + +CONFIGFS_ATTR (nvmet_pciep_epf_, dma_enable);What is the use of making this option user configurable? It is purely a hardware capability and I don't think users would want to have their NVMe device working without DMA voluntarily.
If you feel strongly about it, I can drop this. But it is useful for debugging purpose to check that the DMA API is being used and working correctly.
Quoted 5 lines
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+static int __init nvmet_pciep_init_module (void ) +{ +int ret; + + ret =pci_epf_register_driver (&nvmet_pciep_epf_driver);
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2606
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+if (ret) +return ret; + + ret =nvmet_register_transport (&nvmet_pciep_fabrics_ops);What is the need to register the transport so early? You should consider moving the registration to
bind()so that the transport can be removed once the driver is unbind with the controller.
This registration is needed to enable the configfs configuration of the nvme target subsystem and port. Withoutn this, setting a port transporet type to "pci" would fail to find the pci transport implemented here and the configuration would fail. This configuration of the nvme target susbsystem and port must come before configuring the EPF and binding it.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 45 lines · Damien Le Moal
Quoted 35 lines · Manivannan Sadhasivam
[...]
drivers/nvme/target/pci-ep.c line 199
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* +* PCI EPF driver private data. +*/ +struct nvmet_pciep_epf {
nvmet_pci_epf?Quoted 26 lines
drivers/nvme/target/pci-ep.c new file +26
-0+struct pci_epf *epf; + +const struct pci_epc_features *epc_features; + +void *reg_bar; +size_t msix_table_offset; + +unsigned int irq_type; +unsigned int nr_vectors; + +struct nvmet_pciep_ctrl ctrl; + +struct dma_chan *dma_tx_chan; +struct mutex dma_tx_lock; +struct dma_chan *dma_rx_chan; +struct mutex dma_rx_lock; + +struct mutex mmio_lock; + +/* PCI endpoint function configfs attributes */ +struct config_group group; +bool dma_enable; +__le16 portid; +char subsysnqn[NVMF_NQN_SIZE ]; +unsigned int mdts_kb; +};Quoted 8 lines
drivers/nvme/target/pci-ep.c line 231
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ +static inline u32 nvmet_pciep_bar_read32 (struct nvmet_pciep_ctrl *ctrl,u32 off) +{ +__le32 *bar_reg = ctrl->bar + off; + +return le32_to_cpu (READ_ONCE (*bar_reg));Looks like you can use readl/writel variants here. Any reason to not use them?
Quoted 4 lines
A bar memory comes from
dma_alloc_coherent(), which is a "void *" pointer and *not* iomem. So using readl/writel for accessing that memory seems awefully wrong to me.
'iomem' is just a token that is used by sparse tool to verify correctness of MMIO address handling. I know that the memory here is returned by dma_alloc_coherent() and is not exactly a MMIO. But since you are using READ_ONCE() and le32_to_cpu() on these addresses, I thought about suggesting readl/writel that does exactly the same internally. If you do not need ordering, then you can use relaxed variants as well.
Quoted 100 lines
Quoted 14 lines
Quoted 14 lines
drivers/nvme/target/pci-ep.c new file +14
-0+static bool nvmet_pciep_epf_init_dma (struct nvmet_pciep_epf *nvme_epf) +{ +struct pci_epf *epf = nvme_epf->epf; +struct device *dev = &epf->dev; +struct nvmet_pciep_epf_dma_filter filter; +struct dma_chan *chan; +dma_cap_mask_t mask; + +mutex_init (&nvme_epf->dma_rx_lock); +mutex_init (&nvme_epf->dma_tx_lock); + +dma_cap_zero (mask); +dma_cap_set (DMA_SLAVE , mask); +Quoted 8 lines
drivers/nvme/target/pci-ep.c line 309
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ filter.dev = epf->epc->dev.parent; + filter.dma_mask =BIT (DMA_DEV_TO_MEM ); + + chan =dma_request_channel (mask, nvmet_pciep_epf_dma_filter, &filter); +if (!chan) +return false;You should also destroy mutexes in error path.
Good catch. Will add that.
[...]
Quoted 56 lines
Quoted 56 lines
drivers/nvme/target/pci-ep.c new file +56
-0+static int nvmet_pciep_epf_dma_transfer (struct nvmet_pciep_epf *nvme_epf, +struct nvmet_pciep_segment *seg,enum dma_data_direction dir) +{ +struct pci_epf *epf = nvme_epf->epf; +struct dma_async_tx_descriptor *desc; +struct dma_slave_config sconf = {}; +struct device *dev = &epf->dev; +struct device *dma_dev; +struct dma_chan *chan; +dma_cookie_t cookie; +dma_addr_t dma_addr; +struct mutex *lock; +int ret; + +switch (dir) { +case DMA_FROM_DEVICE : + lock = &nvme_epf->dma_rx_lock; + chan = nvme_epf->dma_rx_chan; + sconf.direction =DMA_DEV_TO_MEM ; + sconf.src_addr = seg->pci_addr; +break ; +case DMA_TO_DEVICE : + lock = &nvme_epf->dma_tx_lock; + chan = nvme_epf->dma_tx_chan; + sconf.direction =DMA_MEM_TO_DEV ; + sconf.dst_addr = seg->pci_addr; +break ; +default : +return -EINVAL ; + } + +mutex_lock (lock); + + dma_dev =dmaengine_get_dma_device (chan); + dma_addr =dma_map_single (dma_dev, seg->buf, seg->length, dir); + ret =dma_mapping_error (dma_dev, dma_addr); +if (ret) + goto unlock; + + ret =dmaengine_slave_config (chan, &sconf); +if (ret) { +dev_err (dev,"Failed to configure DMA channel\n" ); + goto unmap; + } + + desc =dmaengine_prep_slave_single (chan, dma_addr, seg->length, + sconf.direction,DMA_CTRL_ACK ); +if (!desc) { +dev_err (dev,"Failed to prepare DMA\n" ); + ret = -EIO ; + goto unmap; + } + + cookie =dmaengine_submit (desc); + ret =dma_submit_error (cookie); +if (ret) {Quoted 8 lines
drivers/nvme/target/pci-ep.c line 410
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+dev_err (dev,"DMA submit failed %d\n" , ret); + goto unmap; + } + +if (dma_sync_wait (chan, cookie) !=DMA_COMPLETE ) {Why do you need to do sync tranfer all the time? This defeats the purpose of using DMA.
You may be getting confused about the meaning of sync here. This simply means "spin wait for the completion of the transfer". I initially was using the completion callback like in pci-epf-tets and othe EPF drivers doing DMA, but that causes a context switch for every transfer, even small ones, which really hurts performance. Switching to using
dma_sync_wait()avoids this context switch while achieving what we need, which is to wait for the end of the transfer. I nearly doubled the max IOPS for small IOs with this.
Quoted 3 lines
Note that we cannot easily do asynchronous DMA transfers with the current DMA slave API, unless we create some special work item responsible for DMA transfers.
Right, async transfer requires some changes as well.
Quoted 4 lines
I did not try to micro optimize this driver too much for this first drop. Any improvement around how data transfers are done can come later. The preformance I am getting with this code is decent enough as it is.
That's fine with me to do it later. But if you do it, then you'll get way better performance.
Quoted 9 lines
[...]
Quoted 3 lines
Quoted 1 line
+static inline int nvmet_pciep_epf_transfer(struct
nvmet_pciep_epf*nvme_epf,No need to add 'inline' keyword in .c files.
Yes there is. Because that funtion is tiny and I really want it to be forced to be inlined.
You cannot force the compiler to inline to your function using 'inline' keyword. It just acts as a hint to the compiler and the compiler may or may not inline it depending on its own logic. You can leave the inline keyword and let compiler inline it when needed.
Quoted 24 lines
[...]
Quoted 13 lines
Quoted 13 lines
drivers/nvme/target/pci-ep.c new file +13
-0+invalid_offset: +dev_err (ctrl->dev,"PRPs list invalid offset\n" ); +kfree (prps); + iod->status =NVME_SC_PRP_INVALID_OFFSET |NVME_STATUS_DNR ; +return -EINVAL ; + +invalid_field: +dev_err (ctrl->dev,"PRPs list invalid field\n" ); +kfree (prps); + iod->status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; +return -EINVAL ; + +internal:Quoted 8 lines
drivers/nvme/target/pci-ep.c line 911
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+dev_err (ctrl->dev,"PRPs list internal error\n" ); +kfree (prps); + iod->status =NVME_SC_INTERNAL |NVME_STATUS_DNR ; +return -EINVAL ;Can't you organize the labels in such a way that there is only one return path? Current code makes it difficult to read and also would confuse the static checkers.
Quoted 2 lines
I do not see how this is difficult to read... But sure, I can try to simplify this.
Maybe my mind is too much used to single return in error path ;)
Quoted 32 lines
Quoted 23 lines
[...]
Quoted 20 lines
drivers/nvme/target/pci-ep.c new file +20
-0+static void nvmet_pciep_init_bar (struct nvmet_pciep_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + + ctrl->bar = ctrl->nvme_epf->reg_bar; + +/* Copy the target controller capabilities as a base */ + ctrl->cap = tctrl->cap; + +/* Contiguous Queues Required (CQR) */ + ctrl->cap |=0x1ULL <<16 ; + +/* Set Doorbell stride to 4B (DSTRB) */ + ctrl->cap &= ~GENMASK (35 ,32 ); + +/* Clear NVM Subsystem Reset Supported (NSSRS) */ + ctrl->cap &= ~(0x1ULL <<36 ); + +/* Clear Boot Partition Support (BPS) */ + ctrl->cap &= ~(0x1ULL <<45 );Quoted 8 lines
drivers/nvme/target/pci-ep.c line 1960
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+ +/* Clear Persistent Memory Region Supported (PMRS) */ + ctrl->cap &= ~(0x1ULL <<56 ); + +/* Clear Controller Memory Buffer Supported (CMBS) */ + ctrl->cap &= ~(0x1ULL <<57 );Can you use macros for these?
Quoted 4 lines
I could. But that would mean defining *all* bits of the cap register, which is a lot... The NVMe code does not have all these defined now. That is a cleanup that can come later I think.
No not needed to define all the caps, but just the ones you are using.
Quoted 17 lines
[...]
Quoted 6 lines
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+if (epc_features->msix_capable) { +size_t pba_size; + + msix_table_size =PCI_MSIX_ENTRY_SIZE * epf->msix_interrupts; + nvme_epf->msix_table_offset = reg_size; + pba_size =ALIGN (DIV_ROUND_UP (epf->msix_interrupts,8 ),8 );Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2135
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+ + reg_size += msix_table_size + pba_size; + } + + reg_bar_size =ALIGN (reg_size,max (epc_features->align,4096 ));From where does this 4k alignment comes from? NVMe spec? If so, is it OK to use
fixed_sizeBAR?
Quoted 4 lines
NVMe BAR cannot be fixed size as its size depends on the number of queue pairs the controller can support. Will check if the 4K alignment is mandated by the specs. But it sure does not hurt...
My question was more about if the 4K alignment is enforced by the spec, then you are ignoring alignment for fixed size BAR by using its fixed size:
reg_bar_size = epc_features->bar[BAR_0].fixed_size;
Quoted 10 lines
[...]
Quoted 7 lines
drivers/nvme/target/pci-ep.c line 2246
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+/* Create the target controller. */ + ret =nvmet_pciep_create_ctrl (nvme_epf, max_nr_queues); +if (ret) { +dev_err (&epf->dev, +"Create NVMe PCI target controller failed\n" );Failed to create NVMe PCI target controller
Quoted 2 lines
How is that better ?
It is common for the error messages to start with 'Failed to...'. Also 'Create NVMe PCI target controller failed' doesn't sound correct to me. But I am not a native english speaker, so my views could be wrong.
Quoted 9 lines
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2250
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +4
-0+return ret; + } + +if (epf->vfunc_no <=1 ) {Are you really supporting virtual functions? If supported, '
vfunc_no< 1' is not possible.
Quoted 5 lines
NVMe does support virtual functions and nothing in the configuration path prevents the user from setting one such function. So yes, this is supposed to work if the endpoint controller supports it. Though I must say that I have not tried/tested yet.
If virtual functions are supported, then 'epf->vfunc_no < 1' is invalid:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/endpoint/pci-epf-core.c#n79
If virtual functions are not tested, then I'd suggest dropping the check for it.
Quoted 9 lines
Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2254
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +4
-0+/* Set device ID, class, etc */ + epf->header->vendorid = ctrl->tctrl->subsys->vendor_id; + epf->header->subsys_vendor_id = + ctrl->tctrl->subsys->subsys_vendor_id;Why these are coming from somewhere else and not configured within the EPF driver?
Quoted 4 lines
They are set through the nvme target configfs. So there is no need to have these again setup through the epf configfs. We just grab the values set for the NVME target subsystem config.
But in documentation you were configuring the vendor_id twice:
# echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id
...
# echo 0x1b96 > nvmepf.0/vendoridAnd that's what confused me. You need to get rid of the second command and add a note that the vendor_id used in target configfs will be reused.
Quoted 10 lines
Quoted 9 lines
drivers/nvme/target/pci-ep.c line 2313
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+static int nvmet_pciep_epf_link_up (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + +dev_info (nvme_epf->ctrl.dev,"PCI link up\n" );These prints are supposed to come from the controller drivers. So no need to have them here also.
Quoted 3 lines
Nope, the controller driver does not print anything. At least the DWC driver does not print anything.
Which DWC driver? pcie-dw-rockchip? But other drivers like pcie-qcom-ep have these prints already. And this EPF driver is not tied to a single controller driver. As said earlier, these prints are supposed to be added to the controller drivers.
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
drivers/nvme/target/pci-ep.c new file +14
-0+static ssize_t nvmet_pciep_epf_dma_enable_store (struct config_item *item, +const char *page,size_t len) +{ +struct config_group *group =to_config_group (item); +struct nvmet_pciep_epf *nvme_epf =to_nvme_epf (group); +int ret; + +if (nvme_epf->ctrl.tctrl) +return -EBUSY ; + + ret =kstrtobool (page, &nvme_epf->dma_enable); +if (ret) +return ret; +Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2453
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+return len; +} + +CONFIGFS_ATTR (nvmet_pciep_epf_, dma_enable);What is the use of making this option user configurable? It is purely a hardware capability and I don't think users would want to have their NVMe device working without DMA voluntarily.
Quoted 3 lines
If you feel strongly about it, I can drop this. But it is useful for debugging purpose to check that the DMA API is being used and working correctly.
But if you enable DMA by default, that would test the DMA APIs all the time. I don't see a necessity to allow users to disable DMA.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+static int __init nvmet_pciep_init_module (void ) +{ +int ret; + + ret =pci_epf_register_driver (&nvmet_pciep_epf_driver);Quoted 8 lines
drivers/nvme/target/pci-ep.c line 2606
Quoted 4 lines
drivers/nvme/target/pci-ep.c new file +4
-0+if (ret) +return ret; + + ret =nvmet_register_transport (&nvmet_pciep_fabrics_ops);What is the need to register the transport so early? You should consider moving the registration to
bind()so that the transport can be removed once the driver is unbind with the controller.
Quoted 6 lines
This registration is needed to enable the configfs configuration of the nvme target subsystem and port. Withoutn this, setting a port transporet type to "pci" would fail to find the pci transport implemented here and the configuration would fail. This configuration of the nvme target susbsystem and port must come before configuring the EPF and binding it.
Ok, fair enough.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 8 lines · Manivannan Sadhasivam
Quoted 8 lines
Quoted 7 lines
drivers/nvme/target/pci-ep.c line 2246
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+/* Create the target controller. */ + ret =nvmet_pciep_create_ctrl (nvme_epf, max_nr_queues); +if (ret) { +dev_err (&epf->dev, +"Create NVMe PCI target controller failed\n" );Failed to create NVMe PCI target controller
Quoted 6 lines
Quoted 2 lines
How is that better ?
It is common for the error messages to start with 'Failed to...'. Also 'Create NVMe PCI target controller failed' doesn't sound correct to me. But I am not a native english speaker, so my views could be wrong.
I do not think this is true for all subsystems. But sure, I can change the message.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Why these are coming from somewhere else and not configured within the EPF driver?
They are set through the nvme target configfs. So there is no need to have these again setup through the epf configfs. We just grab the values set for the NVME target subsystem config.
Quoted 8 lines
But in documentation you were configuring the
vendor_idtwice:# echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id ... # echo 0x1b96 > nvmepf.0/vendoridAnd that's what confused me. You need to get rid of the second command and add a note that the
vendor_idused in target configfs will be reused.
vendor_id != subsys_vendor_id :) These are 2 different fields. subsys_vendor_id is reported by the identify controller command and is also present in the PCI config space. vendor_id is not reported by the identify controller command and present only in the PCI config space.
For the config example, I simply used the same values for both fields, but they can be different. NVMe PCIe specs are a bit of a mess around these IDs...
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
drivers/nvme/target/pci-ep.c line 2313
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+static int nvmet_pciep_epf_link_up (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + +dev_info (nvme_epf->ctrl.dev,"PCI link up\n" );These prints are supposed to come from the controller drivers. So no need to have them here also.
Quoted 8 lines
Quoted 3 lines
Nope, the controller driver does not print anything. At least the DWC driver does not print anything.
Which DWC driver? pcie-dw-rockchip? But other drivers like pcie-qcom-ep have these prints already. And this EPF driver is not tied to a single controller driver. As said earlier, these prints are supposed to be added to the controller drivers.
The DWC driver for the rk2588 (drivers/pci/controllers/dwc/*) is missing this message.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
Quoted 32 lines · Damien Le Moal
Quoted 8 lines · Manivannan Sadhasivam
Quoted 8 lines
Quoted 7 lines
drivers/nvme/target/pci-ep.c line 2246
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+/* Create the target controller. */ + ret =nvmet_pciep_create_ctrl (nvme_epf, max_nr_queues); +if (ret) { +dev_err (&epf->dev, +"Create NVMe PCI target controller failed\n" );Failed to create NVMe PCI target controller
Quoted 6 lines
Quoted 2 lines
How is that better ?
It is common for the error messages to start with 'Failed to...'. Also 'Create NVMe PCI target controller failed' doesn't sound correct to me. But I am not a native english speaker, so my views could be wrong.
I do not think this is true for all subsystems. But sure, I can change the message.
Quoted 14 lines
Quoted 7 lines
Quoted 2 lines
Why these are coming from somewhere else and not configured within the EPF driver?
They are set through the nvme target configfs. So there is no need to have these again setup through the epf configfs. We just grab the values set for the NVME target subsystem config.
But in documentation you were configuring the
vendor_idtwice:# echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id ... # echo 0x1b96 > nvmepf.0/vendorid
Quoted 8 lines
Quoted 2 lines
And that's what confused me. You need to get rid of the second command and add a note that the
vendor_idused in target configfs will be reused.
vendor_id!=subsys_vendor_id:) These are 2 different fields.subsys_vendor_idis reported by the identify controller command and is also present in the PCI config space.vendor_idis not reported by the identify controller command and present only in the PCI config space.
I know the difference between vendor_id and subsys_vendor_id :) But as I quoted, you are using the same vendor id value in 2 places. One in nvmet configfs and another in epf configfs. But internally, you just reuse the nvmet configfs value in epf. And this is not evident in the documentation.
Quoted 17 lines
For the config example, I simply used the same values for both fields, but they can be different. NVMe PCIe specs are a bit of a mess around these IDs...
Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
drivers/nvme/target/pci-ep.c line 2313
Quoted 6 lines
drivers/nvme/target/pci-ep.c new file +6
-0+static int nvmet_pciep_epf_link_up (struct pci_epf *epf) +{ +struct nvmet_pciep_epf *nvme_epf =epf_get_drvdata (epf); +struct nvmet_pciep_ctrl *ctrl = &nvme_epf->ctrl; + +dev_info (nvme_epf->ctrl.dev,"PCI link up\n" );These prints are supposed to come from the controller drivers. So no need to have them here also.
Nope, the controller driver does not print anything. At least the DWC driver does not print anything.
Quoted 8 lines
Quoted 4 lines
Which DWC driver? pcie-dw-rockchip? But other drivers like pcie-qcom-ep have these prints already. And this EPF driver is not tied to a single controller driver. As said earlier, these prints are supposed to be added to the controller drivers.
The DWC driver for the rk2588 (
drivers/pci/controllers/dwc/*) is missing this message.
Yeah, maybe you should add it later.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Christoph Hellwig
maintainerRe: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Manivannan SadhasivamQuoted 2 lines · Manivannan Sadhasivam
'iomem' is just a token that is used by sparse tool to verify correctness of MMIO address handling.
That is true, but we don't have the token just for fun, but because iomem needs to be treated differently.
Quoted 5 lines
I know that the memory here is returned by
dma_alloc_coherent()and is not exactly a MMIO. But since you are usingREAD_ONCE()andle32_to_cpu()on these addresses, I thought about suggesting readl/writel that does exactly the same internally. If you do not need ordering, then you can use relaxed variants as well.
No, using readl/writel on dma_alloc_coherent buffers is simply wrong.
Christoph Hellwig
maintainerRe: [PATCH v4 17/18] nvmet: New NVMe PCI endpoint target driver
In reply to Damien Le MoalQuoted 11 lines · Damien Le Moal
Quoted 7 lines
drivers/nvme/target/pci-ep.c line 231
Quoted 5 lines
drivers/nvme/target/pci-ep.c new file +5
-0+static inline u32 nvmet_pciep_bar_read32 (struct nvmet_pciep_ctrl *ctrl,u32 off) +{ +__le32 *bar_reg = ctrl->bar + off; + +return le32_to_cpu (READ_ONCE (*bar_reg));Looks like you can use readl/writel variants here. Any reason to not use them?
A bar memory comes from
dma_alloc_coherent(), which is a "void *" pointer and *not* iomem. So using readl/writel for accessing that memory seems awefully wrong to me.
Using readl/writel not only sounds wrong, but fundamentally is wrong. readl/writel should only be used on pointers returned from ioremap and friends. On some architectures it might use entirely different instructions from normal loads and stores, and on others it needs to be a lot more careful.
Quoted 3 lines
NVMe BAR cannot be fixed size as its size depends on the number of queue pairs the controller can support. Will check if the 4K alignment is mandated by the specs. But it sure does not hurt...
Keeping it 4k alignment will make everyones life simpler.
PATCH v4 18/18 Documentation: Document the NVMe PCI endpoint target driver
In reply to Damien Le MoalAdd a documentation file (Documentation/nvme/nvme-pci-endpoint-target.rst) for the new NVMe PCI endpoint target driver. This provides an overview of the driver requirements, capabilities and limitations. A user guide describing how to setup a NVMe PCI endpoint device using this driver is also provided.
This document is made accessible also from the PCI endpoint documentation using a link. Furthermore, since the existing nvme documentation was not accessible from the top documentation index, an index file is added to Documentation/nvme and this index listed as "NVMe Subsystem" in the "Storage interfaces" section of the subsystem API index.
--- Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 14 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 365 ++++++++++++++++++ Documentation/subsystem-apis.rst | 1 + 5 files changed, 393 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst
Documentation/PCI/endpoint/index.rst +1 -0
@@ -15,6 +15,7 @@ PCI Endpoint Framework pci-ntb-howto pci-vntb-function pci-vntb-howto + pci-nvme-function function/binding/pci-test function/binding/pci-ntb
Documentation/PCI/endpoint/pci-nvme-function.rst new file +14 -0
@@ -0,0 +1,14 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NVMe Function +================= + +:Author: Damien Le Moal <[email protected]> + +The PCI NVMe endpoint function implements a PCI NVMe controller using the NVMe +subsystem target core code. The driver for this function resides with the NVMe +subsystem as drivers/nvme/target/nvmet-pciep.c. + +See Documentation/nvme/nvme-pci-endpoint-target.rst for more details. +
Documentation/nvme/index.rst new file +12 -0
@@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +NVMe Subsystem +============== + +.. toctree:: + :maxdepth: 2 + :numbered: + + feature-and-quirk-policy + nvme-pci-endpoint-target
Documentation/nvme/nvme-pci-endpoint-target.rst new file +365 -0
365 lines not on this page · raw message
Documentation/subsystem-apis.rst +1 -0
@@ -60,6 +60,7 @@ Storage interfaces cdrom/index scsi/index target/index + nvme/index Other subsystems ----------------
Signature · 1 line
2.47.1
Re: [PATCH v4 18/18] Documentation: Document the NVMe PCI endpoint target driver
Quoted 6 lines · Damien Le Moal
Add a documentation file (
Documentation/nvme/nvme-pci-endpoint-target.rst) for the new NVMe PCI endpoint target driver. This provides an overview of the driver requirements, capabilities and limitations. A user guide describing how to setup a NVMe PCI endpoint device using this driver is also provided.
Quoted 6 lines
This document is made accessible also from the PCI endpoint documentation using a link. Furthermore, since the existing nvme documentation was not accessible from the top documentation index, an index file is added to
Documentation/nvmeand this index listed as "NVMe Subsystem" in the "Storage interfaces" section of the subsystem API index.
Applying: Documentation: Document the NVMe PCI endpoint target driver .git/rebase-apply/patch:43: new blank line at EOF.
+
warning: 1 line adds whitespace errors.
Quoted 5 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +5
-0+The NVMe PCI endpoint target driver allows exposing a NVMe target controller +over a PCIe link, thus implementing an NVMe PCIe device similar to a regular +M.2 SSD. The target controller is created in the same manner as when using NVMe +over fabrics: the controller represents the interface to an NVMe subsystem +using a port. The port transfer type must be configured to be "pci". The
Documentation/nvme/nvme-pci-endpoint-target.rst line 29
Quoted 10 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +10
-0+subsystem can be configured to have namespaces backed by regular files or block +devices, or can use NVMe passthrough to expose an existing physical NVMe device +or a NVMe fabrics host controller (e.g. a NVMe TCP host controller). + +The NVMe PCI endpoint target driver relies as much as possible on the NVMe +target core code to parse and execute NVMe commands submitted by the PCI RC +host. However, using the PCI endpoint framework API and DMA API, the driver is +also responsible for managing all data transfers over the PCI link. This +implies that the NVMe PCI endpoint target driver implements several NVMe data +structure management and some command parsing.
Sort of a mix of "PCIe link" vs "PCI link", maybe make them consistent.
Does "PCI RC" mean "Root Complex"? If so, maybe "PCIe Root Complex" the first time, and "PCIe RC" subsequently? I don't know enough about this to know whether "Root Complex" is necessary in this context, or whether "host" might be enough.
Quoted 2 lines
+4) The boot partition support (BPS), Persistent Memory Region Supported (PMRS)
+ and Controller Memory Buffer Supported (CMBS) capabilities are never reported.
Gratuitous >80 column line.
Documentation/nvme/nvme-pci-endpoint-target.rst line 262
Quoted 2 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +2
-0+If the PCI endpoint controller used does not support MSIX, MSI can be +configured instead::
s/MSIX/MSI-X/ as is used elsewhere
Quoted 1 line
+The NVMe PCI endpoint target driver uses the PCI endpoint configfs device attributes as follows.
Gratuitous >80 column line.
Re: [PATCH v4 18/18] Documentation: Document the NVMe PCI endpoint target driver
In reply to Damien Le MoalQuoted 295 lines · Damien Le Moal
Add a documentation file (
Documentation/nvme/nvme-pci-endpoint-target.rst) for the new NVMe PCI endpoint target driver. This provides an overview of the driver requirements, capabilities and limitations. A user guide describing how to setup a NVMe PCI endpoint device using this driver is also provided.This document is made accessible also from the PCI endpoint documentation using a link. Furthermore, since the existing nvme documentation was not accessible from the top documentation index, an index file is added to
Documentation/nvmeand this index listed as "NVMe Subsystem" in the "Storage interfaces" section of the subsystem API index.Signed-off-by Damien Le Moal <[email protected]>Reviewed-by Christoph Hellwig <[email protected]>--- Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 14 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 365 ++++++++++++++++++ Documentation/subsystem-apis.rst | 1 + 5 files changed, 393 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rstDocumentation/PCI/endpoint/index.rst +1
-0@@ -15,6 +15,7 @@ PCI Endpoint Framework pci-ntb-howto pci-vntb-function pci-vntb-howto + pci-nvme-function function/binding/pci-test function/binding/pci-ntbDocumentation/PCI/endpoint/pci-nvme-function.rst new file +14
-0@@ -0,0 +1,14 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NVMe Function +================= + +:Author: Damien Le Moal <[email protected]> + +The PCI NVMe endpoint function implements a PCI NVMe controller using the NVMe +subsystem target core code. The driver for this function resides with the NVMe +subsystem as drivers/nvme/target/nvmet-pciep.c. + +See Documentation/nvme/nvme-pci-endpoint-target.rst for more details. +Documentation/nvme/index.rst new file +12
-0@@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +NVMe Subsystem +============== + +.. toctree:: + :maxdepth: 2 + :numbered: + + feature-and-quirk-policy + nvme-pci-endpoint-targetDocumentation/nvme/nvme-pci-endpoint-target.rst new file +213
-0@@ -0,0 +1,365 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================== +NVMe PCI Endpoint Target +======================== + +:Author: Damien Le Moal <[email protected]> + +The NVMe PCI endpoint target driver implements a PCIe NVMe controller using a +NVMe fabrics target controller using the PCI transport type. + +Overview +======== + +The NVMe PCI endpoint target driver allows exposing a NVMe target controller +over a PCIe link, thus implementing an NVMe PCIe device similar to a regular +M.2 SSD. The target controller is created in the same manner as when using NVMe +over fabrics: the controller represents the interface to an NVMe subsystem +using a port. The port transfer type must be configured to be "pci". The +subsystem can be configured to have namespaces backed by regular files or block +devices, or can use NVMe passthrough to expose an existing physical NVMe device +or a NVMe fabrics host controller (e.g. a NVMe TCP host controller). + +The NVMe PCI endpoint target driver relies as much as possible on the NVMe +target core code to parse and execute NVMe commands submitted by the PCI RC +host. However, using the PCI endpoint framework API and DMA API, the driver is +also responsible for managing all data transfers over the PCI link. This +implies that the NVMe PCI endpoint target driver implements several NVMe data +structure management and some command parsing. + +1) The driver manages retrieval of NVMe commands in submission queues using DMA + if supported, or MMIO otherwise. Each command retrieved is then executed + using a work item to maximize performance with the parallel execution of + multiple commands on different CPUs. The driver uses a work item to + constantly poll the doorbell of all submission queues to detect command + submissions from the PCI RC host. + +2) The driver transfers completion queues entries of completed commands to the + PCI RC host using MMIO copy of the entries in the host completion queue. + After posting completion entries in a completion queue, the driver uses the + PCI endpoint framework API to raise an interrupt to the host to signal the + commands completion. + +3) For any command that has a data buffer, the NVMe PCI endpoint target driver + parses the command PRPs or SGLs lists to create a list of PCI address + segments representing the mapping of the command data buffer on the host. + The command data buffer is transferred over the PCI link using this list of + PCI address segments using DMA, if supported. If DMA is not supported, MMIO + is used, which results in poor performance. For write commands, the command + data buffer is transferred from the host into a local memory buffer before + executing the command using the target core code. For read commands, a local + memory buffer is allocated to execute the command and the content of that + buffer is transferred to the host once the command completes. + +Controller Capabilities +----------------------- + +The NVMe capabilities exposed to the PCI RC host through the BAR 0 registers +are almost identical to the capabilities of the NVMe target controller +implemented by the target core code. There are some exceptions. + +1) The NVMe PCI endpoint target driver always sets the controller capability + CQR bit to request "Contiguous Queues Required". This is to facilitate the + mapping of a queue PCI address range to the local CPU address space. + +2) The doorbell stride (DSTRB) is always set to be 4B + +3) Since the PCI endpoint framework does not provide a way to handle PCI level + resets, the controller capability NSSR bit (NVM Subsystem Reset Supported) + is always cleared. + +4) The boot partition support (BPS), Persistent Memory Region Supported (PMRS) + and Controller Memory Buffer Supported (CMBS) capabilities are never reported. + +Supported Features +------------------ + +The NVMe PCI endpoint target driver implements support for both PRPs and SGLs. +The driver also implements IRQ vector coalescing and submission queue +arbitration burst. + +The maximum number of queues and the maximum data transfer size (MDTS) are +configurable through configfs before starting the controller. To avoid issues +with excessive local memory usage for executing commands, MDTS defaults to 512 +KB and is limited to a maximum of 2 MB (arbitrary limit). + +Mimimum number of PCI Address Mapping Windows Required +------------------------------------------------------ + +Most PCI endpoint controllers provide a limited number of mapping windows for +mapping a PCI address range to local CPU memory addresses. The NVMe PCI +endpoint target controllers uses mapping windows for the following. + +1) One memory window for raising MSI or MSI-X interrupts +2) One memory window for MMIO transfers +3) One memory window for each completion queue + +Given the highly asynchronous nature of the NVMe PCI endpoint target driver +operation, the memory windows as described above will generally not be used +simultaneously, but that may happen. So a safe maximum number of completion +queues that can be supported is equal to the total number of memory mapping +windows of the PCI endpoint controller minus two. E.g. for an endpoint PCI +controller with 32 outbound memory windows available, up to 30 completion +queues can be safely operated without any risk of getting PCI address mapping +errors due to the lack of memory windows. + +Maximum Number of Queue Pairs +----------------------------- + +Upon binding of the NVMe PCI endpoint target driver to the PCI endpoint +controller, BAR 0 is allocated with enough space to accommodate the admin queue +and multiple I/O queues. The maximum of number of I/O queues pairs that can be +supported is limited by several factors. + +1) The NVMe target core code limits the maximum number of I/O queues to the + number of online CPUs. +2) The total number of queue pairs, including the admin queue, cannot exceed + the number of MSI-X or MSI vectors available. +3) The total number of completion queues must not exceed the total number of + PCI mapping windows minus 2 (see above). + +The NVMe endpoint function driver allows configuring the maximum number of +queue pairs through configfs. + +Limitations and NVMe Specification Non-Compliance +------------------------------------------------- + +Similar to the NVMe target core code, the NVMe PCI endpoint target driver does +not support multiple submission queues using the same completion queue. All +submission queues must specify a unique completion queue. + + +User Guide +========== + +This section describes the hardware requirements and how to setup an NVMe PCI +endpoint target device. + +Kernel Requirements +------------------- + +The kernel must be compiled with the configuration options CONFIG_PCI_ENDPOINT, +CONFIG_PCI_ENDPOINT_CONFIGFS, and CONFIG_NVME_TARGET_PCI_EP enabled. +CONFIG_PCI, CONFIG_BLK_DEV_NVME and CONFIG_NVME_TARGET must also be enabled +(obviously). + +In addition to this, at least one PCI endpoint controller driver should be +available for the endpoint hardware used. + +To facilitate testing, enabling the null-blk driver (CONFIG_BLK_DEV_NULL_BLK) +is also recommended. With this, a simple setup using a null_blk block device +as a subsystem namespace can be used. + +Hardware Requirements +--------------------- + +To use the NVMe PCI endpoint target driver, at least one endpoint controller +device is required. + +To find the list of endpoint controller devices in the system:: + + # ls /sys/class/pci_epc/ + a40000000.pcie-ep + +If PCI_ENDPOINT_CONFIGFS is enabled:: + + # ls /sys/kernel/config/pci_ep/controllers + a40000000.pcie-ep + +The endpoint board must of course also be connected to a host with a PCI cable +with RX-TX signal swapped. If the host PCI slot used does not have +plug-and-play capabilities, the host should be powered off when the NVMe PCI +endpoint device is configured. + +NVMe Endpoint Device +-------------------- + +Creating an NVMe endpoint device is a two step process. First, an NVMe target +subsystem and port must be defined. Second, the NVMe PCI endpoint device must be +setup and bound to the subsystem and port created. + +Creating a NVMe Subsystem and Port +---------------------------------- + +Details about how to configure a NVMe target subsystem and port are outside the +scope of this document. The following only provides a simple example of a port +and subsystem with a single namespace backed by a null_blk device. + +First, make sure that configfs is enabled:: + + # mount -t configfs none /sys/kernel/config + +Next, create a null_blk device (default settings give a 250 GB device without +memory backing). The block device created will be /dev/nullb0 by default:: + + # modprobe null_blk + # ls /dev/nullb0 + /dev/nullb0 + +The NVMe target core driver must be loaded:: + + # modprobe nvmet + # lsmod | grep nvmet + nvmet 118784 0 + nvme_core 131072 1 nvmet + +Now, create a subsystem and a port that we will use to create a PCI target +controller when setting up the NVMe PCI endpoint target device. In this +example, the port is created with a maximum of 4 I/O queue pairs:: + + # cd /sys/kernel/config/nvmet/subsystems + # mkdir nvmepf.0.nqn + # echo -n "Linux-nvmet-pciep" > nvmepf.0.nqn/attr_model
Documentation/nvme/nvme-pci-endpoint-target.rst line 223
Quoted 10 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +10
-0+ # echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id + # echo "0x1b96" > nvmepf.0.nqn/attr_subsys_vendor_id + # echo 1 > nvmepf.0.nqn/attr_allow_any_host + # echo 4 > nvmepf.0.nqn/attr_qid_max + +Next, create and enable the subsystem namespace using the null_blk block device:: + + # mkdir nvmepf.0.nqn/namespaces/1 + # echo -n "/dev/nullb0" > nvmepf.0.nqn/namespaces/1/device_path + # echo 1 > "pci_epf_nvme.0.nqn/namespaces/1/enable"
I have to do, 'echo 1 > nvmepf.0.nqn/namespaces/1/enable'
Quoted 36 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +36
-0+ +Finally, create the target port and link it to the subsystem:: + + # cd /sys/kernel/config/nvmet/ports + # mkdir 1 + # echo -n "pci" > 1/addr_trtype + # ln -s /sys/kernel/config/nvmet/subsystems/nvmepf.0.nqn \ + /sys/kernel/config/nvmet/ports/1/subsystems/nvmepf.0.nqn + +Creating a NVMe PCI Endpoint Device +----------------------------------- + +With the NVMe target subsystem and port ready for use, the NVMe PCI endpoint +device can now be created and enabled. The NVMe PCI endpoint target driver +should already be loaded (that is done automatically when the port is created):: + + # ls /sys/kernel/config/pci_ep/functions + nvmet_pciep + +Next, create function 0:: + + # cd /sys/kernel/config/pci_ep/functions/nvmet_pciep + # mkdir nvmepf.0 + # ls nvmepf.0/ + baseclass_code msix_interrupts secondary + cache_line_size nvme subclass_code + deviceid primary subsys_id + interrupt_pin progif_code subsys_vendor_id + msi_interrupts revid vendorid + +Configure the function using any vendor ID and device ID:: + + # cd /sys/kernel/config/pci_ep/functions/nvmet_pciep + # echo 0x1b96 > nvmepf.0/vendorid + # echo 0xBEEF > nvmepf.0/deviceid + # echo 32 > nvmepf.0/msix_interrupts
Documentation/nvme/nvme-pci-endpoint-target.rst line 269
Quoted 10 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +10
-0+ +If the PCI endpoint controller used does not support MSIX, MSI can be +configured instead:: + + # echo 32 > nvmepf.0/msi_interrupts + +Next, let's bind our endpoint device with the target subsystem and port that we +created:: + + # echo 1 > nvmepf.0/portid
'echo 1 > nvmepf.0/nvme/portid'
Documentation/nvme/nvme-pci-endpoint-target.rst line 270
Quoted 1 line
Documentation/nvme/nvme-pci-endpoint-target.rst new file +1
-0+ # echo "nvmepf.0.nqn" > nvmepf.0/subsysnqn
'echo 1 > nvmepf.0/nvme/subsysnqn'
Documentation/nvme/nvme-pci-endpoint-target.rst line 281
Quoted 11 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +11
-0+ +The endpoint function can then be bound to the endpoint controller and the +controller started:: + + # cd /sys/kernel/config/pci_ep + # ln -s functions/nvmet_pciep/nvmepf.0 controllers/a40000000.pcie-ep/ + # echo 1 > controllers/a40000000.pcie-ep/start + +On the endpoint machine, kernel messages will show information as the NVMe +target device and endpoint device are created and connected. +
For some reason, I cannot get the function driver working. Getting this warning on the ep:
nvmet: connect request for invalid subsystem 1!
I didn't debug it further. Will do it tomorrow morning and let you know.
- Mani
Signature · 1 line
மணிவண்ணன் சதாசிவம்
Re: [PATCH v4 18/18] Documentation: Document the NVMe PCI endpoint target driver
Quoted 11 lines · Manivannan Sadhasivam
Quoted 11 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +11
-0+Now, create a subsystem and a port that we will use to create a PCI target +controller when setting up the NVMe PCI endpoint target device. In this +example, the port is created with a maximum of 4 I/O queue pairs:: + + # cd /sys/kernel/config/nvmet/subsystems + # mkdir nvmepf.0.nqn + # echo -n "Linux-nvmet-pciep" > nvmepf.0.nqn/attr_model + # echo "0x1b96" > nvmepf.0.nqn/attr_vendor_id + # echo "0x1b96" > nvmepf.0.nqn/attr_subsys_vendor_id + # echo 1 > nvmepf.0.nqn/attr_allow_any_host + # echo 4 > nvmepf.0.nqn/attr_qid_max
Quoted 8 lines
Documentation/nvme/nvme-pci-endpoint-target.rst line 223
Quoted 6 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +6
-0+ +Next, create and enable the subsystem namespace using the null_blk block device:: + + # mkdir nvmepf.0.nqn/namespaces/1 + # echo -n "/dev/nullb0" > nvmepf.0.nqn/namespaces/1/device_path + # echo 1 > "pci_epf_nvme.0.nqn/namespaces/1/enable"I have to do, 'echo 1 > nvmepf.0.nqn/namespaces/1/enable'
Good catch. That is the old name from previous version. Will fix this.
Quoted 49 lines
Quoted 36 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +36
-0+ +Finally, create the target port and link it to the subsystem:: + + # cd /sys/kernel/config/nvmet/ports + # mkdir 1 + # echo -n "pci" > 1/addr_trtype + # ln -s /sys/kernel/config/nvmet/subsystems/nvmepf.0.nqn \ + /sys/kernel/config/nvmet/ports/1/subsystems/nvmepf.0.nqn + +Creating a NVMe PCI Endpoint Device +----------------------------------- + +With the NVMe target subsystem and port ready for use, the NVMe PCI endpoint +device can now be created and enabled. The NVMe PCI endpoint target driver +should already be loaded (that is done automatically when the port is created):: + + # ls /sys/kernel/config/pci_ep/functions + nvmet_pciep + +Next, create function 0:: + + # cd /sys/kernel/config/pci_ep/functions/nvmet_pciep + # mkdir nvmepf.0 + # ls nvmepf.0/ + baseclass_code msix_interrupts secondary + cache_line_size nvme subclass_code + deviceid primary subsys_id + interrupt_pin progif_code subsys_vendor_id + msi_interrupts revid vendorid + +Configure the function using any vendor ID and device ID:: + + # cd /sys/kernel/config/pci_ep/functions/nvmet_pciep + # echo 0x1b96 > nvmepf.0/vendorid + # echo 0xBEEF > nvmepf.0/deviceid + # echo 32 > nvmepf.0/msix_interruptsDocumentation/nvme/nvme-pci-endpoint-target.rst line 269
Quoted 10 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +10
-0+ +If the PCI endpoint controller used does not support MSIX, MSI can be +configured instead:: + + # echo 32 > nvmepf.0/msi_interrupts + +Next, let's bind our endpoint device with the target subsystem and port that we +created:: + + # echo 1 > nvmepf.0/portid'echo 1 > nvmepf.0/nvme/portid'
Quoted 3 lines
Quoted 1 line
+ # echo "nvmepf.0.nqn" > nvmepf.0/subsysnqn
'echo 1 > nvmepf.0/nvme/subsysnqn'
Yep. Good catch.
Quoted 16 lines
Documentation/nvme/nvme-pci-endpoint-target.rst line 281
Quoted 12 lines
Documentation/nvme/nvme-pci-endpoint-target.rst new file +11
-0+ +The endpoint function can then be bound to the endpoint controller and the +controller started:: + + # cd /sys/kernel/config/pci_ep + # ln -s functions/nvmet_pciep/nvmepf.0 controllers/a40000000.pcie-ep/ + # echo 1 > controllers/a40000000.pcie-ep/start + +On the endpoint machine, kernel messages will show information as the NVMe +target device and endpoint device are created and connected. +For some reason, I cannot get the function driver working. Getting this warning on the ep:
Quoted 3 lines
nvmet: connect request for invalid subsystem 1!
I didn't debug it further. Will do it tomorrow morning and let you know.
Hmmm... Weird. You should not ever see a connect request/command at all. Can you try this script:
Just run "./nvmet-pciep start" after booting the endpoint board.
The command example in the documentation is an extract from what this script does. I think that:
echo 1 > ${SUBSYSNQN}/attr_allow_any_hostmissing may be the reason for this error.
Signature · 2 lines
Damien Le Moal Western Digital Research
Quoted 52 lines · Damien Le Moal
This patch series implements an NVMe target driver for the PCI transport using the PCI endpoint framework.
The first 4 patches of this series move and cleanup some nvme code that will be reused in following patches.
Patch 5 introduces the PCI transport type to allow setting up ports for the new PCI target controller driver. Patch 6 to 9 are improvements of the target core code to allow creating the PCI controller and processing its nvme commands without the need to rely on fabrics commands like the connect command to create the admin and I/O queues.
Patch 10 relaxes the SGL check in
nvmet_req_init()to allow for PCI admin commands (which must use PRPs).Patches 11 to 15 improve the set/get feature support of the target code to get closer to achieving NVMe specification compliance. These patches though do not implement support for some mandatory features.
Patch 16 is the main patch which introduces the NVMe PCI endpoint target driver. This patch commit message provides and overview of the driver design and operation.
Finally, patch 17 documents the NVMe PCI endpoint target driver and provides a user guide explaning how to setup an NVMe PCI endpoint device.
The patches are base on Linus 6.13-rc2 tree.
This driver has been extensively tested using a Radxa Rock5B board (RK3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board. However, this board does not support DMA channels for the PCI endpoint controller, leading to very poor performance.
Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:
+----------------------------------+------------------------+ | Workload | IOPS (BW) | +----------------------------------+------------------------+ | Rand read, 4KB, QD=1, 1 job | 14.3k IOPS | | Rand read, 4KB, QD=32, 1 job | 80.8k IOPS | | Rand read, 4KB, QD=32, 4 jobs | 131k IOPS | | Rand read, 128KB, QD=32, 1 job | 16.7k IOPS (2.18 GB/s) | | Rand read, 128KB, QD=32, 4 jobs | 17.4k IOPS (2.27 GB/s) | | Rand read, 512KB, QD=32, 1 job | 5380 IOPS (2.82 GB/s) | | Rand read, 512KB, QD=32, 4 jobs | 5206 IOPS (2.27 GB/s) | | Rand write, 128KB, QD=32, 1 job | 9617 IOPS (1.26 GB/s) | | Rand write, 128KB, QD=32, 4 jobs | 8405 IOPS (1.10 GB/s) | +----------------------------------+------------------------+
Quoted 6 lines
These results use the default MDTS of the NVMe enpoint driver of 512 KB.
This driver is not intended for production use but rather to be a playground for learning NVMe and exploring/testing new NVMe features while providing reasonably good performance.
Damien, thanks for the work! Please wait for my review comments before posting next version. And sorry about the delay.
- Mani
Quoted 72 lines
Changes from v3:
- Added patch 1 which was missing from v3 and caused the 0day build failure
- Corrected a few typos in the documentation (patch 18)
- Added Christoph's review tag and Rick's tested tagChanges from v2:
- Changed all preparatory patches before patch 16 to move more NVMe generic code out of the PCI endpoint target driver and into the target core.
- Changed patch 16 to use directly a target controller instead of a host controller. Many aspects of the command management and DMA transfer management have also been simplified, leading to higher performance.
- Change the documentation patch to match the above changesChanges from v1:
- Added review tag to patch 1
- Modified patch 4 to:
- Add Rick's copyright notice
- Improve admin command handling (set_featurescommand) to handle the number of queues feature (among others) to enable Windows host
- Improved SQ and CQ work items handlingDamien Le Moal (18):
nvme: Move opcode string helper functions declarations
nvmet: Addvendor_idandsubsys_vendor_idsubsystem attributes
nvmet: Exportnvmet_update_cc()andnvmet_cc_xxx()helpers
nvmet: Introducenvmet_get_cmd_effects_admin()
nvmet: Add drvdata field to structnvmet_ctrl
nvme: Add PCI transport type
nvmet: Improvenvmet_alloc_ctrl()interface and implementation
nvmet: Introducenvmet_req_transfer_len()
nvmet: Introducenvmet_sq_create()andnvmet_cq_create()
nvmet: Add support for I/O queue management admin commands
nvmet: Do not require SGL for PCI target controller commands
nvmet: Introduce get/set_feature controller operations
nvmet: Implement host identifier set feature support
nvmet: Implement interrupt coalescing feature support
nvmet: Implement interrupt config feature support
nvmet: Implement arbitration feature support
nvmet: New NVMe PCI endpoint target driver
Documentation: Document the NVMe PCI endpoint target driverDocumentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 14 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 365 +++ Documentation/subsystem-apis.rst | 1 + drivers/nvme/host/nvme.h | 39 - drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/admin-cmd.c | 388 ++- drivers/nvme/target/configfs.c | 49 + drivers/nvme/target/core.c | 266 +- drivers/nvme/target/discovery.c | 17 + drivers/nvme/target/fabrics-cmd-auth.c | 14 +- drivers/nvme/target/fabrics-cmd.c | 101 +- drivers/nvme/target/nvmet.h | 110 +- drivers/nvme/target/pci-ep.c | 2626 +++++++++++++++++ include/linux/nvme.h | 42 + 17 files changed, 3897 insertions(+), 160 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst create mode 100644 drivers/nvme/target/pci-ep.cbase-commit:
231825b2e1ff6ba799c5eaf396d3ab2354e37c6bSignature · 1 line
2.47.1
Signature · 1 line
மணிவண்ணன் சதாசிவம்