NVMe PCI endpoint target driver
Subsystems (6)
v9 cover NVMe PCI endpoint target driver
This patch series implements an NVMe target driver for the PCI transport using the PCI endpoint framework.
The first 5 patches of this series move and cleanup some nvme code that will be reused in following patches.
Patch 6 introduces the PCI transport type to allow setting up ports for the new PCI target controller driver. Patch 7 to 10 are improvements of the target core code to allow creating the PCI controller and processing its nvme commands without the need to rely on fabrics commands like the connect command to create the admin and I/O queues.
Patch 11 relaxes the SGL check in nvmet_req_init() to allow for PCI admin commands (which must use PRPs).
Patches 12 to 16 improve the set/get feature support of the target code to get closer to achieving NVMe specification compliance. These patches though do not implement support for some mandatory features.
Patch 17 is the main patch which introduces the NVMe PCI endpoint target driver. This patch commit message provides and overview of the driver design and operation.
Finally, patch 18 documents the NVMe PCI endpoint target driver and provides a user guide explaning how to setup an NVMe PCI endpoint device.
The patches are base on Linus 6.13-rc3 tree.
This driver has been extensively tested using a Radxa Rock5B board (RK3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board. However, this board does not support DMA channels for the PCI endpoint controller, leading to very poor performance.
Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows:
+----------------------------------+------------------------+ | Workload | IOPS (BW) | +----------------------------------+------------------------+ | Rand read, 4KB, QD=1, 1 job | 14.3k IOPS | | Rand read, 4KB, QD=32, 1 job | 80.8k IOPS | | Rand read, 4KB, QD=32, 4 jobs | 131k IOPS | | Rand read, 128KB, QD=32, 1 job | 16.7k IOPS (2.18 GB/s) | | Rand read, 128KB, QD=32, 4 jobs | 17.4k IOPS (2.27 GB/s) | | Rand read, 512KB, QD=32, 1 job | 5380 IOPS (2.82 GB/s) | | Rand read, 512KB, QD=32, 4 jobs | 5206 IOPS (2.27 GB/s) | | Rand write, 128KB, QD=32, 1 job | 9617 IOPS (1.26 GB/s) | | Rand write, 128KB, QD=32, 4 jobs | 8405 IOPS (1.10 GB/s) | +----------------------------------+------------------------+
These results use the default MDTS of the NVMe enpoint driver of 512 KB.
This driver is not intended for production use but rather to be a playground for learning NVMe and exploring/testing new NVMe features while providing reasonably good performance.
Changes from v8:
- Addressed compilation error signaled by the build-bot for patch 17 when the PCI endpoint target driver is compiled in but the nvme core is compiled as a module. For such case, only allow the pci endpoint target driver to be compiled as a module.
Changes from v7:
- Addressed compilation warnings signaled by the build-bot for patch 17:
- Use GENMASK_ULL instead of GENMASK for 64-bits fields
- Removed unused funtion nvmet_pci_epf_prp_addr()
- Reworded the Kconfig entry and some comments and error messages as suggested by Krzysztof (all in patch 17).
- Added Tested-by and Review tag from Mani
- Added Review tag from Krzysztof
Changes from v6:
- Fixed incorrect bar cleanup in patch 17 causing a NULL pointer dereference when PERST# is asserted
- Added Mani's review tag to path 18
Changes from v5:
- Rebased on 6.13-rc3
- Addressed most of Mani's comment on patch 17:
- Renaming of functions and data structures
- Error messages format
- Removed dma_enable configfs knob and simplified DMA initialization and cleanup.
- Simplified some error path
- Fixed up handling of icontrollers with a fixed bar 0 size Of note is that I did not define macros for the bits of the CAP register as that would be too much for now since we want to do this correctly in include/linux/nvme.h by defining all of them, and by using these definitions in the nvme host and target code. This can be done in a followup patch series once this is applied.
- Fixed the command examples in patch 18
Changes from v4:
- Fixed typos in patch 13 and 17 commit message
- Addressed Bjorn's comments (typos and text clarity) in patch 18.
- Added Bjorn's Acked-by tag to patch 18
Changes from v3:
- Added patch 1 which was missing from v3 and caused the 0day build failure
- Corrected a few typos in the documentation (patch 18)
- Added Christoph's review tag and Rick's tested tag
Changes from v2:
- Changed all preparatory patches before patch 16 to move more NVMe generic code out of the PCI endpoint target driver and into the target core.
- Changed patch 16 to use directly a target controller instead of a host controller. Many aspects of the command management and DMA transfer management have also been simplified, leading to higher performance.
- Change the documentation patch to match the above changes
Changes from v1:
- Added review tag to patch 1
- Modified patch 4 to:
- Add Rick's copyright notice
- Improve admin command handling (set_features command) to handle the number of queues feature (among others) to enable Windows host
- Improved SQ and CQ work items handling
Damien Le Moal (18):
nvme: Move opcode string helper functions declarations
nvmet: Add vendor_id and subsys_vendor_id subsystem attributes
nvmet: Export nvmet_update_cc() and nvmet_cc_xxx() helpers
nvmet: Introduce nvmet_get_cmd_effects_admin()
nvmet: Add drvdata field to struct nvmet_ctrl
nvme: Add PCI transport type
nvmet: Improve nvmet_alloc_ctrl() interface and implementation
nvmet: Introduce nvmet_req_transfer_len()
nvmet: Introduce nvmet_sq_create() and nvmet_cq_create()
nvmet: Add support for I/O queue management admin commands
nvmet: Do not require SGL for PCI target controller commands
nvmet: Introduce get/set_feature controller operations
nvmet: Implement host identifier set feature support
nvmet: Implement interrupt coalescing feature support
nvmet: Implement interrupt config feature support
nvmet: Implement arbitration feature support
nvmet: New NVMe PCI endpoint function target driver
Documentation: Document the NVMe PCI endpoint target driver
Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 13 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 368 +++ Documentation/subsystem-apis.rst | 1 + drivers/nvme/host/nvme.h | 39 - drivers/nvme/target/Kconfig | 11 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/admin-cmd.c | 388 ++- drivers/nvme/target/configfs.c | 49 + drivers/nvme/target/core.c | 266 +- drivers/nvme/target/discovery.c | 17 + drivers/nvme/target/fabrics-cmd-auth.c | 14 +- drivers/nvme/target/fabrics-cmd.c | 101 +- drivers/nvme/target/nvmet.h | 110 +- drivers/nvme/target/pci-epf.c | 2591 +++++++++++++++++ include/linux/nvme.h | 42 + 17 files changed, 3865 insertions(+), 160 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst create mode 100644 drivers/nvme/target/pci-epf.c
base-commit: fc033cf25e612e840e545f8d5ad2edd6ba613ed5
Signature · 1 line
2.47.1
PATCH v9 1/18 nvme: Move opcode string helper functions declarations
Move the declaration of all helper functions converting NVMe command opcodes and status codes into strings from drivers/nvme/host/nvme.h into include/linux/nvme.h, together with the commands definitions. This allows NVMe target drivers to call these functions without having to include a host header file.
--- drivers/nvme/host/nvme.h | 39 --------------------------------------- include/linux/nvme.h | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 39 deletions(-)
drivers/nvme/host/nvme.h +0 -39
@@ -1182,43 +1182,4 @@ static inline bool nvme_multi_css(struct nvme_ctrl *ctrl) return (ctrl->ctrl_config & NVME_CC_CSS_MASK) == NVME_CC_CSS_CSI; } -#ifdef CONFIG_NVME_VERBOSE_ERRORS -const char *nvme_get_error_status_str (u16 status); -const char *nvme_get_opcode_str (u8 opcode); -const char *nvme_get_admin_opcode_str (u8 opcode); -const char *nvme_get_fabrics_opcode_str (u8 opcode); -#else /* CONFIG_NVME_VERBOSE_ERRORS */ -static inline const char *nvme_get_error_status_str (u16 status) -{ -return "I/O Error" ; -} -static inline const char *nvme_get_opcode_str (u8 opcode) -{ -return "I/O Cmd" ; -} -static inline const char *nvme_get_admin_opcode_str (u8 opcode) -{ -return "Admin Cmd" ; -} - -static inline const char *nvme_get_fabrics_opcode_str (u8 opcode) -{ -return "Fabrics Cmd" ; -} -#endif /* CONFIG_NVME_VERBOSE_ERRORS */ - -static inline const char *nvme_opcode_str (int qid,u8 opcode) -{ -return qid ?nvme_get_opcode_str (opcode) : -nvme_get_admin_opcode_str (opcode); -} - -static inline const char *nvme_fabrics_opcode_str ( -int qid,const struct nvme_command *cmd) -{ -if (nvme_is_fabrics (cmd)) -return nvme_get_fabrics_opcode_str (cmd->fabrics.fctype); - -return nvme_opcode_str (qid, cmd->common.opcode); -}#endif /* _NVME_H */
include/linux/nvme.h +40 -0
@@ -1896,6 +1896,46 @@ static inline bool nvme_is_fabrics(const struct nvme_command *cmd)return cmd->common.opcode == nvme_fabrics_command; } +#ifdef CONFIG_NVME_VERBOSE_ERRORS +const char *nvme_get_error_status_str (u16 status); +const char *nvme_get_opcode_str (u8 opcode); +const char *nvme_get_admin_opcode_str (u8 opcode); +const char *nvme_get_fabrics_opcode_str (u8 opcode); +#else /* CONFIG_NVME_VERBOSE_ERRORS */ +static inline const char *nvme_get_error_status_str (u16 status) +{ +return "I/O Error" ; +} +static inline const char *nvme_get_opcode_str (u8 opcode) +{ +return "I/O Cmd" ; +} +static inline const char *nvme_get_admin_opcode_str (u8 opcode) +{ +return "Admin Cmd" ; +} + +static inline const char *nvme_get_fabrics_opcode_str (u8 opcode) +{ +return "Fabrics Cmd" ; +} +#endif /* CONFIG_NVME_VERBOSE_ERRORS */ + +static inline const char *nvme_opcode_str (int qid,u8 opcode) +{ +return qid ?nvme_get_opcode_str (opcode) : +nvme_get_admin_opcode_str (opcode); +} + +static inline const char *nvme_fabrics_opcode_str ( +int qid,const struct nvme_command *cmd) +{ +if (nvme_is_fabrics (cmd)) +return nvme_get_fabrics_opcode_str (cmd->fabrics.fctype); + +return nvme_opcode_str (qid, cmd->common.opcode); +} +struct nvme_error_slot {__le64 error_count;__le16 sqid;
Signature · 1 line
2.47.1
PATCH v9 2/18 nvmet: Add vendor_id and subsys_vendor_id subsystem attributes
In reply to Damien Le MoalDefine the new vendor_id and subsys_vendor_id configfs attribute for target subsystems. These attributes are respectively reported as the vid field and as the ssvid field of the identify controller data of a target controllers using the subsystem for which these attributes are set.
--- drivers/nvme/target/admin-cmd.c | 5 ++-- drivers/nvme/target/configfs.c | 45 +++++++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 2 ++ 3 files changed, 49 insertions(+), 3 deletions(-)
drivers/nvme/target/admin-cmd.c +2 -3
@@ -522,9 +522,8 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) goto out; } -/* XXX: figure out how to assign real vendors IDs. */ - id->vid =0 ; - id->ssvid =0 ; + id->vid =cpu_to_le16 (subsys->vendor_id); + id->ssvid =cpu_to_le16 (subsys->subsys_vendor_id);memcpy (id->sn, ctrl->subsys->serial,NVMET_SN_MAX_SIZE ); memcpy_and_pad(id->mn, sizeof(id->mn), subsys->model_number,
drivers/nvme/target/configfs.c +45 -0
57 lines not on this page · raw message
drivers/nvme/target/nvmet.h +2 -0
@@ -324,6 +324,8 @@ struct nvmet_subsys {struct config_group namespaces_group;struct config_group allowed_hosts_group; +u16 vendor_id; +u16 subsys_vendor_id;char *model_number;u32 ieee_oui;char *firmware_rev;
Signature · 1 line
2.47.1
PATCH v9 3/18 nvmet: Export nvmet_update_cc() and nvmet_cc_xxx() helpers
In reply to Damien Le MoalMake the function nvmet_update_cc() available to target drivers by exporting it. To also facilitate the manipulation of the cc register bits, move the inline helper functions nvmet_cc_en(), nvmet_cc_css(), nvmet_cc_mps(), nvmet_cc_ams(), nvmet_cc_shn(), nvmet_cc_iosqes(), and nvmet_cc_iocqes() from core.c to nvmet.h so that these functions can be reused in target controller drivers.
--- drivers/nvme/target/core.c | 36 +----------------------------------- drivers/nvme/target/nvmet.h | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 36 insertions(+), 35 deletions(-)
drivers/nvme/target/core.c +1 -35
@@ -1166,41 +1166,6 @@ void nvmet_req_free_sgls(struct nvmet_req *req) }EXPORT_SYMBOL_GPL (nvmet_req_free_sgls); -static inline bool nvmet_cc_en (u32 cc) -{ -return (cc >>NVME_CC_EN_SHIFT ) &0x1 ; -} - -static inline u8 nvmet_cc_css (u32 cc) -{ -return (cc >>NVME_CC_CSS_SHIFT ) &0x7 ; -} - -static inline u8 nvmet_cc_mps (u32 cc) -{ -return (cc >>NVME_CC_MPS_SHIFT ) &0xf ; -} - -static inline u8 nvmet_cc_ams (u32 cc) -{ -return (cc >>NVME_CC_AMS_SHIFT ) &0x7 ; -} - -static inline u8 nvmet_cc_shn (u32 cc) -{ -return (cc >>NVME_CC_SHN_SHIFT ) &0x3 ; -} - -static inline u8 nvmet_cc_iosqes (u32 cc) -{ -return (cc >>NVME_CC_IOSQES_SHIFT ) &0xf ; -} - -static inline u8 nvmet_cc_iocqes (u32 cc) -{ -return (cc >>NVME_CC_IOCQES_SHIFT ) &0xf ; -} -static inline bool nvmet_css_supported (u8 cc_css) {switch (cc_css <<NVME_CC_CSS_SHIFT ) {
@@ -1277,6 +1242,7 @@ void nvmet_update_cc(struct nvmet_ctrl *ctrl, u32 new) ctrl->csts &= ~NVME_CSTS_SHST_CMPLT ;mutex_unlock (&ctrl->lock); } +EXPORT_SYMBOL_GPL (nvmet_update_cc);static void nvmet_init_cap (struct nvmet_ctrl *ctrl) {
drivers/nvme/target/nvmet.h +35 -0
@@ -732,6 +732,41 @@ void nvmet_passthrough_override_cap(struct nvmet_ctrl *ctrl);u16 errno_to_nvme_status (struct nvmet_req *req,int errno);u16 nvmet_report_invalid_opcode (struct nvmet_req *req); +static inline bool nvmet_cc_en (u32 cc) +{ +return (cc >>NVME_CC_EN_SHIFT ) &0x1 ; +} + +static inline u8 nvmet_cc_css (u32 cc) +{ +return (cc >>NVME_CC_CSS_SHIFT ) &0x7 ; +} + +static inline u8 nvmet_cc_mps (u32 cc) +{ +return (cc >>NVME_CC_MPS_SHIFT ) &0xf ; +} + +static inline u8 nvmet_cc_ams (u32 cc) +{ +return (cc >>NVME_CC_AMS_SHIFT ) &0x7 ; +} + +static inline u8 nvmet_cc_shn (u32 cc) +{ +return (cc >>NVME_CC_SHN_SHIFT ) &0x3 ; +} + +static inline u8 nvmet_cc_iosqes (u32 cc) +{ +return (cc >>NVME_CC_IOSQES_SHIFT ) &0xf ; +} + +static inline u8 nvmet_cc_iocqes (u32 cc) +{ +return (cc >>NVME_CC_IOCQES_SHIFT ) &0xf ; +} +/* Convert a 32-bit number to a 16-bit 0's based number */ static inline __le16 to0based (u32 a) {
Signature · 1 line
2.47.1
PATCH v9 4/18 nvmet: Introduce nvmet_get_cmd_effects_admin()
In reply to Damien Le MoalIn order to have a logically better organized implementation of the effects log page, split out reporting the supported admin commands from nvmet_get_cmd_effects_nvm() into the new function nvmet_get_cmd_effects_admin().
--- drivers/nvme/target/admin-cmd.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/nvme/target/admin-cmd.c +6 -1
@@ -230,7 +230,7 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)nvmet_req_complete (req, status); } -static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log) +static void nvmet_get_cmd_effects_admin (struct nvme_effects_log *log) { log->acs[nvme_admin_get_log_page] = log->acs[nvme_admin_identify] =
@@ -240,7 +240,10 @@ static void nvmet_get_cmd_effects_nvm(struct nvme_effects_log *log) log->acs[nvme_admin_async_event] = log->acs[nvme_admin_keep_alive] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); +} +static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log) +{ log->iocs[nvme_cmd_read] = log->iocs[nvme_cmd_flush] = log->iocs[nvme_cmd_dsm] =
@@ -276,6 +279,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)switch (req->cmd->get_log_page.csi) {case NVME_CSI_NVM : +nvmet_get_cmd_effects_admin (log);nvmet_get_cmd_effects_nvm (log);break ;case NVME_CSI_ZNS :
@@ -283,6 +287,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req) status =NVME_SC_INVALID_IO_CMD_SET ; goto free; } +nvmet_get_cmd_effects_admin (log);nvmet_get_cmd_effects_nvm (log);nvmet_get_cmd_effects_zns (log);break ;
Signature · 1 line
2.47.1
PATCH v9 5/18 nvmet: Add drvdata field to struct nvmet_ctrl
In reply to Damien Le MoalAllow a target driver to attach private data to a target controller by adding the new field drvdata to struct nvmet_ctrl.
--- drivers/nvme/target/nvmet.h | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/target/nvmet.h +2 -0
@@ -238,6 +238,8 @@ struct nvmet_ctrl {struct nvmet_subsys *subsys;struct nvmet_sq **sqs; +void *drvdata; +bool reset_tbkas;struct mutex lock;
Signature · 1 line
2.47.1
PATCH v9 6/18 nvme: Add PCI transport type
In reply to Damien Le MoalDefine the transport type NVMF_TRTYPE_PCI for PCI endpoint targets. This transport type is defined using the value 0 which is reserved in the NVMe base specifications v2.1 (Figure 294). Given that struct nvmet_port are zeroed out on creation, to avoid having this transsport type becoming the new default, nvmet_referral_make() and nvmet_ports_make() are modified to initialize a port discovery address transport type field (disc_addr.trtype) to NVMF_TRTYPE_MAX.
Any port using this transport type is also skipped and not reported in the discovery log page (nvmet_execute_disc_get_log_page()).
The helper function nvmet_is_pci_ctrl() is also introduced to check if a target controller uses the PCI transport.
--- drivers/nvme/target/configfs.c | 4 ++++ drivers/nvme/target/discovery.c | 3 +++ drivers/nvme/target/nvmet.h | 5 +++++ include/linux/nvme.h | 1 + 4 files changed, 13 insertions(+)
drivers/nvme/target/configfs.c +4 -0
@@ -37,6 +37,7 @@ static struct nvmet_type_name_map nvmet_transport[] = { { NVMF_TRTYPE_RDMA, "rdma" }, { NVMF_TRTYPE_FC, "fc" }, { NVMF_TRTYPE_TCP, "tcp" }, + { NVMF_TRTYPE_PCI, "pci" }, { NVMF_TRTYPE_LOOP, "loop" }, };
@@ -46,6 +47,7 @@ static const struct nvmet_type_name_map nvmet_addr_family[] = { { NVMF_ADDR_FAMILY_IP6, "ipv6" }, { NVMF_ADDR_FAMILY_IB, "ib" }, { NVMF_ADDR_FAMILY_FC, "fc" }, + { NVMF_ADDR_FAMILY_PCI, "pci" }, { NVMF_ADDR_FAMILY_LOOP, "loop" }, };
@@ -1839,6 +1841,7 @@ static struct config_group *nvmet_referral_make(return ERR_PTR (-ENOMEM );INIT_LIST_HEAD (&port->entry); + port->disc_addr.trtype =NVMF_TRTYPE_MAX ;config_group_init_type_name (&port->group, name, &nvmet_referral_type);return &port->group;
@@ -2064,6 +2067,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group, port->inline_data_size =-1 ;/* < 0 == let the transport choose */ port->max_queue_size =-1 ;/* < 0 == let the transport choose */ + port->disc_addr.trtype =NVMF_TRTYPE_MAX ; port->disc_addr.portid =cpu_to_le16 (portid); port->disc_addr.adrfam =NVMF_ADDR_FAMILY_MAX ; port->disc_addr.treq =NVMF_TREQ_DISABLE_SQFLOW ;
drivers/nvme/target/discovery.c +3 -0
@@ -224,6 +224,9 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) }list_for_each_entry (r, &req->port->referrals, entry) { +if (r->disc_addr.trtype ==NVMF_TRTYPE_PCI ) +continue ; + nvmet_format_discovery_entry(hdr, r, NVME_DISC_SUBSYS_NAME, r->disc_addr.traddr,
drivers/nvme/target/nvmet.h +5 -0
@@ -693,6 +693,11 @@ static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)return subsys->type !=NVME_NQN_NVME ; } +static inline bool nvmet_is_pci_ctrl (struct nvmet_ctrl *ctrl) +{ +return ctrl->port->disc_addr.trtype ==NVMF_TRTYPE_PCI ; +} +#ifdef CONFIG_NVME_TARGET_PASSTHRU void nvmet_passthru_subsys_free (struct nvmet_subsys *subsys);int nvmet_passthru_ctrl_enable (struct nvmet_subsys *subsys);
include/linux/nvme.h +1 -0
@@ -64,6 +64,7 @@ enum {/* Transport Type codes for Discovery Log Page entry TRTYPE field */ enum { +NVMF_TRTYPE_PCI =0 ,/* PCI */ NVMF_TRTYPE_RDMA =1 ,/* RDMA */ NVMF_TRTYPE_FC =2 ,/* Fibre Channel */ NVMF_TRTYPE_TCP =3 ,/* TCP/IP */
Signature · 1 line
2.47.1
PATCH v9 7/18 nvmet: Improve nvmet_alloc_ctrl() interface and implementation
In reply to Damien Le MoalIntroduce struct nvmet_alloc_ctrl_args to define the arguments for the function nvmet_alloc_ctrl() to avoid the need for passing a pointer to a struct nvmet_req as an argument. This new data structure aggregates together the arguments that were passed to nvmet_alloc_ctrl() (subsysnqn, hostnqn and kato), together with the struct nvmet_req fields used by nvmet_alloc_ctrl(), that is, the fields port, p2p_client, and ops as input and the result and error_loc fields as output, as well as a status field. nvmet_alloc_ctrl() is also changed to return a pointer to the allocated and initialized controller structure instead of a status code, as the status is now returned through the status field of struct nvmet_alloc_ctrl_args.
The function nvmet_setup_p2p_ns_map() is changed to not take a pointer to a struct nvmet_req as argument, instead, directly specify the p2p_client device pointer needed as argument.
The code in nvmet_execute_admin_connect() that initializes a new target controller after allocating it is moved into nvmet_alloc_ctrl(). The code that sets up an admin queue for the controller (and the call to nvmet_install_queue()) remains in nvmet_execute_admin_connect().
Finally, nvmet_alloc_ctrl() is also exported to allow target drivers to use this function directly to allocate and initialize a new controller structure without the need to rely on a fabrics connect command request.
--- drivers/nvme/target/core.c | 83 ++++++++++++++++++++----------- drivers/nvme/target/fabrics-cmd.c | 58 ++++++++++----------- drivers/nvme/target/nvmet.h | 18 +++++-- 3 files changed, 94 insertions(+), 65 deletions(-)
drivers/nvme/target/core.c +53 -30
@@ -1350,15 +1350,15 @@ bool nvmet_host_allowed(struct nvmet_subsys *subsys, const char *hostnqn)* Note: ctrl->subsys->lock should be held when calling this function */ static void nvmet_setup_p2p_ns_map (struct nvmet_ctrl *ctrl, -struct nvmet_req *req) +struct device *p2p_client) {struct nvmet_ns *ns;unsigned long idx; -if (!req->p2p_client) +if (!p2p_client)return ; - ctrl->p2p_client =get_device (req->p2p_client); + ctrl->p2p_client =get_device (p2p_client);xa_for_each (&ctrl->subsys->namespaces, idx, ns)nvmet_p2pmem_ns_add_p2p (ctrl, ns);
@@ -1387,45 +1387,44 @@ static void nvmet_fatal_error_handler(struct work_struct *work) ctrl->ops->delete_ctrl (ctrl); } -u16 nvmet_alloc_ctrl (const char *subsysnqn,const char *hostnqn, -struct nvmet_req *req,u32 kato,struct nvmet_ctrl **ctrlp, -uuid_t *hostid) +struct nvmet_ctrl *nvmet_alloc_ctrl (struct nvmet_alloc_ctrl_args *args) {struct nvmet_subsys *subsys;struct nvmet_ctrl *ctrl; +u32 kato = args->kato; +u8 dhchap_status;int ret; -u16 status; - status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; - subsys =nvmet_find_get_subsys (req->port, subsysnqn); + args->status =NVME_SC_CONNECT_INVALID_PARAM |NVME_STATUS_DNR ; + subsys =nvmet_find_get_subsys (args->port, args->subsysnqn);if (!subsys) {pr_warn ("connect request for invalid subsystem %s!\n" , - subsysnqn); - req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (subsysnqn); - req->error_loc = offsetof(struct nvme_common_command , dptr); - goto out; + args->subsysnqn); + args->result =IPO_IATTR_CONNECT_DATA (subsysnqn); + args->error_loc = offsetof(struct nvme_common_command , dptr); +return NULL ; }down_read (&nvmet_config_sem); -if (!nvmet_host_allowed (subsys, hostnqn)) { +if (!nvmet_host_allowed (subsys, args->hostnqn)) {pr_info ("connect by host %s for subsystem %s not allowed\n" , - hostnqn, subsysnqn); - req->cqe->result.u32 =IPO_IATTR_CONNECT_DATA (hostnqn); + args->hostnqn, args->subsysnqn); + args->result =IPO_IATTR_CONNECT_DATA (hostnqn);up_read (&nvmet_config_sem); - status =NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; - req->error_loc = offsetof(struct nvme_common_command , dptr); + args->status =NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; + args->error_loc = offsetof(struct nvme_common_command , dptr); goto out_put_subsystem; }up_read (&nvmet_config_sem); - status =NVME_SC_INTERNAL ; + args->status =NVME_SC_INTERNAL ; ctrl =kzalloc (sizeof (*ctrl),GFP_KERNEL );if (!ctrl) goto out_put_subsystem;mutex_init (&ctrl->lock); - ctrl->port = req->port; - ctrl->ops = req->ops; + ctrl->port = args->port; + ctrl->ops = args->ops;#ifdef CONFIG_NVME_TARGET_PASSTHRU /* By default, set loop targets to clear IDS by default */
@@ -1439,8 +1438,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,INIT_WORK (&ctrl->fatal_err_work, nvmet_fatal_error_handler);INIT_DELAYED_WORK (&ctrl->ka_work, nvmet_keep_alive_timer); -memcpy (ctrl->subsysnqn, subsysnqn,NVMF_NQN_SIZE ); -memcpy (ctrl->hostnqn, hostnqn,NVMF_NQN_SIZE ); +memcpy (ctrl->subsysnqn, args->subsysnqn,NVMF_NQN_SIZE ); +memcpy (ctrl->hostnqn, args->hostnqn,NVMF_NQN_SIZE );kref_init (&ctrl->ref); ctrl->subsys = subsys;
@@ -1463,12 +1462,12 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, subsys->cntlid_min, subsys->cntlid_max, GFP_KERNEL);if (ret <0 ) { - status =NVME_SC_CONNECT_CTRL_BUSY |NVME_STATUS_DNR ; + args->status =NVME_SC_CONNECT_CTRL_BUSY |NVME_STATUS_DNR ; goto out_free_sqs; } ctrl->cntlid = ret; -uuid_copy (&ctrl->hostid, hostid); +uuid_copy (&ctrl->hostid, args->hostid);/* * Discovery controllers may use some arbitrary high value
@@ -1490,12 +1489,35 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,if (ret) goto init_pr_fail;list_add_tail (&ctrl->subsys_entry, &subsys->ctrls); -nvmet_setup_p2p_ns_map (ctrl, req); +nvmet_setup_p2p_ns_map (ctrl, args->p2p_client);nvmet_debugfs_ctrl_setup (ctrl);mutex_unlock (&subsys->lock); - *ctrlp = ctrl; -return 0 ; +if (args->hostid) +uuid_copy (&ctrl->hostid, args->hostid); + + dhchap_status =nvmet_setup_auth (ctrl); +if (dhchap_status) { +pr_err ("Failed to setup authentication, dhchap status %u\n" , + dhchap_status); +nvmet_ctrl_put (ctrl); +if (dhchap_status ==NVME_AUTH_DHCHAP_FAILURE_FAILED ) + args->status = +NVME_SC_CONNECT_INVALID_HOST |NVME_STATUS_DNR ; +else + args->status =NVME_SC_INTERNAL ; +return NULL ; + } + + args->status =NVME_SC_SUCCESS ; + +pr_info ("Created %s controller %d for subsystem %s for NQN %s%s%s.\n" , +nvmet_is_disc_subsys (ctrl->subsys) ?"discovery" :"nvm" , + ctrl->cntlid, ctrl->subsys->subsysnqn, ctrl->hostnqn, + ctrl->pi_support ?" T10-PI is enabled" :"" , +nvmet_has_auth (ctrl) ?" with DH-HMAC-CHAP" :"" ); + +return ctrl; init_pr_fail:mutex_unlock (&subsys->lock);
@@ -1509,9 +1531,9 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,kfree (ctrl); out_put_subsystem:nvmet_subsys_put (subsys); -out: -return status; +return NULL ; } +EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl);static void nvmet_ctrl_free (struct kref *ref) {
@@ -1547,6 +1569,7 @@ void nvmet_ctrl_put(struct nvmet_ctrl *ctrl) {kref_put (&ctrl->ref, nvmet_ctrl_free); } +EXPORT_SYMBOL_GPL (nvmet_ctrl_put);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl) {
drivers/nvme/target/fabrics-cmd.c +26 -32
99 lines not on this page · raw message
drivers/nvme/target/nvmet.h +15 -3
@@ -549,9 +549,21 @@ int nvmet_sq_init(struct nvmet_sq *sq);void nvmet_ctrl_fatal_error (struct nvmet_ctrl *ctrl);void nvmet_update_cc (struct nvmet_ctrl *ctrl,u32 new); -u16 nvmet_alloc_ctrl (const char *subsysnqn,const char *hostnqn, -struct nvmet_req *req,u32 kato,struct nvmet_ctrl **ctrlp, -uuid_t *hostid); + +struct nvmet_alloc_ctrl_args { +struct nvmet_port *port; +char *subsysnqn; +char *hostnqn; +uuid_t *hostid; +const struct nvmet_fabrics_ops *ops; +struct device *p2p_client; +u32 kato; +u32 result; +u16 error_loc; +u16 status; +}; + +struct nvmet_ctrl *nvmet_alloc_ctrl (struct nvmet_alloc_ctrl_args *args);struct nvmet_ctrl *nvmet_ctrl_find_get (const char *subsysnqn,const char *hostnqn,u16 cntlid,struct nvmet_req *req);
Signature · 1 line
2.47.1
PATCH v9 8/18 nvmet: Introduce nvmet_req_transfer_len()
In reply to Damien Le MoalAdd the new function nvmet_req_transfer_len() to parse a request command to extract the transfer length of the command. This function implementation relies on multiple helper functions for parsing I/O commands (nvmet_io_cmd_transfer_len()), admin commands (nvmet_admin_cmd_data_len()) and fabrics connect commands (nvmet_connect_cmd_data_len).
--- drivers/nvme/target/admin-cmd.c | 21 +++++++++++++ drivers/nvme/target/core.c | 37 ++++++++++++++++++++++ drivers/nvme/target/discovery.c | 14 +++++++++ drivers/nvme/target/fabrics-cmd-auth.c | 14 +++++++-- drivers/nvme/target/fabrics-cmd.c | 43 ++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 8 +++++ 6 files changed, 135 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +21 -0
@@ -1296,6 +1296,27 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)nvmet_req_complete (req, status); } +u32 nvmet_admin_cmd_data_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; + +if (nvme_is_fabrics (cmd)) +return nvmet_fabrics_admin_cmd_data_len (req); +if (nvmet_is_disc_subsys (nvmet_req_subsys (req))) +return nvmet_discovery_cmd_data_len (req); + +switch (cmd->common.opcode) { +case nvme_admin_get_log_page: +return nvmet_get_log_page_len (cmd); +case nvme_admin_identify: +return NVME_IDENTIFY_DATA_SIZE ; +case nvme_admin_get_features: +return nvmet_feat_data_len (req,le32_to_cpu (cmd->common.cdw10)); +default : +return 0 ; + } +} +u16 nvmet_parse_admin_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
drivers/nvme/target/core.c +37 -0
@@ -911,6 +911,33 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req)return 0 ; } +static u32 nvmet_io_cmd_transfer_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; +u32 metadata_len =0 ; + +if (nvme_is_fabrics (cmd)) +return nvmet_fabrics_io_cmd_data_len (req); + +if (!req->ns) +return 0 ; + +switch (req->cmd->common.opcode) { +case nvme_cmd_read: +case nvme_cmd_write: +case nvme_cmd_zone_append: +if (req->sq->ctrl->pi_support &&nvmet_ns_has_pi (req->ns)) + metadata_len =nvmet_rw_metadata_len (req); +return nvmet_rw_data_len (req) + metadata_len; +case nvme_cmd_dsm: +return nvmet_dsm_len (req); +case nvme_cmd_zone_mgmt_recv: +return (le32_to_cpu (req->cmd->zmr.numd) +1 ) <<2 ; +default : +return 0 ; + } +} +static u16 nvmet_parse_io_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
@@ -1059,6 +1086,16 @@ void nvmet_req_uninit(struct nvmet_req *req) }EXPORT_SYMBOL_GPL (nvmet_req_uninit); +size_t nvmet_req_transfer_len (struct nvmet_req *req) +{ +if (likely (req->sq->qid !=0 )) +return nvmet_io_cmd_transfer_len (req); +if (unlikely (!req->sq->ctrl)) +return nvmet_connect_cmd_data_len (req); +return nvmet_admin_cmd_data_len (req); +} +EXPORT_SYMBOL_GPL (nvmet_req_transfer_len); +bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len) {if (unlikely (len != req->transfer_len)) {
drivers/nvme/target/discovery.c +14 -0
@@ -355,6 +355,20 @@ static void nvmet_execute_disc_get_features(struct nvmet_req *req)nvmet_req_complete (req, stat); } +u32 nvmet_discovery_cmd_data_len (struct nvmet_req *req) +{ +struct nvme_command *cmd = req->cmd; + +switch (cmd->common.opcode) { +case nvme_admin_get_log_page: +return nvmet_get_log_page_len (req->cmd); +case nvme_admin_identify: +return NVME_IDENTIFY_DATA_SIZE ; +default : +return 0 ; + } +} +u16 nvmet_parse_discovery_cmd (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
drivers/nvme/target/fabrics-cmd-auth.c +12 -2
@@ -179,6 +179,11 @@ static u8 nvmet_auth_failure2(void *d)return data->rescode_exp; } +u32 nvmet_auth_send_data_len (struct nvmet_req *req) +{ +return le32_to_cpu (req->cmd->auth_send.tl); +} +void nvmet_execute_auth_send (struct nvmet_req *req) {struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -206,7 +211,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) offsetof(struct nvmf_auth_send_command , spsp1); goto done; } - tl =le32_to_cpu (req->cmd->auth_send.tl); + tl =nvmet_auth_send_data_len (req);if (!tl) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
@@ -429,6 +434,11 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al) data->rescode_exp = req->sq->dhchap_status; } +u32 nvmet_auth_receive_data_len (struct nvmet_req *req) +{ +return le32_to_cpu (req->cmd->auth_receive.al); +} +void nvmet_execute_auth_receive (struct nvmet_req *req) {struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -454,7 +464,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) offsetof(struct nvmf_auth_receive_command , spsp1); goto done; } - al =le32_to_cpu (req->cmd->auth_receive.al); + al =nvmet_auth_receive_data_len (req);if (!al) { status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; req->error_loc =
drivers/nvme/target/fabrics-cmd.c +43 -0
61 lines not on this page · raw message
drivers/nvme/target/nvmet.h +8 -0
@@ -517,18 +517,24 @@ void nvmet_start_keep_alive_timer(struct nvmet_ctrl *ctrl);void nvmet_stop_keep_alive_timer (struct nvmet_ctrl *ctrl);u16 nvmet_parse_connect_cmd (struct nvmet_req *req); +u32 nvmet_connect_cmd_data_len (struct nvmet_req *req);void nvmet_bdev_set_limits (struct block_device *bdev,struct nvme_id_ns *id);u16 nvmet_bdev_parse_io_cmd (struct nvmet_req *req);u16 nvmet_file_parse_io_cmd (struct nvmet_req *req);u16 nvmet_bdev_zns_parse_io_cmd (struct nvmet_req *req); +u32 nvmet_admin_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_admin_cmd (struct nvmet_req *req); +u32 nvmet_discovery_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_discovery_cmd (struct nvmet_req *req);u16 nvmet_parse_fabrics_admin_cmd (struct nvmet_req *req); +u32 nvmet_fabrics_admin_cmd_data_len (struct nvmet_req *req);u16 nvmet_parse_fabrics_io_cmd (struct nvmet_req *req); +u32 nvmet_fabrics_io_cmd_data_len (struct nvmet_req *req);bool nvmet_req_init (struct nvmet_req *req,struct nvmet_cq *cq,struct nvmet_sq *sq,const struct nvmet_fabrics_ops *ops);void nvmet_req_uninit (struct nvmet_req *req); +size_t nvmet_req_transfer_len (struct nvmet_req *req);bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len);bool nvmet_check_data_len_lte (struct nvmet_req *req,size_t data_len);void nvmet_req_complete (struct nvmet_req *req,u16 status);
@@ -822,7 +828,9 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio) }#ifdef CONFIG_NVME_TARGET_AUTH +u32 nvmet_auth_send_data_len (struct nvmet_req *req);void nvmet_execute_auth_send (struct nvmet_req *req); +u32 nvmet_auth_receive_data_len (struct nvmet_req *req);void nvmet_execute_auth_receive (struct nvmet_req *req);int nvmet_auth_set_key (struct nvmet_host *host,const char *secret,bool set_ctrl);
Signature · 1 line
2.47.1
PATCH v9 9/18 nvmet: Introduce nvmet_sq_create() and nvmet_cq_create()
In reply to Damien Le MoalIntroduce the new functions nvmet_sq_create() and nvmet_cq_create() to allow a target driver to initialize and setup admin and IO queues directly, without needing to execute connect fabrics commands. The helper functions nvmet_check_cqid() and nvmet_check_sqid() are implemented to check the correctness of SQ and CQ IDs when nvmet_sq_create() and nvmet_cq_create() are called.
nvmet_sq_create() and nvmet_cq_create() are primarily intended for use with PCI target controller drivers and thus are not well integrated with the current queue creation of fabrics controllers using the connect command. These fabrices drivers are not modified to use these functions. This simple implementation of SQ and CQ management for PCI target controller drivers does not allow multiple SQs to share the same CQ, similarly to other fabrics transports. This is a specification violation. A more involved set of changes will follow to add support for this required completion queue sharing feature.
--- drivers/nvme/target/core.c | 83 +++++++++++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 6 +++ 2 files changed, 89 insertions(+)
drivers/nvme/target/core.c +83 -0
@@ -818,6 +818,89 @@ static void nvmet_confirm_sq(struct percpu_ref *ref)complete (&sq->confirm_done); } +u16 nvmet_check_cqid (struct nvmet_ctrl *ctrl,u16 cqid) +{ +if (!ctrl->sqs) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + +if (cqid > ctrl->subsys->max_qid) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +/* +* Note: For PCI controllers, the NVMe specifications allows multiple +* SQs to share a single CQ. However, we do not support this yet, so +* check that there is no SQ defined for a CQ. If one exist, then the +* CQ ID is invalid for creation as well as when the CQ is being +* deleted (as that would mean that the SQ was not deleted before the +* CQ). +*/ +if (ctrl->sqs[cqid]) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +return NVME_SC_SUCCESS ; +} + +u16 nvmet_cq_create (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq, +u16 qid,u16 size) +{ +u16 status; + + status =nvmet_check_cqid (ctrl, qid); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_cq_setup (ctrl, cq, qid, size); + +return NVME_SC_SUCCESS ; +} +EXPORT_SYMBOL_GPL (nvmet_cq_create); + +u16 nvmet_check_sqid (struct nvmet_ctrl *ctrl,u16 sqid, +bool create) +{ +if (!ctrl->sqs) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + +if (sqid > ctrl->subsys->max_qid) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +if ((create && ctrl->sqs[sqid]) || + (!create && !ctrl->sqs[sqid])) +return NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + +return NVME_SC_SUCCESS ; +} + +u16 nvmet_sq_create (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq, +u16 sqid,u16 size) +{ +u16 status; +int ret; + +if (!kref_get_unless_zero (&ctrl->ref)) +return NVME_SC_INTERNAL |NVME_STATUS_DNR ; + + status =nvmet_check_sqid (ctrl, sqid, true); +if (status !=NVME_SC_SUCCESS ) +return status; + + ret =nvmet_sq_init (sq); +if (ret) { + status =NVME_SC_INTERNAL |NVME_STATUS_DNR ; + goto ctrl_put; + } + +nvmet_sq_setup (ctrl, sq, sqid, size); + sq->ctrl = ctrl; + +return NVME_SC_SUCCESS ; + +ctrl_put: +nvmet_ctrl_put (ctrl); +return status; +} +EXPORT_SYMBOL_GPL (nvmet_sq_create); +void nvmet_sq_destroy (struct nvmet_sq *sq) {struct nvmet_ctrl *ctrl = sq->ctrl;
drivers/nvme/target/nvmet.h +6 -0
@@ -545,10 +545,16 @@ void nvmet_execute_set_features(struct nvmet_req *req);void nvmet_execute_get_features (struct nvmet_req *req);void nvmet_execute_keep_alive (struct nvmet_req *req); +u16 nvmet_check_cqid (struct nvmet_ctrl *ctrl,u16 cqid);void nvmet_cq_setup (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq,u16 qid,u16 size); +u16 nvmet_cq_create (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq,u16 qid, +u16 size); +u16 nvmet_check_sqid (struct nvmet_ctrl *ctrl,u16 sqid,bool create);void nvmet_sq_setup (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq,u16 qid,u16 size); +u16 nvmet_sq_create (struct nvmet_ctrl *ctrl,struct nvmet_sq *sq,u16 qid, +u16 size);void nvmet_sq_destroy (struct nvmet_sq *sq);int nvmet_sq_init (struct nvmet_sq *sq);
Signature · 1 line
2.47.1
PATCH v9 10/18 nvmet: Add support for I/O queue management admin commands
In reply to Damien Le MoalThe I/O submission queue management admin commands (nvme_admin_delete_sq, nvme_admin_create_sq, nvme_admin_delete_cq, and nvme_admin_create_cq) are mandatory admin commands for I/O controllers using the PCI transport, that is, support for these commands is mandatory for a a PCI target I/O controller.
Implement support for these commands by adding the functions nvmet_execute_delete_sq(), nvmet_execute_create_sq(), nvmet_execute_delete_cq() and nvmet_execute_create_cq() to set as the execute method of requests for these commands. These functions will return an invalid opcode error for any controller that is not a PCI target controller. Support for the I/O queue management commands is also reported in the command effect log of PCI target controllers (using nvmet_get_cmd_effects_admin()).
Each management command is backed by a controller fabric operation that can be defined by a PCI target controller driver to setup I/O queues using nvmet_sq_create() and nvmet_cq_create() or delete I/O queues using nvmet_sq_destroy().
As noted in a comment in nvmet_execute_create_sq(), we do not yet support sharing a single CQ between multiple SQs.
--- drivers/nvme/target/admin-cmd.c | 165 +++++++++++++++++++++++++++++++- drivers/nvme/target/nvmet.h | 8 ++ 2 files changed, 170 insertions(+), 3 deletions(-)
drivers/nvme/target/admin-cmd.c +162 -3
@@ -12,6 +12,142 @@#include <linux/unaligned.h> #include "nvmet.h" +static void nvmet_execute_delete_sq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u16 sqid =le16_to_cpu (req->cmd->delete_queue.qid); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!sqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_sqid (ctrl, sqid, false); +if (status !=NVME_SC_SUCCESS ) + goto complete; + + status = ctrl->ops->delete_sq (ctrl, sqid); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_create_sq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +u16 sqid =le16_to_cpu (cmd->create_sq.sqid); +u16 cqid =le16_to_cpu (cmd->create_sq.cqid); +u16 sq_flags =le16_to_cpu (cmd->create_sq.sq_flags); +u16 qsize =le16_to_cpu (cmd->create_sq.qsize); +u64 prp1 =le64_to_cpu (cmd->create_sq.prp1); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!sqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_sqid (ctrl, sqid, true); +if (status !=NVME_SC_SUCCESS ) + goto complete; + +/* +* Note: The NVMe specification allows multiple SQs to use the same CQ. +* However, the target code does not really support that. So for now, +* prevent this and fail the command if sqid and cqid are different. +*/ +if (!cqid || cqid != sqid) { +pr_err ("SQ %u: Unsupported CQID %u\n" , sqid, cqid); + status =NVME_SC_CQ_INVALID |NVME_STATUS_DNR ; + goto complete; + } + +if (!qsize || qsize >NVME_CAP_MQES (ctrl->cap)) { + status =NVME_SC_QUEUE_SIZE |NVME_STATUS_DNR ; + goto complete; + } + + status = ctrl->ops->create_sq (ctrl, sqid, sq_flags, qsize, prp1); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_delete_cq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u16 cqid =le16_to_cpu (req->cmd->delete_queue.qid); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!cqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_cqid (ctrl, cqid); +if (status !=NVME_SC_SUCCESS ) + goto complete; + + status = ctrl->ops->delete_cq (ctrl, cqid); + +complete: +nvmet_req_complete (req, status); +} + +static void nvmet_execute_create_cq (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +u16 cqid =le16_to_cpu (cmd->create_cq.cqid); +u16 cq_flags =le16_to_cpu (cmd->create_cq.cq_flags); +u16 qsize =le16_to_cpu (cmd->create_cq.qsize); +u16 irq_vector =le16_to_cpu (cmd->create_cq.irq_vector); +u64 prp1 =le64_to_cpu (cmd->create_cq.prp1); +u16 status; + +if (!nvmet_is_pci_ctrl (ctrl)) { + status =nvmet_report_invalid_opcode (req); + goto complete; + } + +if (!cqid) { + status =NVME_SC_QID_INVALID |NVME_STATUS_DNR ; + goto complete; + } + + status =nvmet_check_cqid (ctrl, cqid); +if (status !=NVME_SC_SUCCESS ) + goto complete; + +if (!qsize || qsize >NVME_CAP_MQES (ctrl->cap)) { + status =NVME_SC_QUEUE_SIZE |NVME_STATUS_DNR ; + goto complete; + } + + status = ctrl->ops->create_cq (ctrl, cqid, cq_flags, qsize, + prp1, irq_vector); + +complete: +nvmet_req_complete (req, status); +} +u32 nvmet_get_log_page_len (struct nvme_command *cmd) {u32 len =le16_to_cpu (cmd->get_log_page.numdu);
@@ -230,8 +366,18 @@ static void nvmet_execute_get_log_page_smart(struct nvmet_req *req)nvmet_req_complete (req, status); } -static void nvmet_get_cmd_effects_admin (struct nvme_effects_log *log) +static void nvmet_get_cmd_effects_admin (struct nvmet_ctrl *ctrl, +struct nvme_effects_log *log) { +/* For a PCI target controller, advertize support for the . */ +if (nvmet_is_pci_ctrl (ctrl)) { + log->acs[nvme_admin_delete_sq] = + log->acs[nvme_admin_create_sq] = + log->acs[nvme_admin_delete_cq] = + log->acs[nvme_admin_create_cq] = +cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); + } + log->acs[nvme_admin_get_log_page] = log->acs[nvme_admin_identify] = log->acs[nvme_admin_abort_cmd] =
@@ -268,6 +414,7 @@ static void nvmet_get_cmd_effects_zns(struct nvme_effects_log *log)static void nvmet_execute_get_log_cmd_effects_ns (struct nvmet_req *req) { +struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvme_effects_log *log;u16 status =NVME_SC_SUCCESS ;
@@ -279,7 +426,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req)switch (req->cmd->get_log_page.csi) {case NVME_CSI_NVM : -nvmet_get_cmd_effects_admin (log); +nvmet_get_cmd_effects_admin (ctrl, log);nvmet_get_cmd_effects_nvm (log);break ;case NVME_CSI_ZNS :
@@ -287,7 +434,7 @@ static void nvmet_execute_get_log_cmd_effects_ns(struct nvmet_req *req) status =NVME_SC_INVALID_IO_CMD_SET ; goto free; } -nvmet_get_cmd_effects_admin (log); +nvmet_get_cmd_effects_admin (ctrl, log);nvmet_get_cmd_effects_nvm (log);nvmet_get_cmd_effects_zns (log);break ;
@@ -1335,9 +1482,21 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)return nvmet_parse_passthru_admin_cmd (req);switch (cmd->common.opcode) { +case nvme_admin_delete_sq: + req->execute = nvmet_execute_delete_sq; +return 0 ; +case nvme_admin_create_sq: + req->execute = nvmet_execute_create_sq; +return 0 ;case nvme_admin_get_log_page: req->execute = nvmet_execute_get_log_page;return 0 ; +case nvme_admin_delete_cq: + req->execute = nvmet_execute_delete_cq; +return 0 ; +case nvme_admin_create_cq: + req->execute = nvmet_execute_create_cq; +return 0 ;case nvme_admin_identify: req->execute = nvmet_execute_identify;return 0 ;
drivers/nvme/target/nvmet.h +8 -0
@@ -408,6 +408,14 @@ struct nvmet_fabrics_ops {void (*discovery_chg)(struct nvmet_port *port);u8 (*get_mdts)(const struct nvmet_ctrl *ctrl);u16 (*get_max_queue_size)(const struct nvmet_ctrl *ctrl); + +/* Operations mandatory for PCI target controllers */ +u16 (*create_sq)(struct nvmet_ctrl *ctrl,u16 sqid,u16 flags, +u16 qsize,u64 prp1); +u16 (*delete_sq)(struct nvmet_ctrl *ctrl,u16 sqid); +u16 (*create_cq)(struct nvmet_ctrl *ctrl,u16 cqid,u16 flags, +u16 qsize,u64 prp1,u16 irq_vector); +u16 (*delete_cq)(struct nvmet_ctrl *ctrl,u16 cqid); };#define NVMET_MAX_INLINE_BIOVEC 8
Signature · 1 line
2.47.1
PATCH v9 11/18 nvmet: Do not require SGL for PCI target controller commands
In reply to Damien Le MoalSupport for SGL is optional for the PCI transport. Modify nvmet_req_init() to not require the NVME_CMD_SGL_METABUF command flag to be set if the target controller transport type is NVMF_TRTYPE_PCI. In addition to this, the NVMe base specification v2.1 mandate that all admin commands use PRP, that is, have CDW0.PSDT cleared to 0. Modify nvmet_parse_admin_cmd() to check this.
Finally, modify nvmet_check_transfer_len() and nvmet_check_data_len_lte() to return the appropriate error status depending on the command using SGL or PRP. Since for fabrics nvmet_req_init() checks that a command uses SGL, always, this change affects only PCI target controllers.
--- drivers/nvme/target/admin-cmd.c | 5 +++++ drivers/nvme/target/core.c | 27 +++++++++++++++++++++------ 2 files changed, 26 insertions(+), 6 deletions(-)
drivers/nvme/target/admin-cmd.c +5 -0
@@ -1478,6 +1478,11 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)if (unlikely (ret))return ret; +/* For PCI controllers, admin commands shall not use SGL. */ +if (nvmet_is_pci_ctrl (req->sq->ctrl) && !req->sq->qid && + cmd->common.flags &NVME_CMD_SGL_ALL ) +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; +if (nvmet_is_passthru_req (req))return nvmet_parse_passthru_admin_cmd (req);
drivers/nvme/target/core.c +21 -6
@@ -1122,12 +1122,15 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,/* * For fabrics, PSDT field shall describe metadata pointer (MPTR) that * contains an address of a single contiguous physical buffer that is -* byte aligned. +* byte aligned. For PCI controllers, this is optional so not enforced. */ if (unlikely ((flags &NVME_CMD_SGL_ALL ) !=NVME_CMD_SGL_METABUF )) { - req->error_loc = offsetof(struct nvme_common_command , flags); - status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; - goto fail; +if (!req->sq->ctrl || !nvmet_is_pci_ctrl (req->sq->ctrl)) { + req->error_loc = + offsetof(struct nvme_common_command , flags); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto fail; + } }if (unlikely (!req->sq->ctrl))
@@ -1182,8 +1185,14 @@ EXPORT_SYMBOL_GPL(nvmet_req_transfer_len);bool nvmet_check_transfer_len (struct nvmet_req *req,size_t len) {if (unlikely (len != req->transfer_len)) { +u16 status; + req->error_loc = offsetof(struct nvme_common_command , dptr); -nvmet_req_complete (req,NVME_SC_SGL_INVALID_DATA |NVME_STATUS_DNR ); +if (req->cmd->common.flags &NVME_CMD_SGL_ALL ) + status =NVME_SC_SGL_INVALID_DATA ; +else + status =NVME_SC_INVALID_FIELD ; +nvmet_req_complete (req, status |NVME_STATUS_DNR );return false; }
@@ -1194,8 +1203,14 @@ EXPORT_SYMBOL_GPL(nvmet_check_transfer_len);bool nvmet_check_data_len_lte (struct nvmet_req *req,size_t data_len) {if (unlikely (data_len > req->transfer_len)) { +u16 status; + req->error_loc = offsetof(struct nvme_common_command , dptr); -nvmet_req_complete (req,NVME_SC_SGL_INVALID_DATA |NVME_STATUS_DNR ); +if (req->cmd->common.flags &NVME_CMD_SGL_ALL ) + status =NVME_SC_SGL_INVALID_DATA ; +else + status =NVME_SC_INVALID_FIELD ; +nvmet_req_complete (req, status |NVME_STATUS_DNR );return false; }
Signature · 1 line
2.47.1
PATCH v9 12/18 nvmet: Introduce get/set_feature controller operations
In reply to Damien Le MoalThe implementation of some features cannot always be done generically by the target core code. Arbitraion and IRQ coalescing features are examples of such features: their implementation must be provided (at least partially) by the target controller driver.
Introduce the set_feature() and get_feature() controller fabrics operations (in struct nvmet_fabrics_ops) to allow supporting such features.
--- drivers/nvme/target/nvmet.h | 4 ++++ 1 file changed, 4 insertions(+)
drivers/nvme/target/nvmet.h +4 -0
@@ -416,6 +416,10 @@ struct nvmet_fabrics_ops {u16 (*create_cq)(struct nvmet_ctrl *ctrl,u16 cqid,u16 flags,u16 qsize,u64 prp1,u16 irq_vector);u16 (*delete_cq)(struct nvmet_ctrl *ctrl,u16 cqid); +u16 (*set_feature)(const struct nvmet_ctrl *ctrl,u8 feat, +void *feat_data); +u16 (*get_feature)(const struct nvmet_ctrl *ctrl,u8 feat, +void *feat_data); };#define NVMET_MAX_INLINE_BIOVEC 8
Signature · 1 line
2.47.1
PATCH v9 13/18 nvmet: Implement host identifier set feature support
In reply to Damien Le MoalThe NVMe specifications mandate support for the host identifier set_features for controllers that also supports reservations. Satisfy this requirement by implementing handling of the NVME_FEAT_HOST_ID feature for the nvme_set_features command. This implementation is for now effective only for PCI target controllers. For other controller types, the set features command is failed with a NVME_SC_CMD_SEQ_ERROR status as before.
As noted in the code, 128 bits host identifiers are supported since the NVMe base specifications version 2.1 indicate in section 5.1.25.1.28.1 that "The controller may support a 64-bit Host Identifier...".
The RHII (Reservations and Host Identifier Interaction) bit of the controller attribute (ctratt) field of the identify controller data is also set to indicate that a host ID of "0" is supported but that the host ID must be a non-zero value to use reservations.
--- drivers/nvme/target/admin-cmd.c | 35 +++++++++++++++++++++++++++++---- include/linux/nvme.h | 1 + 2 files changed, 32 insertions(+), 4 deletions(-)
drivers/nvme/target/admin-cmd.c +31 -4
@@ -659,7 +659,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)struct nvmet_ctrl *ctrl = req->sq->ctrl;struct nvmet_subsys *subsys = ctrl->subsys;struct nvme_id_ctrl *id; -u32 cmd_capsule_size; +u32 cmd_capsule_size, ctratt;u16 status =0 ;if (!subsys->subsys_discovered) {
@@ -707,8 +707,10 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)/* XXX: figure out what to do about RTD3R/RTD3 */ id->oaes =cpu_to_le32 (NVMET_AEN_CFG_OPTIONAL ); - id->ctratt =cpu_to_le32 (NVME_CTRL_ATTR_HID_128_BIT | -NVME_CTRL_ATTR_TBKAS ); + ctratt =NVME_CTRL_ATTR_HID_128_BIT |NVME_CTRL_ATTR_TBKAS ; +if (nvmet_is_pci_ctrl (ctrl)) + ctratt |=NVME_CTRL_ATTR_RHII ; + id->ctratt =cpu_to_le32 (ctratt); id->oacs =0 ;
@@ -1255,6 +1257,31 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)return 0 ; } +static u16 nvmet_set_feat_host_id (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; + +if (!nvmet_is_pci_ctrl (ctrl)) +return NVME_SC_CMD_SEQ_ERROR |NVME_STATUS_DNR ; + +/* +* The NVMe base specifications v2.1 recommends supporting 128-bits host +* IDs (section 5.1.25.1.28.1). However, that same section also says +* that "The controller may support a 64-bit Host Identifier and/or an +* extended 128-bit Host Identifier". So simplify this support and do +* not support 64-bits host IDs to avoid needing to check that all +* controllers associated with the same subsystem all use the same host +* ID size. +*/ +if (!(req->cmd->common.cdw11 &cpu_to_le32 (1 <<0 ))) { + req->error_loc = offsetof(struct nvme_common_command , cdw11); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return nvmet_copy_from_sgl (req,0 , &req->sq->ctrl->hostid, +sizeof (req->sq->ctrl->hostid)); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1285,7 +1312,7 @@ void nvmet_execute_set_features(struct nvmet_req *req) status =nvmet_set_feat_async_event (req,NVMET_AEN_CFG_ALL );break ;case NVME_FEAT_HOST_ID : - status =NVME_SC_CMD_SEQ_ERROR |NVME_STATUS_DNR ; + status =nvmet_set_feat_host_id (req);break ;case NVME_FEAT_WRITE_PROTECT : status =nvmet_set_feat_write_protect (req);
include/linux/nvme.h +1 -0
@@ -276,6 +276,7 @@ enum nvme_ctrl_attr {NVME_CTRL_ATTR_HID_128_BIT = (1 <<0 ),NVME_CTRL_ATTR_TBKAS = (1 <<6 ),NVME_CTRL_ATTR_ELBAS = (1 <<15 ), + NVME_CTRL_ATTR_RHII = (1 << 18), }; struct nvme_id_ctrl {
Signature · 1 line
2.47.1
PATCH v9 14/18 nvmet: Implement interrupt coalescing feature support
In reply to Damien Le MoalThe NVMe base specifications v2.1 mandate Supporting the interrupt coalescing feature (NVME_FEAT_IRQ_COALESCE) for PCI controllers. Introduce the data structure struct nvmet_feat_irq_coalesce to define the time and threshold (thr) fields of this feature and implement the functions nvmet_get_feat_irq_coalesce() and nvmet_set_feat_irq_coalesce() to get and set this feature. These functions respectively use the controller get_feature() and set_feature() operations to fill and handle the fields of struct nvmet_feat_irq_coalesce.
While the Linux kernel nvme driver does not use this feature and thus will not complain if it is not implemented, other major OSes fail initializing the NVMe device if this feature support is missing.
Support for this feature is prohibited for fabrics controllers. If a get feature or set feature command for this feature is received for a fabrics controller, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 53 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 10 +++++++ 2 files changed, 61 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +51 -2
@@ -1282,6 +1282,27 @@ static u16 nvmet_set_feat_host_id(struct nvmet_req *req) sizeof(req->sq->ctrl->hostid)); } +static u16 nvmet_set_feat_irq_coalesce (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_irq_coalesce irqc = { + .time = (cdw11 >>8 ) &0xff , + .thr = cdw11 &0xff , + }; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1305,6 +1326,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)nvmet_set_result (req, (subsys->max_qid -1 ) | ((subsys->max_qid -1 ) <<16 ));break ; +case NVME_FEAT_IRQ_COALESCE : + status =nvmet_set_feat_irq_coalesce (req); +break ;case NVME_FEAT_KATO : status =nvmet_set_feat_kato (req);break ;
@@ -1349,6 +1373,30 @@ static u16 nvmet_get_feat_write_protect(struct nvmet_req *req)return 0 ; } +static u16 nvmet_get_feat_irq_coalesce (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvmet_feat_irq_coalesce irqc = { }; +u16 status; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, ((u32 )irqc.time <<8 ) | (u32 )irqc.thr); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1383,13 +1431,14 @@ void nvmet_execute_get_features(struct nvmet_req *req)break ;case NVME_FEAT_ERR_RECOVERY :break ; -case NVME_FEAT_IRQ_COALESCE : -break ;case NVME_FEAT_IRQ_CONFIG :break ;case NVME_FEAT_WRITE_ATOMIC :break ;#endif +case NVME_FEAT_IRQ_COALESCE : + status =nvmet_get_feat_irq_coalesce (req); +break ;case NVME_FEAT_ASYNC_EVENT :nvmet_get_feat_async_event (req);break ;
drivers/nvme/target/nvmet.h +10 -0
@@ -906,4 +906,14 @@ static inline void nvmet_pr_put_ns_pc_ref(struct nvmet_pr_per_ctrl_ref *pc_ref) {percpu_ref_put (&pc_ref->ref); } + +/* +* Data for the get_feature() and set_feature() operations of PCI target +* controllers. +*/ +struct nvmet_feat_irq_coalesce { +u8 thr; +u8 time; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v9 15/18 nvmet: Implement interrupt config feature support
In reply to Damien Le MoalThe NVMe base specifications v2.1 mandate supporting the interrupt config feature (NVME_FEAT_IRQ_CONFIG) for PCI controllers. Introduce the data structure struct nvmet_feat_irq_config to define the coalescing disabled (cd) and interrupt vector (iv) fields of this feature and implement the functions nvmet_get_feat_irq_config() and nvmet_set_feat_irq_config() functions to get and set these fields. These functions respectively use the controller get_feature() and set_feature() operations to fill and handle the fields of struct nvmet_feat_irq_config.
Support for this feature is prohibited for fabrics controllers. If a get feature command or a set feature command for this feature is received for a fabrics controller, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 54 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 5 +++ 2 files changed, 57 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +52 -2
@@ -1303,6 +1303,27 @@ static u16 nvmet_set_feat_irq_coalesce(struct nvmet_req *req)return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_COALESCE , &irqc); } +static u16 nvmet_set_feat_irq_config (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_irq_config irqcfg = { + .iv = cdw11 &0xffff , + .cd = (cdw11 >>16 ) &0x1 , + }; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1329,6 +1350,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)case NVME_FEAT_IRQ_COALESCE : status =nvmet_set_feat_irq_coalesce (req);break ; +case NVME_FEAT_IRQ_CONFIG : + status =nvmet_set_feat_irq_config (req); +break ;case NVME_FEAT_KATO : status =nvmet_set_feat_kato (req);break ;
@@ -1397,6 +1421,31 @@ static u16 nvmet_get_feat_irq_coalesce(struct nvmet_req *req)return NVME_SC_SUCCESS ; } +static u16 nvmet_get_feat_irq_config (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 iv =le32_to_cpu (req->cmd->common.cdw11) &0xffff ; +struct nvmet_feat_irq_config irqcfg = { .iv = iv }; +u16 status; + +/* +* This feature is not supported for fabrics controllers and mandatory +* for PCI controllers. +*/ +if (!nvmet_is_pci_ctrl (ctrl)) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, ((u32 )irqcfg.cd <<16 ) | iv); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1431,14 +1480,15 @@ void nvmet_execute_get_features(struct nvmet_req *req)break ;case NVME_FEAT_ERR_RECOVERY :break ; -case NVME_FEAT_IRQ_CONFIG : -break ;case NVME_FEAT_WRITE_ATOMIC :break ;#endif case NVME_FEAT_IRQ_COALESCE : status =nvmet_get_feat_irq_coalesce (req);break ; +case NVME_FEAT_IRQ_CONFIG : + status =nvmet_get_feat_irq_config (req); +break ;case NVME_FEAT_ASYNC_EVENT :nvmet_get_feat_async_event (req);break ;
drivers/nvme/target/nvmet.h +5 -0
@@ -916,4 +916,9 @@ struct nvmet_feat_irq_coalesce { u8 time; }; +struct nvmet_feat_irq_config { +u16 iv; +bool cd; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v9 16/18 nvmet: Implement arbitration feature support
In reply to Damien Le MoalNVMe base specification v2.1 mandates support for the arbitration feature (NVME_FEAT_ARBITRATION). Introduce the data structure struct nvmet_feat_arbitration to define the high, medium and low priority weight fields and the arbitration burst field of this feature and implement the functions nvmet_get_feat_arbitration() and nvmet_set_feat_arbitration() functions to get and set these fields.
Since there is no generic way to implement support for the arbitration feature, these functions respectively use the controller get_feature() and set_feature() operations to process the feature with the help of the controller driver. If the controller driver does not implement these operations and a get feature command or a set feature command for this feature is received, the command is failed with an invalid field error.
--- drivers/nvme/target/admin-cmd.c | 51 +++++++++++++++++++++++++++++++-- drivers/nvme/target/nvmet.h | 7 +++++ 2 files changed, 56 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +49 -2
@@ -1324,6 +1324,25 @@ static u16 nvmet_set_feat_irq_config(struct nvmet_req *req)return ctrl->ops->set_feature (ctrl,NVME_FEAT_IRQ_CONFIG , &irqcfg); } +static u16 nvmet_set_feat_arbitration (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +u32 cdw11 =le32_to_cpu (req->cmd->common.cdw11); +struct nvmet_feat_arbitration arb = { + .hpw = (cdw11 >>24 ) &0xff , + .mpw = (cdw11 >>16 ) &0xff , + .lpw = (cdw11 >>8 ) &0xff , + .ab = cdw11 &0x3 , + }; + +if (!ctrl->ops->set_feature) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + +return ctrl->ops->set_feature (ctrl,NVME_FEAT_ARBITRATION , &arb); +} +void nvmet_execute_set_features (struct nvmet_req *req) {struct nvmet_subsys *subsys =nvmet_req_subsys (req);
@@ -1337,6 +1356,9 @@ void nvmet_execute_set_features(struct nvmet_req *req)return ;switch (cdw10 &0xff ) { +case NVME_FEAT_ARBITRATION : + status =nvmet_set_feat_arbitration (req); +break ;case NVME_FEAT_NUM_QUEUES : ncqr = (cdw11 >>16 ) &0xffff ; nsqr = cdw11 &0xffff ;
@@ -1446,6 +1468,30 @@ static u16 nvmet_get_feat_irq_config(struct nvmet_req *req)return NVME_SC_SUCCESS ; } +static u16 nvmet_get_feat_arbitration (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvmet_feat_arbitration arb = { }; +u16 status; + +if (!ctrl->ops->get_feature) { + req->error_loc = offsetof(struct nvme_common_command , cdw10); +return NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + } + + status = ctrl->ops->get_feature (ctrl,NVME_FEAT_ARBITRATION , &arb); +if (status !=NVME_SC_SUCCESS ) +return status; + +nvmet_set_result (req, + ((u32 )arb.hpw <<24 ) | + ((u32 )arb.mpw <<16 ) | + ((u32 )arb.lpw <<8 ) | + (arb.ab &0x3 )); + +return NVME_SC_SUCCESS ; +} +void nvmet_get_feat_kato (struct nvmet_req *req) {nvmet_set_result (req, req->sq->ctrl->kato *1000 );
@@ -1472,8 +1518,6 @@ void nvmet_execute_get_features(struct nvmet_req *req)* need to come up with some fake values for these. */ #if 0 -case NVME_FEAT_ARBITRATION : -break ;case NVME_FEAT_POWER_MGMT :break ;case NVME_FEAT_TEMP_THRESH :
@@ -1483,6 +1527,9 @@ void nvmet_execute_get_features(struct nvmet_req *req)case NVME_FEAT_WRITE_ATOMIC :break ;#endif +case NVME_FEAT_ARBITRATION : + status =nvmet_get_feat_arbitration (req); +break ;case NVME_FEAT_IRQ_COALESCE : status =nvmet_get_feat_irq_coalesce (req);break ;
drivers/nvme/target/nvmet.h +7 -0
@@ -921,4 +921,11 @@ struct nvmet_feat_irq_config { bool cd; }; +struct nvmet_feat_arbitration { +u8 hpw; +u8 mpw; +u8 lpw; +u8 ab; +}; +#endif /* _NVMET_H */
Signature · 1 line
2.47.1
PATCH v9 17/18 nvmet: New NVMe PCI endpoint function target driver
In reply to Damien Le MoalImplement a PCI target driver using the PCI endpoint framework. This requires hardware with a PCI controller capable of executing in endpoint mode.
The PCI endpoint framework is used to set up a PCI endpoint function and its BAR compatible with a NVMe PCI controller. The framework is also used to map local memory to the PCI address space to execute MMIO accesses for retrieving NVMe commands from submission queues and posting completion entries to completion queues. If supported, DMA is used for command retreival and command data transfers, based on the PCI address segments indicated by the command using either PRPs or SGLs.
The NVMe target driver relies on the NVMe target core code to execute all commands isssued by the host. The PCI target driver is mainly responsible for the following:
- Initialization and teardown of the endpoint device and its backend PCI target controller. The PCI target controller is created using a subsystem and a port defined through configfs. The port used must be initialized with the "pci" transport type. The target controller is allocated and initialized when the PCI endpoint is started by binding it to the endpoint PCI device (nvmet_pci_epf_epc_init() function).
- Manage the endpoint controller state according to the PCI link state and the actions of the host (e.g. checking the CC.EN register) and propagate these actions to the PCI target controller. Polling of the controller enable/disable is done using a delayed work scheduled every 5ms (nvmet_pci_epf_poll_cc() function). This work is started whenever the PCI link comes up (nvmet_pci_epf_link_up() notifier function) and stopped when the PCI link comes down (nvmet_pci_epf_link_down() notifier function). nvmet_pci_epf_poll_cc() enables and disables the PCI controller using the functions nvmet_pci_epf_enable_ctrl() and nvmet_pci_epf_disable_ctrl(). The controller admin queue is created using nvmet_pci_epf_create_cq(), which calls nvmet_cq_create(), and nvmet_pci_epf_create_sq() which uses nvmet_sq_create(). nvmet_pci_epf_disable_ctrl() always resets the PCI controller to its initial state so that nvmet_pci_epf_enable_ctrl() can be called again. This ensures correct operation if, for instance, the host reboots causing the PCI link to be temporarily down.
- Manage the controller admin and I/O submission queues using local memory. Commands are obtained from submission queues using a work item that constantly polls the doorbells of all submissions queues (nvmet_pci_epf_poll_sqs() function). This work is started whenever the controller is enabled (nvmet_pci_epf_enable_ctrl() function) and stopped when the controller is disabled (nvmet_pci_epf_disable_ctrl() function). When new commands are submitted by the host, DMA transfers are used to retrieve the commands.
- Initiate the execution of all admin and I/O commands using the target core code, by calling a requests execute() function. All commands are individually handled using a per-command work item (nvmet_pci_epf_iod_work() function). A command overall execution
includes: initializing a struct nvmet_req request for the command, using nvmet_req_transfer_len() to get a command data transfer length, parse the command PRPs or SGLs to get the PCI address segments of the command data buffer, retrieve data from the host (if the command is a write command), call req->execute() to execute the command and transfer data to the host (for read commands).
- Handle the completions of commands as notified by the ->queue_response() operation of the PCI target controller (nvmet_pci_epf_queue_response() function). Completed commands are added to a list of completed command for their CQ. Each CQ list of completed command is processed using a work item (nvmet_pci_epf_cq_work() function) which posts entries for the completed commands in the CQ memory and raise an IRQ to the host to signal the completion. IRQ coalescing is supported as mandated by the NVMe base specification for PCI controllers. Of note is that completion entries are transmitted to the host using MMIO, after mapping the completion queue memory to the host PCI address space. Unlike for retrieving commands from SQs, DMA is not used as it degrades performance due to the transfer serialization needed (which delays completion entries transmission).
The configuration of a NVMe PCI endpoint controller is done using configfs. First the NVMe PCI target controller configuration must be done to set up a subsystem and a port with the "pci" addr_trtype attribute. The subsystem can be setup using a file or block device backed namespace or using a passthrough NVMe device. After this, the PCI endpoint can be configured and bound to the PCI endpoint controller to start the NVMe endpoint controller.
In order to not overcomplicate this initial implementation of an endpoint PCI target controller driver, protection information is not for now supported. If the PCI controller port and namespace are configured with protection information support, an error will be returned when the controller is created and initialized when the endpoint function is started. Protection information support will be added in a follow-up patch series.
Using a Rock5B board (Rockchip RK3588 SoC, PCI Gen3x4 endpoint controller) with a target PCI controller setup with 4 I/O queues and a null_blk block device as a namespace, the maximum performance using fio was measured at 131 KIOPS for random 4K reads and up to 2.8 GB/S throughput. Some data points are:
Rnd read, 4KB, QD=1, 1 job : IOPS=16.9k, BW=66.2MiB/s (69.4MB/s) Rnd read, 4KB, QD=32, 1 job : IOPS=78.5k, BW=307MiB/s (322MB/s) Rnd read, 4KB, QD=32, 4 jobs: IOPS=131k, BW=511MiB/s (536MB/s)
Seq read, 512KB, QD=32, 1 job : IOPS=5381, BW=2691MiB/s (2821MB/s)
The NVMe PCI endpoint target driver is not intended for production use. It is a tool for learning NVMe, exploring existing features and testing implementations of new NVMe features.
--- drivers/nvme/target/Kconfig | 11 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/pci-epf.c | 2591 +++++++++++++++++++++++++++++++++ 3 files changed, 2604 insertions(+) create mode 100644 drivers/nvme/target/pci-epf.c
drivers/nvme/target/Kconfig +11 -0
@@ -115,3 +115,14 @@ config NVME_TARGET_AUTH target side. If unsure, say N. + +config NVME_TARGET_PCI_EPF + tristate "NVMe PCI Endpoint Function target support" + depends on NVME_TARGET && PCI_ENDPOINT + depends on NVME_CORE=y || NVME_CORE=NVME_TARGET + help + This enables the NVMe PCI Endpoint Function target driver support, + which allows creating a NVMe PCI controller using an endpoint mode + capable PCI controller. + + If unsure, say N.
drivers/nvme/target/Makefile +2 -0
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_TARGET_RDMA) += nvmet-rdma.o obj-$(CONFIG_NVME_TARGET_FC) += nvmet-fc.o obj-$(CONFIG_NVME_TARGET_FCLOOP) += nvme-fcloop.o obj-$(CONFIG_NVME_TARGET_TCP) += nvmet-tcp.o +obj-$(CONFIG_NVME_TARGET_PCI_EPF) += nvmet-pci-epf.o nvmet-y += core.o configfs.o admin-cmd.o fabrics-cmd.o \ discovery.o io-cmd-file.o io-cmd-bdev.o pr.o
@@ -20,4 +21,5 @@ nvmet-rdma-y += rdma.o nvmet-fc-y += fc.o nvme-fcloop-y += fcloop.o nvmet-tcp-y += tcp.o +nvmet-pci-epf-y += pci-epf.o nvmet-$(CONFIG_TRACING) += trace.o
drivers/nvme/target/pci-epf.c new file +2591 -0
2,591 lines not on this page · raw message
Signature · 1 line
2.47.1
PATCH v9 18/18 Documentation: Document the NVMe PCI endpoint target driver
In reply to Damien Le MoalAdd a documentation file (Documentation/nvme/nvme-pci-endpoint-target.rst) for the new NVMe PCI endpoint target driver. This provides an overview of the driver requirements, capabilities and limitations. A user guide describing how to setup a NVMe PCI endpoint device using this driver is also provided.
This document is made accessible also from the PCI endpoint documentation using a link. Furthermore, since the existing nvme documentation was not accessible from the top documentation index, an index file is added to Documentation/nvme and this index listed as "NVMe Subsystem" in the "Storage interfaces" section of the subsystem API index.
--- Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 13 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 368 ++++++++++++++++++ Documentation/subsystem-apis.rst | 1 + 5 files changed, 395 insertions(+) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst
Documentation/PCI/endpoint/index.rst +1 -0
@@ -15,6 +15,7 @@ PCI Endpoint Framework pci-ntb-howto pci-vntb-function pci-vntb-howto + pci-nvme-function function/binding/pci-test function/binding/pci-ntb
Documentation/PCI/endpoint/pci-nvme-function.rst new file +13 -0
@@ -0,0 +1,13 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================= +PCI NVMe Function +================= + +:Author: Damien Le Moal <[email protected]> + +The PCI NVMe endpoint function implements a PCI NVMe controller using the NVMe +subsystem target core code. The driver for this function resides with the NVMe +subsystem as drivers/nvme/target/nvmet-pciep.c. + +See Documentation/nvme/nvme-pci-endpoint-target.rst for more details.
Documentation/nvme/index.rst new file +12 -0
@@ -0,0 +1,12 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +NVMe Subsystem +============== + +.. toctree:: + :maxdepth: 2 + :numbered: + + feature-and-quirk-policy + nvme-pci-endpoint-target
Documentation/nvme/nvme-pci-endpoint-target.rst new file +368 -0
368 lines not on this page · raw message
Documentation/subsystem-apis.rst +1 -0
@@ -60,6 +60,7 @@ Storage interfaces cdrom/index scsi/index target/index + nvme/index Other subsystems ----------------
Signature · 1 line
2.47.1
Quoted 2 lines · Damien Le Moal
This patch series implements an NVMe target driver for the PCI transport using the PCI endpoint framework.
Thanks, applied to nvme-6.14.