Skip to contentSkip to conversation

nvmet: add target passthru commands support

PATCH Superseded Logan Gunthorpe 5 participants 28 messages Jul 16, 2020 – Jul 22, 2020

Subsystems (2)

v15 of 16 Superseded Latest is v16 →

lore
Share & tools
lore
#0 lore raw

v15 cover nvmet: add target passthru commands support

This is v15 of the passthru patchset which fixes a random kconfig bug caught by the kernel test robot.

A git branch is available here and is based on nvme-5.9 on Jul 16th:

https://github.com/sbates130272/linux-p2pmem nvmet_passthru_v15

--

v15 Changes:
1. Rebased onto nvme-5.9, required minor merge.
2. Ensure NVME_CORE is not a module if NVME_TARGET is builtin when selecting NVME_TARGET_PASSTHRU (per kernel test robot)

v14 Changes:
1. Rebased onto nvme-5.9 (per Chaitanya's suggestion), required minor merge.
2. Dropped the #ifdef around the prototypes in nvme.h because kbuild started complaining about non-staic functions that don't have prototypes.
3. Removed an unecessary if/else in nvmet_passthru_execute_cmd() as spotted by Chaitanya.
4. Removed unecessary casts in a pr_warn(), per Chaitanya

v13 Changes:
1. Rebased onto v5.7-rc5
2. Collected Sagi's Reviewed-by tags

v12 Changes:
1. Rebased onto v5.7-rc1
2. Collected Sagi's Reviewed-by tags
3. Per Sagi's feedback implement an whitelist for set/get features and audit the features for whether they are suitable to be passed-through

v11 Changes:
1. Rebased onto v5.6-rc2
2. Collected Max's Reviewed-By tag

v10 Changes:
1. Rebased onto v5.5-rc1
2. Disable all exports in core nvme if CONFIG_NVME_TARGET_PASSTHRU is not set and put them near the end of the file with a big fat comment (per Christoph)
3. Don't fake up the vs field: pass it through as is and bump it to 1.2.1 if it is below that (per Christoph)
4. Rework how passthru requests are submitted into the core with proper nvme_passthru_start/end handling (per Christoph)
5. Rework how commands are parsed with passthru hooks in parse_admin_cmd() and nvmet_parse_io_cmd() (per Christoph)
6. Rework commands are handled so they are only done in a work item if absolutely necessary (per Christoph)
7. The data_len hack was dropped as a patchset was introduced to remove data_len altogether (per Christoph)
8. The passthru accounting changes are now in v5.5-rc1
9. A large number of other minor cleanups that were pointed out by Christoph

v9 Changes:
1. Rebased onto v5.4-rc2 (required adjusting nvme_identify_ns() usage)
2. Collected Sagi's Reviewed-By Tags
3. Squashed seperate Kconfig patch into passthru patch (Per Sagi)
4. Set REQ_FUA for flush requests and remove special casing on RQF_IO_STAT (Per Sagi)

v8 Changes:
1. Rebased onto v5.3-rc6
2. Collected Max's Reviewed-By tags
3. Converted admin command black-list to a white-list, but allow all vendor specific commands. With this, we feel it's safe to allow multiple connections from hosts. (As per Sagi's feedback)

v7 Changes:
1. Rebased onto v5.3-rc2
2. Rework nvme_ctrl_get_by_path() to use filp_open() instead of the cdev changes that were in v6. (Per Al)
3. Override the cmic bit to allow multipath and allow multiple connections from the same hostnqn. (At the same time I cleaned up the method of rejecting multiple connections.) See Patch 8)
4. Found a bug when used with the tcp transport (See Patch 10)

v6 Changes:
1. Rebased onto v5.3-rc1
2. Rework configfs interface to simply be a passthru directory within the existing subsystem. The directory is similar to and consistent with a namespace directory.
3. Have the configfs take a path instead of a bare controller name
4. Renamed the main passthru file to io-cmd-passthru.c for consistency with the file and block-dev methods.
5. Cleaned up all the CONFIG_NVME_TARGET_PASSTHRU usage to remove all the inline #ifdefs
6. Restructured nvmet_passthru_make_request() a bit for clearer code
7. Moved nvme_find_get_ns() call into nvmet_passthru_execute_cmd() seeing calling it in nvmet_req_init() causes a lockdep warning due to nvme_find_get_ns() being able to sleep.
8. Added a check in nvmet_passthru_execute_cmd() to ensure we don't violate queue_max_segments or queue_max_hw_sectors and overrode mdts to ensure hosts don't intentionally submit commands that will exceed these limits.
9. Reworked the code which ensures there's only one subsystem per passthru controller to use an xarray instead of a list as this is simpler and more easily fixed some bugs triggered by disabling subsystems that weren't enabled.
10. Removed the overide of the target cntlid with the passthru cntlid; this seemed like a really bad idea especially in the presence of mixed systems as you could end up with two ctrlrs with the same cntlid. For now, commands that depend on cntlid are black listed.
11. Implement block accounting for passthru so the target can track usage using /proc/diskstats
12. A number of other minor bug fixes and cleanups

v5 Changes (not sent to list, from Chaitanya):
1. Added workqueue for admin commands.
2. Added kconfig option for the pass-thru feature.
3. Restructure the parsing code according to your suggestion, call nvmet_xxx_parse_cmd() from nvmet_passthru_parse_cmd().
4. Use pass-thru instead of pt.
5. Several cleanups and add comments at the appropriate locations.
6. Minimize the code for checking pass-thru ns across all the subsystems.
7. Removed the delays in the ns related admin commands since I was not able to reproduce the previous bug.

v4 Changes:
1. Add request polling interface to the block layer.
2. Use request polling interface in the NVMEoF target passthru code path.
3. Add checks suggested by Sagi for creating one target ctrl per passthru ctrl.
4. Don't enable the namespace if it belongs to the configured passthru ctrl.
5. Adjust the code latest kernel.

v3 Changes:
1. Split the addition of passthru command handlers and integration into two different patches since we add guards to create one target controller per passthru controller. This way it will be easier to review the code.
2. Adjust the code for 4.18.

v2 Changes:
1. Update the new nvme core controller find API naming and changed the string comparison of the ctrl.
2. Get rid of the newly added #defines for target ctrl values.
3. Use the newly added structure members in the same patch where they are used. Aggregate the passthru command handling support and integration with nvmet-core into one patch.
4. Introduce global NVMe Target subsystem list for connected and not connected subsystems on the target side.
5. Add check when configuring the target ns and target passthru ctrl to allow only one target controller to be created for one passthru subsystem.
6. Use the passthru ctrl cntlid when creating the target controller.

Chaitanya Kulkarni (1):
nvmet-passthru: Introduce NVMet passthru Kconfig option

Logan Gunthorpe (8):
nvme-core: Clear any SGL flags in passthru commands
nvme: Create helper function to obtain command effects
nvme: Move nvme_passthru_[start|end]() calls to common helper
nvme-core: Introduce nvme_ctrl_get_by_path()
nvme: Export existing nvme core functions
nvmet-passthru: Add passthru code to process commands
nvmet-passthru: Add enable/disable helpers
nvmet-configfs: Introduce passthru configfs interface

 drivers/nvme/host/core.c        | 251 +++++++++------
 drivers/nvme/host/nvme.h        |  11 +
 drivers/nvme/target/Kconfig     |  12 +
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/configfs.c  | 103 ++++++
 drivers/nvme/target/core.c      |  13 +-
 drivers/nvme/target/nvmet.h     |  52 +++
 drivers/nvme/target/passthru.c  | 543 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 10 files changed, 893 insertions(+), 104 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c

base-commit: 84c2c94f1adc59eaed24554666f755bcdd0140e8 --
2.20.1

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#1 lore raw

PATCH v15 1/9 nvme-core: Clear any SGL flags in passthru commands

The host driver should decide whether to use SGLs or PRPs and they currently assume the flags are cleared after the call to nvme_setup_cmd(). However, passed-through commands may erroneously set these bits; so clear them for all cases.

---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/core.c +2 -0
index f49085bcaa42..63a59186e68b 100644
@@ -759,6 +759,8 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
 	case REQ_OP_DRV_IN:
 	case REQ_OP_DRV_OUT:
 		memcpy(cmd, nvme_req(req)->cmd, sizeof(*cmd));
+		/* passthru commands should let the driver set the SGL flags */
+		cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 		break;
 	case REQ_OP_FLUSH:
 		nvme_setup_flush(ns, cmd);
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

Re: [PATCH v15 1/9] nvme-core: Clear any SGL flags in passthru commands

Quoted 16 lines · Logan Gunthorpe

The host driver should decide whether to use SGLs or PRPs and they currently assume the flags are cleared after the call to nvme_setup_cmd(). However, passed-through commands may erroneously set these bits; so clear them for all cases.

---
 drivers/nvme/host/core.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/host/core.c +0 -0
index f49085bcaa42..63a59186e68b 100644

drivers/nvme/host/core.c blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req, · line 764

Quoted 6 lines
drivers/nvme/host/core.c +2 -0
@@ -759,6 +759,8 @@ blk_status_t nvme_setup_cmd(struct nvme_ns *ns, struct request *req,
 	case REQ_OP_DRV_IN:
 	case REQ_OP_DRV_OUT:
 		memcpy(cmd, nvme_req(req)->cmd, sizeof(*cmd));
+		/* passthru commands should let the driver set the SGL flags */
+		cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 		break;

This probably wants to be split into a little helper like the other cases.

#3 lore raw

PATCH v15 2/9 nvme: Create helper function to obtain command effects

In reply to Logan Gunthorpe

Separate the code to obtain command effects from the code to start a passthru request and open code nvme_known_admin_effects() in the new helper.

The new helper function will be necessary for nvmet passthru code to determine if we need to change out of interrupt context to handle the effects.

---
 drivers/nvme/host/core.c | 39 ++++++++++++++++++++++-----------------
 1 file changed, 22 insertions(+), 17 deletions(-)
drivers/nvme/host/core.c +22 -17
index 63a59186e68b..80d534c5bc44 100644
@@ -1377,22 +1377,8 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_known_admin_effects(u8 opcode)
-{
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		return NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-					NVME_CMD_EFFECTS_CSE_MASK;
-	case nvme_admin_sanitize_nvm:
-		return NVME_CMD_EFFECTS_CSE_MASK;
-	default:
-		break;
-	}
-	return 0;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-								u8 opcode)
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
 {
 	u32 effects = 0;
 
@@ -1408,7 +1394,26 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 
 	if (ctrl->effects)
 		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-	effects |= nvme_known_admin_effects(opcode);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
 
 	/*
 	 * For simplicity, IO to all namespaces is quiesced even if the command
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#4 lore raw

Re: [PATCH v15 2/9] nvme: Create helper function to obtain command effects

Quoted 7 lines · Logan Gunthorpe

Separate the code to obtain command effects from the code to start a passthru request and open code nvme_known_admin_effects() in the new helper.

The new helper function will be necessary for nvmet passthru code to determine if we need to change out of interrupt context to handle the effects.

I actually found the nvme_known_admin_effects pretty nice, as the function name documents why it is there. The rest looks fine to me.

#5 lore raw

PATCH v15 3/9 nvme: Move nvme_passthru_[start|end]() calls to common helper

In reply to Logan Gunthorpe

Introduce a new nvme_execute_passthru_rq() helper which calls nvme_passthru_[start|end]() around blk_execute_rq(). This ensures all passthru calls (including nvme_submit_io()) will be wrapped appropriately.

nvme_execute_passthru_rq() will also be useful for the nvmet passthru code.

---
 drivers/nvme/host/core.c | 215 ++++++++++++++++++++-------------------
 1 file changed, 111 insertions(+), 104 deletions(-)
drivers/nvme/host/core.c +111 -104
index 80d534c5bc44..817ab76d2838 100644
@@ -911,6 +911,116 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
+static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+				u8 opcode)
+{
+	u32 effects = 0;
+
+	if (ns) {
+		if (ns->head->effects)
+			effects = le32_to_cpu(ns->head->effects->iocs[opcode]);
+		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
+			dev_warn(ctrl->device,
+				 "IO command:%02x has unhandled effects:%08x\n",
+				 opcode, effects);
+		return 0;
+	}
+
+	if (ctrl->effects)
+		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
+
+	switch (opcode) {
+	case nvme_admin_format_nvm:
+		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
+			NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	case nvme_admin_sanitize_nvm:
+		effects |= NVME_CMD_EFFECTS_CSE_MASK;
+		break;
+	default:
+		break;
+	}
+
+	return effects;
+}
+
+static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			       u8 opcode)
+{
+	u32 effects = nvme_command_effects(ctrl, ns, opcode);
+
+	/*
+	 * For simplicity, IO to all namespaces is quiesced even if the command
+	 * effects say only one namespace is affected.
+	 */
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		mutex_lock(&ctrl->scan_lock);
+		mutex_lock(&ctrl->subsys->lock);
+		nvme_mpath_start_freeze(ctrl->subsys);
+		nvme_mpath_wait_freeze(ctrl->subsys);
+		nvme_start_freeze(ctrl);
+		nvme_wait_freeze(ctrl);
+	}
+	return effects;
+}
+
+static void nvme_update_formats(struct nvme_ctrl *ctrl, u32 *effects)
+{
+	struct nvme_ns *ns;
+
+	down_read(&ctrl->namespaces_rwsem);
+	list_for_each_entry(ns, &ctrl->namespaces, list)
+		if (_nvme_revalidate_disk(ns->disk))
+			nvme_set_queue_dying(ns);
+		else if (blk_queue_is_zoned(ns->disk->queue)) {
+			/*
+			 * IO commands are required to fully revalidate a zoned
+			 * device. Force the command effects to trigger rescan
+			 * work so report zones can run in a context with
+			 * unfrozen IO queues.
+			 */
+			*effects |= NVME_CMD_EFFECTS_NCC;
+		}
+	up_read(&ctrl->namespaces_rwsem);
+}
+
+static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
+{
+	/*
+	 * Revalidate LBA changes prior to unfreezing. This is necessary to
+	 * prevent memory corruption if a logical block size was changed by
+	 * this command.
+	 */
+	if (effects & NVME_CMD_EFFECTS_LBCC)
+		nvme_update_formats(ctrl, &effects);
+	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
+		nvme_unfreeze(ctrl);
+		nvme_mpath_unfreeze(ctrl->subsys);
+		mutex_unlock(&ctrl->subsys->lock);
+		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
+		mutex_unlock(&ctrl->scan_lock);
+	}
+	if (effects & NVME_CMD_EFFECTS_CCC)
+		nvme_init_identify(ctrl);
+	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC)) {
+		nvme_queue_scan(ctrl);
+		flush_work(&ctrl->scan_work);
+	}
+}
+
+static void nvme_execute_passthru_rq(struct request *rq)
+{
+	struct nvme_command *cmd = nvme_req(rq)->cmd;
+	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
+	struct nvme_ns *ns = rq->q->queuedata;
+	struct gendisk *disk = ns ? ns->disk : NULL;
+	u32 effects;
+
+	effects = nvme_passthru_start(ctrl, ns, cmd->common.opcode);
+	blk_execute_rq(rq->q, disk, rq, 0);
+	nvme_passthru_end(ctrl, effects);
+}
+
 static int nvme_submit_user_cmd(struct request_queue *q,
 		struct nvme_command *cmd, void __user *ubuffer,
 		unsigned bufflen, void __user *meta_buffer, unsigned meta_len,
@@ -949,7 +1059,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 		}
 	}
 
-	blk_execute_rq(req->q, disk, req, 0);
+	nvme_execute_passthru_rq(req);
 	if (nvme_req(req)->flags & NVME_REQ_CANCELLED)
 		ret = -EINTR;
 	else
@@ -1377,110 +1487,12 @@ static int nvme_submit_io(struct nvme_ns *ns, struct nvme_user_io __user *uio)
 			metadata, meta_len, lower_32_bits(io.slba), NULL, 0);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
-{
-	u32 effects = 0;
-
-	if (ns) {
-		if (ns->head->effects)
-			effects = le32_to_cpu(ns->head->effects->iocs[opcode]);
-		if (effects & ~(NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC))
-			dev_warn(ctrl->device,
-				 "IO command:%02x has unhandled effects:%08x\n",
-				 opcode, effects);
-		return 0;
-	}
-
-	if (ctrl->effects)
-		effects = le32_to_cpu(ctrl->effects->acs[opcode]);
-
-	switch (opcode) {
-	case nvme_admin_format_nvm:
-		effects |= NVME_CMD_EFFECTS_CSUPP | NVME_CMD_EFFECTS_LBCC |
-			NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	case nvme_admin_sanitize_nvm:
-		effects |= NVME_CMD_EFFECTS_CSE_MASK;
-		break;
-	default:
-		break;
-	}
-
-	return effects;
-}
-
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-			       u8 opcode)
-{
-	u32 effects = nvme_command_effects(ctrl, ns, opcode);
-
-	/*
-	 * For simplicity, IO to all namespaces is quiesced even if the command
-	 * effects say only one namespace is affected.
-	 */
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		mutex_lock(&ctrl->scan_lock);
-		mutex_lock(&ctrl->subsys->lock);
-		nvme_mpath_start_freeze(ctrl->subsys);
-		nvme_mpath_wait_freeze(ctrl->subsys);
-		nvme_start_freeze(ctrl);
-		nvme_wait_freeze(ctrl);
-	}
-	return effects;
-}
-
-static void nvme_update_formats(struct nvme_ctrl *ctrl, u32 *effects)
-{
-	struct nvme_ns *ns;
-
-	down_read(&ctrl->namespaces_rwsem);
-	list_for_each_entry(ns, &ctrl->namespaces, list)
-		if (_nvme_revalidate_disk(ns->disk))
-			nvme_set_queue_dying(ns);
-		else if (blk_queue_is_zoned(ns->disk->queue)) {
-			/*
-			 * IO commands are required to fully revalidate a zoned
-			 * device. Force the command effects to trigger rescan
-			 * work so report zones can run in a context with
-			 * unfrozen IO queues.
-			 */
-			*effects |= NVME_CMD_EFFECTS_NCC;
-		}
-	up_read(&ctrl->namespaces_rwsem);
-}
-
-static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
-{
-	/*
-	 * Revalidate LBA changes prior to unfreezing. This is necessary to
-	 * prevent memory corruption if a logical block size was changed by
-	 * this command.
-	 */
-	if (effects & NVME_CMD_EFFECTS_LBCC)
-		nvme_update_formats(ctrl, &effects);
-	if (effects & (NVME_CMD_EFFECTS_LBCC | NVME_CMD_EFFECTS_CSE_MASK)) {
-		nvme_unfreeze(ctrl);
-		nvme_mpath_unfreeze(ctrl->subsys);
-		mutex_unlock(&ctrl->subsys->lock);
-		nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
-		mutex_unlock(&ctrl->scan_lock);
-	}
-	if (effects & NVME_CMD_EFFECTS_CCC)
-		nvme_init_identify(ctrl);
-	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC)) {
-		nvme_queue_scan(ctrl);
-		flush_work(&ctrl->scan_work);
-	}
-}
-
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
 {
 	struct nvme_passthru_cmd cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	u64 result;
 	int status;
 
@@ -1507,12 +1519,10 @@ static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(result, &ucmd->result))
@@ -1528,7 +1538,6 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	struct nvme_passthru_cmd64 cmd;
 	struct nvme_command c;
 	unsigned timeout = 0;
-	u32 effects;
 	int status;
 
 	if (!capable(CAP_SYS_ADMIN))
@@ -1554,12 +1563,10 @@ static int nvme_user_cmd64(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	if (cmd.timeout_ms)
 		timeout = msecs_to_jiffies(cmd.timeout_ms);
 
-	effects = nvme_passthru_start(ctrl, ns, cmd.opcode);
 	status = nvme_submit_user_cmd(ns ? ns->queue : ctrl->admin_q, &c,
 			nvme_to_user_ptr(cmd.addr), cmd.data_len,
 			nvme_to_user_ptr(cmd.metadata), cmd.metadata_len,
 			0, &cmd.result, timeout);
-	nvme_passthru_end(ctrl, effects);
 
 	if (status >= 0) {
 		if (put_user(cmd.result, &ucmd->result))
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer Reviewed-by on v15 3/9
#6 lore raw

Re: [PATCH v15 3/9] nvme: Move nvme_passthru_[start|end]() calls to common helper

Quoted 10 lines · Logan Gunthorpe

Introduce a new nvme_execute_passthru_rq() helper which calls nvme_passthru_[start|end]() around blk_execute_rq(). This ensures all passthru calls (including nvme_submit_io()) will be wrapped appropriately.

nvme_execute_passthru_rq() will also be useful for the nvmet passthru code.

Looks good,

although maybe you could have moved the code you changed in the last patch up already to not just move it after changing it in the last one.

#7 lore raw

PATCH v15 4/9 nvmet-passthru: Introduce NVMet passthru Kconfig option

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

This patch updates KConfig file for the NVMeOF target where we add new option so that user can selectively enable/disable passthru code.

[[email protected]: fixed some of the wording in the help message]
---
 drivers/nvme/target/Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)
drivers/nvme/target/Kconfig +12 -0
index 4474952d64c6..8056955e652c 100644
@@ -16,6 +16,18 @@ config NVME_TARGET
 	  To configure the NVMe target you probably want to use the nvmetcli
 	  tool from http://git.infradead.org/users/hch/nvmetcli.git.
 
+config NVME_TARGET_PASSTHRU
+	bool "NVMe Target Passthrough support"
+	depends on NVME_TARGET
+	depends on NVME_CORE=y || NVME_CORE=NVME_TARGET
+	help
+	  This enables target side NVMe passthru controller support for the
+	  NVMe Over Fabrics protocol. It allows for hosts to manage and
+	  directly access an actual NVMe controller residing on the target
+	  side, incuding executing Vendor Unique Commands.
+
+	  If unsure, say N.
+
 config NVME_TARGET_LOOP
 	tristate "NVMe loopback device support"
 	depends on NVME_TARGET
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#8 lore raw

Re: [PATCH v15 4/9] nvmet-passthru: Introduce NVMet passthru Kconfig option

Quoted 9 lines · Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

This patch updates KConfig file for the NVMeOF target where we add new option so that user can selectively enable/disable passthru code.

[[email protected]: fixed some of the wording in the help message]

The Kconfig should go last for split up patches.

#9 lore raw

PATCH v15 5/9 nvme-core: Introduce nvme_ctrl_get_by_path()

In reply to Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

---
 drivers/nvme/host/core.c | 31 +++++++++++++++++++++++++++++++
 drivers/nvme/host/nvme.h |  6 ++++++
 2 files changed, 37 insertions(+)
drivers/nvme/host/core.c +31 -0
index 817ab76d2838..6e7d5cb43a06 100644
@@ -4572,6 +4572,37 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl)
 }
 EXPORT_SYMBOL_GPL(nvme_sync_queues);
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+/*
+ * The exports that follow within this ifdef are only for
+ * use by the nvmet-passthru and should not be used for
+ * other things.
+ */
+
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+	struct nvme_ctrl *ctrl;
+	struct file *f;
+
+	f = filp_open(path, O_RDWR, 0);
+	if (IS_ERR(f))
+		return ERR_CAST(f);
+
+	if (f->f_op != &nvme_dev_fops) {
+		ctrl = ERR_PTR(-EINVAL);
+		goto out_close;
+	}
+
+	ctrl = f->private_data;
+	nvme_get_ctrl(ctrl);
+
+out_close:
+	filp_close(f, NULL);
+	return ctrl;
+}
+EXPORT_SYMBOL_GPL(nvme_ctrl_get_by_path);
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 /*
  * Check we didn't inadvertently grow the command structure sizes:
  */
drivers/nvme/host/nvme.h +6 -0
index 13ca90bcd352..485492a98ef5 100644
@@ -763,4 +763,10 @@ void nvme_hwmon_init(struct nvme_ctrl *ctrl);
 static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
 #endif
 
+/*
+ * These functions are only for use by nvmet-passthru and are only exported
+ * if CONFIG_NVME_TARGET_PASSTHRU is set.
+ */
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
+
 #endif /* _NVME_H */
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#10 lore raw

Re: [PATCH v15 5/9] nvme-core: Introduce nvme_ctrl_get_by_path()

Quoted 7 lines · Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

This looks generally good. However since the early days of this patchset we've grown symbol namespaces, and the EXPORT_SYMBOL_NS_GPL macro, which are what we really should use here and for the other exports.

#11 lore raw

PATCH v15 6/9 nvme: Export existing nvme core functions

In reply to Logan Gunthorpe

Export nvme_put_ns(), nvme_command_effects(), nvme_execute_passthru_rq() and nvme_find_get_ns() for use in the nvmet passthru code.

The exports are conditional on CONFIG_NVME_TARGET_PASSTHRU.

---
 drivers/nvme/host/core.c | 14 +++++++++-----
 drivers/nvme/host/nvme.h |  5 +++++
 2 files changed, 14 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +9 -5
index 6e7d5cb43a06..2adddc69467c 100644
@@ -454,7 +454,7 @@ static void nvme_free_ns(struct kref *kref)
 	kfree(ns);
 }
 
-static void nvme_put_ns(struct nvme_ns *ns)
+void nvme_put_ns(struct nvme_ns *ns)
 {
 	kref_put(&ns->kref, nvme_free_ns);
 }
@@ -911,8 +911,8 @@ static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 	return ERR_PTR(ret);
 }
 
-static u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-				u8 opcode)
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode)
 {
 	u32 effects = 0;
 
@@ -1008,7 +1008,7 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 	}
 }
 
-static void nvme_execute_passthru_rq(struct request *rq)
+void nvme_execute_passthru_rq(struct request *rq)
 {
 	struct nvme_command *cmd = nvme_req(rq)->cmd;
 	struct nvme_ctrl *ctrl = nvme_req(rq)->ctrl;
@@ -3812,7 +3812,7 @@ static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
-static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 {
 	struct nvme_ns *ns, *ret = NULL;
 
@@ -4578,6 +4578,10 @@ EXPORT_SYMBOL_GPL(nvme_sync_queues);
  * use by the nvmet-passthru and should not be used for
  * other things.
  */
+EXPORT_SYMBOL_GPL(nvme_put_ns);
+EXPORT_SYMBOL_GPL(nvme_command_effects);
+EXPORT_SYMBOL_GPL(nvme_execute_passthru_rq);
+EXPORT_SYMBOL_GPL(nvme_find_get_ns);
 
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
 {
drivers/nvme/host/nvme.h +5 -0
index 485492a98ef5..87fb9e22c79d 100644
@@ -767,6 +767,11 @@ static inline void nvme_hwmon_init(struct nvme_ctrl *ctrl) { }
  * These functions are only for use by nvmet-passthru and are only exported
  * if CONFIG_NVME_TARGET_PASSTHRU is set.
  */
+void nvme_put_ns(struct nvme_ns *ns);
+u32 nvme_command_effects(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+			 u8 opcode);
+void nvme_execute_passthru_rq(struct request *rq);
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned int nsid);
 struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
 
 #endif /* _NVME_H */
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#12 lore raw

PATCH v15 7/9 nvmet-passthru: Add passthru code to process commands

In reply to Logan Gunthorpe

Add passthru command handling capability for the NVMeOF target and export passthru APIs which are used to integrate passthru code with nvmet-core.

The new file passthru.c handles passthru cmd parsing and execution. In the passthru mode, we create a block layer request from the nvmet request and map the data on to the block layer request.

Admin commands and features are on a white list as there are a number of each that don't make too much sense with passthrough. We use a white list so that new commands can be considered before being blindly passed through. In both cases, vendor specific commands are always allowed.

We also blacklist reservation IO commands as the underlying device cannot differentiate between multiple hosts behind a fabric.

---
 drivers/nvme/target/Makefile    |   1 +
 drivers/nvme/target/admin-cmd.c |   7 +-
 drivers/nvme/target/core.c      |   3 +
 drivers/nvme/target/nvmet.h     |  39 +++
 drivers/nvme/target/passthru.c  | 457 ++++++++++++++++++++++++++++++++
 include/linux/nvme.h            |   4 +
 6 files changed, 509 insertions(+), 2 deletions(-)
 create mode 100644 drivers/nvme/target/passthru.c
drivers/nvme/target/Makefile +1 -0
index 2b33836f3d3e..ebf91fc4c72e 100644
@@ -11,6 +11,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
+nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= passthru.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
drivers/nvme/target/admin-cmd.c +5 -2
index 95bb3bc4e335..e8d134535db5 100644
@@ -754,7 +754,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)
 	return 0;
 }
 
-static void nvmet_execute_set_features(struct nvmet_req *req)
+void nvmet_execute_set_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -829,7 +829,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
 	nvmet_set_result(req, READ_ONCE(req->sq->ctrl->aen_enabled));
 }
 
-static void nvmet_execute_get_features(struct nvmet_req *req)
+void nvmet_execute_get_features(struct nvmet_req *req)
 {
 	struct nvmet_subsys *subsys = req->sq->ctrl->subsys;
 	u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
@@ -945,6 +945,9 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_admin_cmd(req);
+
 	switch (cmd->common.opcode) {
 	case nvme_admin_get_log_page:
 		req->execute = nvmet_execute_get_log_page;
drivers/nvme/target/core.c +3 -0
index 9cdc39c8b729..86fc913f481a 100644
@@ -874,6 +874,9 @@ static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
 	if (unlikely(ret))
 		return ret;
 
+	if (nvmet_req_passthru_ctrl(req))
+		return nvmet_parse_passthru_io_cmd(req);
+
 	req->ns = nvmet_find_namespace(req->sq->ctrl, cmd->rw.nsid);
 	if (unlikely(!req->ns)) {
 		req->error_loc = offsetof(struct nvme_common_command, nsid);
drivers/nvme/target/nvmet.h +39 -0
index 6f8bd6a93575..03a92dba24c3 100644
@@ -243,6 +243,10 @@ struct nvmet_subsys {
 	struct config_group	allowed_hosts_group;
 
 	struct nvmet_subsys_model	__rcu *model;
+
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+	struct nvme_ctrl	*passthru_ctrl;
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
 static inline struct nvmet_subsys *to_subsys(struct config_item *item)
@@ -322,6 +326,11 @@ struct nvmet_req {
 			struct bio_vec          *bvec;
 			struct work_struct      work;
 		} f;
+		struct {
+			struct request		*rq;
+			struct work_struct      work;
+			u16 (*end_req)(struct nvmet_req *req);
+		} p;
 	};
 	int			sg_cnt;
 	int			metadata_sg_cnt;
@@ -401,6 +410,8 @@ void nvmet_req_complete(struct nvmet_req *req, u16 status);
 int nvmet_req_alloc_sgls(struct nvmet_req *req);
 void nvmet_req_free_sgls(struct nvmet_req *req);
 
+void nvmet_execute_set_features(struct nvmet_req *req);
+void nvmet_execute_get_features(struct nvmet_req *req);
 void nvmet_execute_keep_alive(struct nvmet_req *req);
 
 void nvmet_cq_setup(struct nvmet_ctrl *ctrl, struct nvmet_cq *cq, u16 qid,
@@ -533,6 +544,34 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 		sizeof(struct nvme_dsm_range);
 }
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
+u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return subsys->passthru_ctrl;
+}
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req)
+{
+	return 0;
+}
+static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
+{
+	return NULL;
+}
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static inline struct nvme_ctrl *
+nvmet_req_passthru_ctrl(struct nvmet_req *req)
+{
+	return nvmet_passthru_ctrl(req->sq->ctrl->subsys);
+}
+
 u16 errno_to_nvme_status(struct nvmet_req *req, int errno);
 
 /* Convert a 32-bit number to a 16-bit 0's based number */
drivers/nvme/target/passthru.c new file +457 -0

457 lines not on this page · raw message

include/linux/nvme.h +4 -0
index 1643005d21e3..d92535997687 100644
@@ -312,6 +312,7 @@ enum {
 	NVME_CTRL_ONCS_WRITE_UNCORRECTABLE	= 1 << 1,
 	NVME_CTRL_ONCS_DSM			= 1 << 2,
 	NVME_CTRL_ONCS_WRITE_ZEROES		= 1 << 3,
+	NVME_CTRL_ONCS_RESERVATIONS		= 1 << 5,
 	NVME_CTRL_ONCS_TIMESTAMP		= 1 << 6,
 	NVME_CTRL_VWC_PRESENT			= 1 << 0,
 	NVME_CTRL_OACS_SEC_SUPP                 = 1 << 0,
@@ -982,6 +983,7 @@ enum nvme_admin_opcode {
 	nvme_admin_security_recv	= 0x82,
 	nvme_admin_sanitize_nvm		= 0x84,
 	nvme_admin_get_lba_status	= 0x86,
+	nvme_admin_vendor_start		= 0xC0,
 };
 
 #define nvme_admin_opcode_name(opcode)	{ opcode, #opcode }
@@ -1045,6 +1047,8 @@ enum {
 	NVME_FEAT_RESV_MASK	= 0x82,
 	NVME_FEAT_RESV_PERSIST	= 0x83,
 	NVME_FEAT_WRITE_PROTECT	= 0x84,
+	NVME_FEAT_VENDOR_START	= 0xC0,
+	NVME_FEAT_VENDOR_END	= 0xFF,
 	NVME_LOG_ERROR		= 0x01,
 	NVME_LOG_SMART		= 0x02,
 	NVME_LOG_FW_SLOT	= 0x03,
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#13 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 8 lines · Logan Gunthorpe

Add passthru command handling capability for the NVMeOF target and export passthru APIs which are used to integrate passthru code with nvmet-core.

The new file passthru.c handles passthru cmd parsing and execution. In the passthru mode, we create a block layer request from the nvmet request and map the data on to the block layer request.

Quoted 8 lines

Admin commands and features are on a white list as there are a number of each that don't make too much sense with passthrough. We use a white list so that new commands can be considered before being blindly passed through. In both cases, vendor specific commands are always allowed.

We also blacklist reservation IO commands as the underlying device cannot differentiate between multiple hosts behind a fabric.

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

A few more comments below:

drivers/nvme/target/nvmet.h struct nvmet_req { · line 333

Quoted 5 lines
drivers/nvme/target/nvmet.h +5 -0
+		struct {
+			struct request		*rq;
+			struct work_struct      work;
+			u16 (*end_req)(struct nvmet_req *req);
+		} p;

Do we really need the callback for the two identify fixups, or should we just hardcode them to avoid indirection function calls?

drivers/nvme/target/passthru.c line 7

Quoted 9 lines

+++ b/drivers/nvme/target/passthru.c

drivers/nvme/target/passthru.c new file +7 -0
@@ -0,0 +1,457 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * NVMe Over Fabrics Target Passthrough command implementation.
+ *
+ * Copyright (c) 2017-2018 Western Digital Corporation or its
+ * affiliates.
+ */

I think you forgot to add your own copyrights here.

Quoted 4 lines
drivers/nvme/target/passthru.c new file +4 -0
+static int nvmet_passthru_map_sg(struct nvmet_req *req, struct request *rq)
+{
+	int sg_cnt = req->sg_cnt;
+	struct scatterlist *sg;

drivers/nvme/target/passthru.c line 54

Quoted 10 lines
drivers/nvme/target/passthru.c new file +10 -0
+	int op_flags = 0;
+	struct bio *bio;
+	int i, ret;
+
+	if (req->cmd->common.opcode == nvme_cmd_flush)
+		op_flags = REQ_FUA;
+	else if (nvme_is_write(req->cmd))
+		op_flags = REQ_SYNC | REQ_IDLE;
+
+

Double empty line here..

drivers/nvme/target/passthru.c line 60

Quoted 6 lines
drivers/nvme/target/passthru.c new file +6 -0
+	bio = bio_alloc(GFP_KERNEL, min(sg_cnt, BIO_MAX_PAGES));
+	bio->bi_end_io = bio_put;
+
+	for_each_sg(req->sg, sg, req->sg_cnt, i) {
+		if (bio_add_page(bio, sg_page(sg), sg->length,
+				 sg->offset) != sg->length) {

bio_add_pages is only for non-passthrough requests, this needs to use bio_add_pc_page.

drivers/nvme/target/passthru.c line 125

Quoted 9 lines
drivers/nvme/target/passthru.c new file +9 -0
+	if (blk_rq_nr_phys_segments(rq) > queue_max_segments(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}
+
+	if ((blk_rq_payload_bytes(rq) >> 9) > queue_max_hw_sectors(rq->q)) {
+		status = NVME_SC_INVALID_FIELD;
+		goto fail_out;
+	}

Which should also take care of these checks.

drivers/nvme/target/passthru.c line 294

Quoted 10 lines
drivers/nvme/target/passthru.c new file +10 -0
+static void nvmet_passthru_set_host_behaviour(struct nvmet_req *req)
+{
+	struct nvme_ctrl *ctrl = nvmet_req_passthru_ctrl(req);
+	struct nvme_feat_host_behavior *host;
+	u16 status;
+	int ret;
+
+	host = kzalloc(sizeof(*host) * 2, GFP_KERNEL);
+	ret = nvme_get_features(ctrl, NVME_FEAT_HOST_BEHAVIOR, 0,
+				host, sizeof(*host), NULL);

Mising kzalloc return check.

maintainer
#14 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 15 lines
Quoted 15 lines

Add passthru command handling capability for the NVMeOF target and export passthru APIs which are used to integrate passthru code with nvmet-core.

The new file passthru.c handles passthru cmd parsing and execution. In the passthru mode, we create a block layer request from the nvmet request and map the data on to the block layer request.

Admin commands and features are on a white list as there are a number of each that don't make too much sense with passthrough. We use a white list so that new commands can be considered before being blindly passed through. In both cases, vendor specific commands are always allowed.

Quoted 6 lines
Quoted 2 lines

We also blacklist reservation IO commands as the underlying device cannot differentiate between multiple hosts behind a fabric.

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

#15 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

Logan

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#16 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 4 lines · Logan Gunthorpe
Quoted 4 lines

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

Quoted 8 lines

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

I didn't understand it either how having a passthru q can get rid of the struct nvme_ns search when nsid is embedded in the nvme_cmd in the I/O path ?

Sagi can you please explain ?

maintainer
#17 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

In reply to Logan Gunthorpe
Quoted 15 lines

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

Quoted 8 lines

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

The only part is to check the effects, but that can probably be handled when we setup the passthru controller or something...

#18 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 21 lines · Sagi Grimberg
Quoted 21 lines

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Quoted 7 lines
Quoted 2 lines

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

Thanks, that helps clarify things a bit, but which xarray were you talking about?

Logan

maintainer
#19 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q. But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

Quoted 7 lines
Quoted 4 lines

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

Thanks, that helps clarify things a bit, but which xarray were you talking about?

The patch from Chaitanya

See "[PATCH V4 2/2] nvme-core: use xarray for ctrl ns tracking"

maintainer
#20 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

In reply to Logan Gunthorpe
Quoted 6 lines · Logan Gunthorpe
Quoted 4 lines · Sagi Grimberg

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

We still need a request_queue to dispatch the command. I guess you could make a generic one for the controller that isn't tied to a namespace, but we lose the fair shared tag allocation.

Quoted 2 lines

Thanks, that helps clarify things a bit, but which xarray were you talking about

That was something that was replacing the list lookup:

http://lists.infradead.org/pipermail/linux-nvme/2020-July/018242.html

maintainer
#21 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 11 lines · Keith Busch
Quoted 6 lines · Logan Gunthorpe
Quoted 4 lines · Sagi Grimberg

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

We still need a request_queue to dispatch the command. I guess you could make a generic one for the controller that isn't tied to a namespace, but we lose the fair shared tag allocation.

What do you mean fair shared tag allocation?

maintainer
#22 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 9 lines · Sagi Grimberg
Quoted 8 lines · Keith Busch
Quoted 6 lines · Logan Gunthorpe
Quoted 4 lines · Sagi Grimberg

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

Quoted 5 lines
Quoted 3 lines

We still need a request_queue to dispatch the command. I guess you could make a generic one for the controller that isn't tied to a namespace, but we lose the fair shared tag allocation.

What do you mean fair shared tag allocation?

See hctx_may_queue().

maintainer
#23 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines · Logan Gunthorpe
Quoted 6 lines · Sagi Grimberg
Quoted 5 lines

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

We still need a request_queue to dispatch the command. I guess you could make a generic one for the controller that isn't tied to a namespace, but we lose the fair shared tag allocation.

What do you mean fair shared tag allocation?

See hctx_may_queue().

Still not following your point... this queue is yet another request queue on the I/O tagset, e.g.

ctrl->passthru_q = blk_mq_init_queue(ctrl->tagset);

#24 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

In reply to Sagi Grimberg
Quoted 24 lines · Sagi Grimberg
Quoted 15 lines

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q.  But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

Quoted 8 lines

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

Quoted 7 lines

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

The only part is to check the effects, but that can probably be handled when we setup the passthru controller or something...

Yes, I implemented the passthru_q (which was quite simple). But I'm not sure how we are supposed to call nvme_command_effects() correctly without the ns. You can't possibly do that during setup for every possible opcode on every namespace. And even if we do, we'll still need the same nvme_find_get_ns() and nvme_put_ns() exports and probably another xarray to lookup the information.

Also, we pass the namespace's disk to in order to get proper block accounting for the underlying disk. (Which is pretty important for debugging). So we need to lookup the namespace for this too.

Unless there are some other ideas to solve these issues, I don't think this change will gain us anything.

Logan

maintainer
#25 lore raw

Re: [PATCH v15 7/9] nvmet-passthru: Add passthru code to process commands

Quoted 29 lines
Quoted 29 lines
Quoted 15 lines

Thanks for the review Christoph. I think I should be able to make all the requested changes in the next week or two.

Quoted 9 lines · Sagi Grimberg
Quoted 4 lines

I'm still not so happy about having to look up the namespace and still wonder if we should generalize the connect_q to a passthrough_q.  But I guess we can do that later and then reduce some of the exports here..

That is a neat idea! should be easy to do (and we can then lose the host xarray stuff). I don't mind having it on a later patch, but it should be easy enough to do even before...

Quoted 8 lines

I sort of follow this. I can try to work something up but it will probably take me a few iterations to get it to where you want it. So, roughly, we'd create a passthrough_q in core with the controller's IO tagset and then cleanup the fabrics hosts to use that instead of each independently creating their connect_q?

Though, I don't understand how this relates to the host xarray stuff that Sagi mentioned...

passthru commands are in essence REQ_OP_DRV_IN/REQ_OP_DRV_OUT, which means that the driver shouldn't need the ns at all. So if you have a dedicated request queue (mapped to the I/O tagset), you don't need the ns->queue and we can lose the ns lookup altogether.

Quoted 4 lines
Quoted 2 lines

The only part is to check the effects, but that can probably be handled when we setup the passthru controller or something...

Yes, I implemented the passthru_q (which was quite simple).

Nice..

Quoted 7 lines

But I'm not
sure how we are supposed to call nvme_command_effects() correctly without the ns. You can't possibly do that during setup for every possible opcode on every namespace. And even if we do, we'll still need the same nvme_find_get_ns() and nvme_put_ns() exports and probably another xarray to lookup the information.

Quoted 6 lines

Also, we pass the namespace's disk to in order to get proper block accounting for the underlying disk. (Which is pretty important for debugging). So we need to lookup the namespace for this too.

Unless there are some other ideas to solve these issues, I don't think this change will gain us anything.

Let's defer it to a followup set than.

#26 lore raw

PATCH v15 8/9 nvmet-passthru: Add enable/disable helpers

In reply to Logan Gunthorpe

This patch adds helper functions which are used in the NVMeOF configfs when the user is configuring the passthru subsystem. Here we ensure that only one subsys is assigned to each nvme_ctrl by using an xarray on the cntlid.

The subsystem's version number is overridden by the passed through controller's version. However, if that version is less than 1.2.1, then we bump the advertised version to that and print a warning in dmesg.

---
 drivers/nvme/target/configfs.c |  4 ++
 drivers/nvme/target/core.c     | 10 +++-
 drivers/nvme/target/nvmet.h    | 12 +++++
 drivers/nvme/target/passthru.c | 86 ++++++++++++++++++++++++++++++++++
 4 files changed, 111 insertions(+), 1 deletion(-)
drivers/nvme/target/configfs.c +4 -0
index cdec47de89ed..61c258dea88a 100644
@@ -879,6 +879,10 @@ static ssize_t nvmet_subsys_attr_version_store(struct config_item *item,
 	int major, minor, tertiary = 0;
 	int ret;
 
+	/* passthru subsystems use the underlying controller's version */
+	if (nvmet_passthru_ctrl(subsys))
+		return -EINVAL;
+
 	ret = sscanf(page, "%d.%d.%d\n", &major, &minor, &tertiary);
 	if (ret != 2 && ret != 3)
 		return -EINVAL;
drivers/nvme/target/core.c +9 -1
index 86fc913f481a..c0c7a0d75a01 100644
@@ -558,6 +558,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 
 	mutex_lock(&subsys->lock);
 	ret = 0;
+
+	if (nvmet_passthru_ctrl(subsys)) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
 	if (ns->enabled)
 		goto out_unlock;
 
@@ -1498,7 +1504,7 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
 	if (!subsys)
 		return ERR_PTR(-ENOMEM);
 
-	subsys->ver = NVME_VS(1, 3, 0); /* NVMe 1.3.0 */
+	subsys->ver = NVMET_DEFAULT_VS;
 	/* generate a random serial number as our controllers are ephemeral: */
 	get_random_bytes(&subsys->serial, sizeof(subsys->serial));
 
@@ -1540,6 +1546,8 @@ static void nvmet_subsys_free(struct kref *ref)
 
 	WARN_ON_ONCE(!list_empty(&subsys->namespaces));
 
+	nvmet_passthru_subsys_free(subsys);
+
 	kfree(subsys->subsysnqn);
 	kfree_rcu(subsys->model, rcuhead);
 	kfree(subsys);
drivers/nvme/target/nvmet.h +12 -0
index 03a92dba24c3..d8689744e0f7 100644
@@ -21,6 +21,8 @@
 #include <linux/radix-tree.h>
 #include <linux/t10-pi.h>
 
+#define NVMET_DEFAULT_VS		NVME_VS(1, 3, 0)
+
 #define NVMET_ASYNC_EVENTS		4
 #define NVMET_ERROR_LOG_SLOTS		128
 #define NVMET_NO_ERROR_LOC		((u16)-1)
@@ -246,6 +248,7 @@ struct nvmet_subsys {
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
+	char			*passthru_ctrl_path;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
@@ -545,6 +548,9 @@ static inline u32 nvmet_dsm_len(struct nvmet_req *req)
 }
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
 u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req);
 u16 nvmet_parse_passthru_io_cmd(struct nvmet_req *req);
 static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
@@ -552,6 +558,12 @@ static inline struct nvme_ctrl *nvmet_passthru_ctrl(struct nvmet_subsys *subsys)
 	return subsys->passthru_ctrl;
 }
 #else /* CONFIG_NVME_TARGET_PASSTHRU */
+static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+}
+static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+}
 static inline u16 nvmet_parse_passthru_admin_cmd(struct nvmet_req *req)
 {
 	return 0;
drivers/nvme/target/passthru.c +86 -0

95 lines not on this page · raw message

Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#27 lore raw

PATCH v15 9/9 nvmet-configfs: Introduce passthru configfs interface

In reply to Logan Gunthorpe

When CONFIG_NVME_TARGET_PASSTHRU as 'passthru' directory will be added to each subsystem. The directory is similar to a namespace and has two attributes: device_path and enable. The user must set the path to the nvme controller's char device and write '1' to enable the subsystem to use passthru.

Any given subsystem is prevented from enabling both a regular namespace and the passthru device. If one is enabled, enabling the other will produce an error.

---
 drivers/nvme/target/configfs.c | 99 ++++++++++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h    |  1 +
 2 files changed, 100 insertions(+)
drivers/nvme/target/configfs.c +99 -0
index 61c258dea88a..74b2b61c773b 100644
@@ -666,6 +666,103 @@ static const struct config_item_type nvmet_namespaces_type = {
 	.ct_owner		= THIS_MODULE,
 };
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+
+static ssize_t nvmet_passthru_device_path_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->passthru_ctrl_path);
+}
+
+static ssize_t nvmet_passthru_device_path_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	size_t len;
+	int ret;
+
+	mutex_lock(&subsys->lock);
+
+	ret = -EBUSY;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	ret = -EINVAL;
+	len = strcspn(page, "\n");
+	if (!len)
+		goto out_unlock;
+
+	kfree(subsys->passthru_ctrl_path);
+	ret = -ENOMEM;
+	subsys->passthru_ctrl_path = kstrndup(page, len, GFP_KERNEL);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+
+	mutex_unlock(&subsys->lock);
+
+	return count;
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+CONFIGFS_ATTR(nvmet_passthru_, device_path);
+
+static ssize_t nvmet_passthru_enable_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return sprintf(page, "%d\n", subsys->passthru_ctrl ? 1 : 0);
+}
+
+static ssize_t nvmet_passthru_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	bool enable;
+	int ret = 0;
+
+	if (strtobool(page, &enable))
+		return -EINVAL;
+
+	if (enable)
+		ret = nvmet_passthru_ctrl_enable(subsys);
+	else
+		nvmet_passthru_ctrl_disable(subsys);
+
+	return ret ? ret : count;
+}
+CONFIGFS_ATTR(nvmet_passthru_, enable);
+
+static struct configfs_attribute *nvmet_passthru_attrs[] = {
+	&nvmet_passthru_attr_device_path,
+	&nvmet_passthru_attr_enable,
+	NULL,
+};
+
+static const struct config_item_type nvmet_passthru_type = {
+	.ct_attrs		= nvmet_passthru_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+	config_group_init_type_name(&subsys->passthru_group,
+				    "passthru", &nvmet_passthru_type);
+	configfs_add_default_group(&subsys->passthru_group,
+				   &subsys->group);
+}
+
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+}
+
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 static int nvmet_port_subsys_allow_link(struct config_item *parent,
 		struct config_item *target)
 {
@@ -1125,6 +1222,8 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	configfs_add_default_group(&subsys->allowed_hosts_group,
 			&subsys->group);
 
+	nvmet_add_passthru_group(subsys);
+
 	return &subsys->group;
 }
 
drivers/nvme/target/nvmet.h +1 -0
index d8689744e0f7..deb3c52aae6d 100644
@@ -249,6 +249,7 @@ struct nvmet_subsys {
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
 	char			*passthru_ctrl_path;
+	struct config_group	passthru_group;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
Signature · 7 lines
2.20.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme