Skip to contentSkip to conversation

nvmet: add target passthru commands support

PATCH Superseded Logan Gunthorpe 4 participants 34 messages Aug 1, 2019 – Aug 22, 2019

Subsystems (4)

v7 of 16 Superseded Latest is v16 →

lore
Share & tools
lore
#0 lore raw

v7 cover nvmet: add target passthru commands support

Hi,

This is v7 of the passthru patchset. This largely addresses the feedback from v6: cleaning up the nvme_ctrl_get_by_path() and allowing multipath over fabrics to the passthru target.

Multipath is now supported simply by allowing multiple connections from the same hostnqn and overriding the appropriate cmic bit. Passing through to multiport devices as a target is still not supported.

--

Chaitainya has asked us to take on these patches as we have an interest in getting them into upstream. To that end, we've done a large amount of testing, bug fixes and cleanup.

Passthru support for nvmet allows users to export an entire NVMe controller through NVMe-oF. When exported in this way (as opposed to exporting each namespace as a block device), all the NVMe commands are passed to the given controller unmodified, including most admin commands and Vendor Unique Commands (VUCs). A passthru target will expose all namespaces for a given device to the remote host.

There are three major non-bugfix changes that we've done to the series:

1) Instead of using a seperate special passthru subsystem in configfs simply add a passthru directory that's analogous to the existing namespace directories. The directories have very similar attributes to namespaces but are mutually exclusive. If a user enables a namespaces, they can't then enable passthru controller and vice versa. This simplifies the code required to implement passthru configfs and IMO creates a much clearer and uniform interface.

2) Instead of taking a bare controller name (ie. "nvme1"), take a full device path to the controller's char device. This is more consistent with the regular namespaces which take a path and also allows users to make use of udev rules and symlinks to manage their controllers instead of the potentially unstable device names.

3) Implement block accounting for the passthru devices. This is so the target OS can still track device usage using /proc/diskstats.

Besides these three changes, we've also found a large number of bugs and crashes and did a bunch of testing with KASAN, lockdep and kmemleak. A more complete list of changes is given below.

Additionally, we've written some new blktests to test the passthru code. A branch is available here[1] and can be submitted once these patches are upstream.

These patches are based off of v5.3-rc1 and a git branch is available at [2].

Thanks,

Logan

[1] https://github.com/Eideticom/blktests nvmet_passthru
[2] https://github.com/sbates130272/linux-p2pmem/ nvmet_passthru_v6

--

v7 Changes:
1. Rebased onto v5.3-rc2
2. Rework nvme_ctrl_get_by_path() to use filp_open() instead of the cdev changes that were in v6. (Per Al)
3. Override the cmic bit to allow multipath and allow multiple connections from the same hostnqn. (At the same time I cleaned up the method of rejecting multiple connections.) See Patch 8)
4. Found a bug when used with the tcp transport (See Patch 10)

v6 Changes:
1. Rebased onto v5.3-rc1
2. Rework configfs interface to simply be a passthru directory within the existing subsystem. The directory is similar to and consistent with a namespace directory.
3. Have the configfs take a path instead of a bare controller name
4. Renamed the main passthru file to io-cmd-passthru.c for consistency with the file and block-dev methods.
5. Cleaned up all the CONFIG_NVME_TARGET_PASSTHRU usage to remove all the inline #ifdefs
6. Restructured nvmet_passthru_make_request() a bit for clearer code
7. Moved nvme_find_get_ns() call into nvmet_passthru_execute_cmd() seeing calling it in nvmet_req_init() causes a lockdep warning due to nvme_find_get_ns() being able to sleep.
8. Added a check in nvmet_passthru_execute_cmd() to ensure we don't violate queue_max_segments or queue_max_hw_sectors and overrode mdts to ensure hosts don't intentionally submit commands that will exceed these limits.
9. Reworked the code which ensures there's only one subsystem per passthru controller to use an xarray instead of a list as this is simpler and more easily fixed some bugs triggered by disabling subsystems that weren't enabled.
10. Removed the overide of the target cntlid with the passthru cntlid; this seemed like a really bad idea especially in the presence of mixed systems as you could end up with two ctrlrs with the same cntlid. For now, commands that depend on cntlid are black listed.
11. Implement block accounting for passthru so the target can track usage using /proc/diskstats
12. A number of other minor bug fixes and cleanups

v5 Changes (not sent to list, from Chaitanya):
1. Added workqueue for admin commands.
2. Added kconfig option for the pass-thru feature.
3. Restructure the parsing code according to your suggestion, call nvmet_xxx_parse_cmd() from nvmet_passthru_parse_cmd().
4. Use pass-thru instead of pt.
5. Several cleanups and add comments at the appropriate locations.
6. Minimize the code for checking pass-thru ns across all the subsystems.
7. Removed the delays in the ns related admin commands since I was not able to reproduce the previous bug.

v4 Changes:
1. Add request polling interface to the block layer.
2. Use request polling interface in the NVMEoF target passthru code path.
3. Add checks suggested by Sagi for creating one target ctrl per passthru ctrl.
4. Don't enable the namespace if it belongs to the configured passthru ctrl.
5. Adjust the code latest kernel.

v3 Changes:
1. Split the addition of passthru command handlers and integration into two different patches since we add guards to create one target controller per passthru controller. This way it will be easier to review the code.
2. Adjust the code for 4.18.

v2 Changes:
1. Update the new nvme core controller find API naming and changed the string comparison of the ctrl.
2. Get rid of the newly added #defines for target ctrl values.
3. Use the newly added structure members in the same patch where they are used. Aggregate the passthru command handling support and integration with nvmet-core into one patch.
4. Introduce global NVMe Target subsystem list for connected and not connected subsystems on the target side.
5. Add check when configuring the target ns and target passthru ctrl to allow only one target controller to be created for one passthru subsystem.
6. Use the passthru ctrl cntlid when creating the target controller.

--

Chaitanya Kulkarni (5):
nvme-core: export existing ctrl and ns interfaces
nvmet: add return value to nvmet_add_async_event()
nvmet-passthru: update KConfig with config passthru option
nvmet-passthru: add passthru code to process commands
nvmet-core: don't check the data len for pt-ctrl

Logan Gunthorpe (9):
nvme-core: introduce nvme_ctrl_get_by_path()
nvmet: make nvmet_copy_ns_identifier() non-static
nvmet-passthru: add enable/disable helpers
nvmet-core: allow one host per passthru-ctrl
nvmet-tcp: don't check data_len in nvmet_tcp_map_data()
nvmet-configfs: introduce passthru configfs interface
block: don't check blk_rq_is_passthrough() in blk_do_io_stat()
block: call blk_account_io_start() in blk_execute_rq_nowait()
nvmet-passthru: support block accounting

 block/blk-exec.c                      |   2 +
 block/blk-mq.c                        |   2 +-
 block/blk.h                           |   5 +-
 drivers/nvme/host/core.c              |  41 +-
 drivers/nvme/host/nvme.h              |   9 +
 drivers/nvme/target/Kconfig           |  10 +
 drivers/nvme/target/Makefile          |   1 +
 drivers/nvme/target/admin-cmd.c       |   4 +-
 drivers/nvme/target/configfs.c        |  99 ++++
 drivers/nvme/target/core.c            |  39 +-
 drivers/nvme/target/io-cmd-passthru.c | 683 ++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h           |  68 ++-
 drivers/nvme/target/tcp.c             |   2 +-
 13 files changed, 947 insertions(+), 18 deletions(-)
 create mode 100644 drivers/nvme/target/io-cmd-passthru.c

--
2.20.1

#1 lore raw

PATCH v7 1/14 nvme-core: introduce nvme_ctrl_get_by_path()

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/host/core.c | 24 ++++++++++++++++++++++++
 drivers/nvme/host/nvme.h |  2 ++
 2 files changed, 26 insertions(+)
drivers/nvme/host/core.c +24 -0
index e6ee6f2a3da6..f72334f34a30 100644
@@ -2817,6 +2817,30 @@ static const struct file_operations nvme_dev_fops = {
 	.compat_ioctl	= nvme_dev_ioctl,
 };
 
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+	struct nvme_ctrl *ctrl;
+	struct file *f;
+
+	f = filp_open(path, O_RDWR, 0);
+	if (IS_ERR(f))
+		return ERR_CAST(f);
+
+	if (f->f_op != &nvme_dev_fops) {
+		ctrl = ERR_PTR(-EINVAL);
+		goto out_close;
+	}
+
+	ctrl = f->private_data;
+	nvme_get_ctrl(ctrl);
+
+out_close:
+	filp_close(f, NULL);
+
+	return ctrl;
+}
+EXPORT_SYMBOL_GPL(nvme_ctrl_get_by_path);
+
 static ssize_t nvme_sysfs_reset(struct device *dev,
 				struct device_attribute *attr, const char *buf,
 				size_t count)
drivers/nvme/host/nvme.h +2 -0
index ecbd90c31d0d..7e827c9e892c 100644
@@ -484,6 +484,8 @@ int nvme_get_log(struct nvme_ctrl *ctrl, u32 nsid, u8 log_page, u8 lsp,
 extern const struct attribute_group *nvme_ns_id_attr_groups[];
 extern const struct block_device_operations nvme_ns_head_ops;
 
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
+
 #ifdef CONFIG_NVME_MULTIPATH
 static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
 {
Signature · 1 line
2.20.1
Reviewed-by on v7 1/14
#2 lore raw

Re: [PATCH v7 01/14] nvme-core: introduce nvme_ctrl_get_by_path()

Quoted 6 lines · Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

Quoted 7 lines

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
  drivers/nvme/host/core.c | 24 ++++++++++++++++++++++++
  drivers/nvme/host/nvme.h |  2 ++
  2 files changed, 26 insertions(+)

Looks good,

Reviewed-by Max Gurtovoy <maxg at mellanox.com>
#3 lore raw

Re: [PATCH v7 01/14] nvme-core: introduce nvme_ctrl_get_by_path()

In reply to Logan Gunthorpe
Quoted 25 lines · Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
  drivers/nvme/host/core.c | 24 ++++++++++++++++++++++++
  drivers/nvme/host/nvme.h |  2 ++
  2 files changed, 26 insertions(+)
drivers/nvme/host/core.c +3 -0
index e6ee6f2a3da6..f72334f34a30 100644
@@ -2817,6 +2817,30 @@ static const struct file_operations nvme_dev_fops = {
  	.compat_ioctl	= nvme_dev_ioctl,
  };
 
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+	struct nvme_ctrl *ctrl;

drivers/nvme/host/core.c static const struct file_operations nvme_dev_fops = { · line 2832

Quoted 10 lines
drivers/nvme/host/core.c +10 -0
+	struct file *f;
+
+	f = filp_open(path, O_RDWR, 0);
+	if (IS_ERR(f))
+		return ERR_CAST(f);
+
+	if (f->f_op != &nvme_dev_fops) {
+		ctrl = ERR_PTR(-EINVAL);
+		goto out_close;
+	}

Logan,

this means that the PT is for nvme-pci and also nvme-fabrics as well.

Is this the intention ? or we want to restrict it to pci only.

Quoted 27 lines
drivers/nvme/host/core.c +11 -0
+
+	ctrl = f->private_data;
+	nvme_get_ctrl(ctrl);
+
+out_close:
+	filp_close(f, NULL);
+
+	return ctrl;
+}
+EXPORT_SYMBOL_GPL(nvme_ctrl_get_by_path);
+
  static ssize_t nvme_sysfs_reset(struct device *dev,
  				struct device_attribute *attr, const char *buf,
  				size_t count)
drivers/nvme/host/nvme.h +2 -0
index ecbd90c31d0d..7e827c9e892c 100644
@@ -484,6 +484,8 @@ int nvme_get_log(struct nvme_ctrl *ctrl, u32 nsid, u8 log_page, u8 lsp,
  extern const struct attribute_group *nvme_ns_id_attr_groups[];
  extern const struct block_device_operations nvme_ns_head_ops;
 
+struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path);
+
  #ifdef CONFIG_NVME_MULTIPATH
  static inline bool nvme_ctrl_use_ana(struct nvme_ctrl *ctrl)
  {
#4 lore raw

Re: [PATCH v7 01/14] nvme-core: introduce nvme_ctrl_get_by_path()

Quoted 40 lines · Max Gurtovoy
Quoted 25 lines · Logan Gunthorpe

nvme_ctrl_get_by_path() is analagous to blkdev_get_by_path() except it gets a struct nvme_ctrl from the path to its char dev (/dev/nvme0). It makes use of filp_open() to open the file and uses the private data to obtain a pointer to the struct nvme_ctrl. If the fops of the file do not match, -EINVAL is returned.

The purpose of this function is to support NVMe-OF target passthru.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>

---
? drivers/nvme/host/core.c | 24 ++++++++++++++++++++++++ ? drivers/nvme/host/nvme.h |? 2 ++
? 2 files changed, 26 insertions(+)

drivers/nvme/host/core.c +2 -0
index e6ee6f2a3da6..f72334f34a30 100644
@@ -2817,6 +2817,30 @@ static const struct file_operations
 nvme_dev_fops = {
 ????? .compat_ioctl??? = nvme_dev_ioctl,
 ? };
 ? +struct nvme_ctrl *nvme_ctrl_get_by_path(const char *path)
+{
+??? struct nvme_ctrl *ctrl;

drivers/nvme/host/core.c static const struct file_operations · line 2832

Quoted 10 lines
drivers/nvme/host/core.c +10 -0
+??? struct file *f;
+
+??? f = filp_open(path, O_RDWR, 0);
+??? if (IS_ERR(f))
+??????? return ERR_CAST(f);
+
+??? if (f->f_op != &nvme_dev_fops) {
+??????? ctrl = ERR_PTR(-EINVAL);
+??????? goto out_close;
+??? }

Logan,

Quoted 3 lines

this means that the PT is for nvme-pci and also nvme-fabrics as well.

Is this the intention ? or we want to restrict it to pci only.

Yes, in theory, someone could passthru an nvme-fabrics controller or they could passthru a passthru'd passthru'd nvme-fabrics controller. This probably isn't a good idea but I don't know that we need to specifically reject it. If you think we should I could figure out a way to filter by pci controllers only.

Logan

#5 lore raw

PATCH v7 2/14 nvme-core: export existing ctrl and ns interfaces

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

We export existing nvme ctrl and ns management APIs so that target passthru code can manage the nvme ctrl.

This is a preparation patch for implementing NVMe Over Fabric target passthru feature.

Signed-off-by Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/host/core.c | 17 +++++++++++------
 drivers/nvme/host/nvme.h |  7 +++++++
 2 files changed, 18 insertions(+), 6 deletions(-)
drivers/nvme/host/core.c +11 -6
index f72334f34a30..4445efb16a84 100644
@@ -412,10 +412,11 @@ static void nvme_free_ns(struct kref *kref)
 	kfree(ns);
 }
 
-static void nvme_put_ns(struct nvme_ns *ns)
+void nvme_put_ns(struct nvme_ns *ns)
 {
 	kref_put(&ns->kref, nvme_free_ns);
 }
+EXPORT_SYMBOL_GPL(nvme_put_ns);
 
 static inline void nvme_clear_nvme_request(struct request *req)
 {
@@ -1088,7 +1089,7 @@ static int nvme_identify_ns_list(struct nvme_ctrl *dev, unsigned nsid, __le32 *n
 				    NVME_IDENTIFY_DATA_SIZE);
 }
 
-static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl,
+struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl,
 		unsigned nsid)
 {
 	struct nvme_id_ns *id;
@@ -1113,6 +1114,7 @@ static struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl,
 
 	return id;
 }
+EXPORT_SYMBOL_GPL(nvme_identify_ns);
 
 static int nvme_features(struct nvme_ctrl *dev, u8 op, unsigned int fid,
 		unsigned int dword11, void *buffer, size_t buflen, u32 *result)
@@ -1261,8 +1263,8 @@ static u32 nvme_known_admin_effects(u8 opcode)
 	return 0;
 }
 
-static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
-								u8 opcode)
+u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+		u8 opcode)
 {
 	u32 effects = 0;
 
@@ -1291,6 +1293,7 @@ static u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 	}
 	return effects;
 }
+EXPORT_SYMBOL_GPL(nvme_passthru_start);
 
 static void nvme_update_formats(struct nvme_ctrl *ctrl)
 {
@@ -1305,7 +1308,7 @@ static void nvme_update_formats(struct nvme_ctrl *ctrl)
 	nvme_remove_invalid_namespaces(ctrl, NVME_NSID_ALL);
 }
 
-static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
+void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 {
 	/*
 	 * Revalidate LBA changes prior to unfreezing. This is necessary to
@@ -1323,6 +1326,7 @@ static void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects)
 	if (effects & (NVME_CMD_EFFECTS_NIC | NVME_CMD_EFFECTS_NCC))
 		nvme_queue_scan(ctrl);
 }
+EXPORT_SYMBOL_GPL(nvme_passthru_end);
 
 static int nvme_user_cmd(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
 			struct nvme_passthru_cmd __user *ucmd)
@@ -3265,7 +3269,7 @@ static int ns_cmp(void *priv, struct list_head *a, struct list_head *b)
 	return nsa->head->ns_id - nsb->head->ns_id;
 }
 
-static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 {
 	struct nvme_ns *ns, *ret = NULL;
 
@@ -3283,6 +3287,7 @@ static struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned nsid)
 	up_read(&ctrl->namespaces_rwsem);
 	return ret;
 }
+EXPORT_SYMBOL_GPL(nvme_find_get_ns);
 
 static int nvme_setup_streams_ns(struct nvme_ctrl *ctrl, struct nvme_ns *ns)
 {
drivers/nvme/host/nvme.h +7 -0
index 7e827c9e892c..ccd3aff675fb 100644
@@ -436,6 +436,8 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl);
 void nvme_stop_ctrl(struct nvme_ctrl *ctrl);
 void nvme_put_ctrl(struct nvme_ctrl *ctrl);
 int nvme_init_identify(struct nvme_ctrl *ctrl);
+struct nvme_ns *nvme_find_get_ns(struct nvme_ctrl *ctrl, unsigned int nsid);
+void nvme_put_ns(struct nvme_ns *ns);
 
 void nvme_remove_namespaces(struct nvme_ctrl *ctrl);
 
@@ -472,8 +474,13 @@ int nvme_set_features(struct nvme_ctrl *dev, unsigned int fid,
 int nvme_get_features(struct nvme_ctrl *dev, unsigned int fid,
 		      unsigned int dword11, void *buffer, size_t buflen,
 		      u32 *result);
+u32 nvme_passthru_start(struct nvme_ctrl *ctrl, struct nvme_ns *ns,
+		u8 opcode);
+void nvme_passthru_end(struct nvme_ctrl *ctrl, u32 effects);
 int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count);
 void nvme_stop_keep_alive(struct nvme_ctrl *ctrl);
+struct nvme_id_ns *nvme_identify_ns(struct nvme_ctrl *ctrl,
+		unsigned int nsid);
 int nvme_reset_ctrl(struct nvme_ctrl *ctrl);
 int nvme_reset_ctrl_sync(struct nvme_ctrl *ctrl);
 int nvme_delete_ctrl(struct nvme_ctrl *ctrl);
Signature · 1 line
2.20.1
#6 lore raw

PATCH v7 3/14 nvmet: add return value to nvmet_add_async_event()

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

Change the return value for nvmet_add_async_event().

This change is needed for the target passthru code to generate async events.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com> [logang at deltatee.com: fixed up commit message that was no
longer accurate]
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
 drivers/nvme/target/core.c  | 5 +++--
 drivers/nvme/target/nvmet.h | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +3 -2
index 3a67e244e568..f7f25bdc4763 100644
@@ -173,14 +173,14 @@ static void nvmet_async_event_work(struct work_struct *work)
 	}
 }
 
-void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
+bool nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
 		u8 event_info, u8 log_page)
 {
 	struct nvmet_async_event *aen;
 
 	aen = kmalloc(sizeof(*aen), GFP_KERNEL);
 	if (!aen)
-		return;
+		return false;
 
 	aen->event_type = event_type;
 	aen->event_info = event_info;
@@ -191,6 +191,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
 	mutex_unlock(&ctrl->lock);
 
 	schedule_work(&ctrl->async_event_work);
+	return true;
 }
 
 static void nvmet_add_to_changed_ns_log(struct nvmet_ctrl *ctrl, __le32 nsid)
drivers/nvme/target/nvmet.h +1 -1
index c51f8dd01dc4..217a787952e8 100644
@@ -441,7 +441,7 @@ void nvmet_port_disc_changed(struct nvmet_port *port,
 		struct nvmet_subsys *subsys);
 void nvmet_subsys_disc_changed(struct nvmet_subsys *subsys,
 		struct nvmet_host *host);
-void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
+bool nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
 		u8 event_info, u8 log_page);
 
 #define NVMET_QUEUE_SIZE	1024
Signature · 1 line
2.20.1
#7 lore raw

Re: [PATCH v7 03/14] nvmet: add return value to nvmet_add_async_event()

Quoted 6 lines · Logan Gunthorpe

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Change the return value for nvmet_add_async_event().

This change is needed for the target passthru code to generate async events.

As a stand alone commit it's not clear what is the purpose of it.

Please add some extra explanation in the commit message.

Also better to use integer as return value if the return value should reflect return code.

Quoted 51 lines

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com> [logang at deltatee.com: fixed up commit message that was no
longer accurate]
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
  drivers/nvme/target/core.c  | 5 +++--
  drivers/nvme/target/nvmet.h | 2 +-
  2 files changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +3 -2
index 3a67e244e568..f7f25bdc4763 100644
@@ -173,14 +173,14 @@ static void nvmet_async_event_work(struct work_struct *work)
  	}
  }
 
-void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
+bool nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
  		u8 event_info, u8 log_page)
  {
  	struct nvmet_async_event *aen;
 
  	aen = kmalloc(sizeof(*aen), GFP_KERNEL);
  	if (!aen)
-		return;
+		return false;
 
  	aen->event_type = event_type;
  	aen->event_info = event_info;
@@ -191,6 +191,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
  	mutex_unlock(&ctrl->lock);
 
  	schedule_work(&ctrl->async_event_work);
+	return true;
  }
 
  static void nvmet_add_to_changed_ns_log(struct nvmet_ctrl *ctrl, __le32 nsid)
drivers/nvme/target/nvmet.h +1 -1
index c51f8dd01dc4..217a787952e8 100644
@@ -441,7 +441,7 @@ void nvmet_port_disc_changed(struct nvmet_port *port,
  		struct nvmet_subsys *subsys);
  void nvmet_subsys_disc_changed(struct nvmet_subsys *subsys,
  		struct nvmet_host *host);
-void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
+bool nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
  		u8 event_info, u8 log_page);
 
  #define NVMET_QUEUE_SIZE	1024
#8 lore raw

Re: [PATCH v7 03/14] nvmet: add return value to nvmet_add_async_event()

Quoted 9 lines · Max Gurtovoy
Quoted 6 lines · Logan Gunthorpe

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

Change the return value for nvmet_add_async_event().

This change is needed for the target passthru code to generate async events.

Quoted 6 lines

As a stand alone commit it's not clear what is the purpose of it.

Please add some extra explanation in the commit message.

Also better to use integer as return value if the return value should reflect return code.

Thanks for the Review, Max. I'll queue up these changes for v8.

Logan

#9 lore raw

PATCH v7 4/14 nvmet: make nvmet_copy_ns_identifier() non-static

In reply to Logan Gunthorpe

This function will be needed by the upcoming passthru code.

[chaitanya.kulkarni at wdc.com: this was factored out of a patch
originally authored by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
 drivers/nvme/target/admin-cmd.c | 4 ++--
 drivers/nvme/target/nvmet.h     | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +2 -2
index 4dc12ea52f23..eeb24f606d00 100644
@@ -506,8 +506,8 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req)
 	nvmet_req_complete(req, status);
 }
 
-static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
-				    void *id, off_t *off)
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off)
 {
 	struct nvme_ns_id_desc desc = {
 		.nidt = type,
drivers/nvme/target/nvmet.h +2 -0
index 217a787952e8..d1a0a3190a24 100644
@@ -489,6 +489,8 @@ u16 nvmet_bdev_flush(struct nvmet_req *req);
 u16 nvmet_file_flush(struct nvmet_req *req);
 void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
 
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off);
 static inline u32 nvmet_rw_len(struct nvmet_req *req)
 {
 	return ((u32)le16_to_cpu(req->cmd->rw.length) + 1) <<
Signature · 1 line
2.20.1
#10 lore raw

Re: [PATCH v7 04/14] nvmet: make nvmet_copy_ns_identifier() non-static

Quoted 1 line · Logan Gunthorpe

This function will be needed by the upcoming passthru code.

Same here. As a standalone commit I can't take a lot from here.

Maybe should be squashed ?

Quoted 38 lines

[chaitanya.kulkarni at wdc.com: this was factored out of a patch
originally authored by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
  drivers/nvme/target/admin-cmd.c | 4 ++--
  drivers/nvme/target/nvmet.h     | 2 ++
  2 files changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +2 -2
index 4dc12ea52f23..eeb24f606d00 100644
@@ -506,8 +506,8 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req)
  	nvmet_req_complete(req, status);
  }
 
-static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
-				    void *id, off_t *off)
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off)
  {
  	struct nvme_ns_id_desc desc = {
  		.nidt = type,
drivers/nvme/target/nvmet.h +2 -0
index 217a787952e8..d1a0a3190a24 100644
@@ -489,6 +489,8 @@ u16 nvmet_bdev_flush(struct nvmet_req *req);
  u16 nvmet_file_flush(struct nvmet_req *req);
  void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
 
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off);
  static inline u32 nvmet_rw_len(struct nvmet_req *req)
  {
  	return ((u32)le16_to_cpu(req->cmd->rw.length) + 1) <<
#11 lore raw

Re: [PATCH v7 04/14] nvmet: make nvmet_copy_ns_identifier() non-static

Quoted 7 lines · Max Gurtovoy
Quoted 1 line · Logan Gunthorpe

This function will be needed by the upcoming passthru code.

Same here. As a standalone commit I can't take a lot from here.

Maybe should be squashed ?

This commit changes the existing code which is independent of the passthru code which we are adding. For that reason I've made this standalone as it doesn't have direct modification from w.r.t passthru code.

Perhaps more documentation will make this clear.

Quoted 41 lines
Quoted 40 lines

[chaitanya.kulkarni at wdc.com: this was factored out of a patch
originally authored by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
   drivers/nvme/target/admin-cmd.c | 4 ++--
   drivers/nvme/target/nvmet.h     | 2 ++
   2 files changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/target/admin-cmd.c +2 -2
index 4dc12ea52f23..eeb24f606d00 100644
@@ -506,8 +506,8 @@ static void nvmet_execute_identify_nslist(struct nvmet_req *req)
   	nvmet_req_complete(req, status);
   }
 
-static u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
-				    void *id, off_t *off)
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off)
   {
   	struct nvme_ns_id_desc desc = {
   		.nidt = type,
drivers/nvme/target/nvmet.h +2 -0
index 217a787952e8..d1a0a3190a24 100644
@@ -489,6 +489,8 @@ u16 nvmet_bdev_flush(struct nvmet_req *req);
   u16 nvmet_file_flush(struct nvmet_req *req);
   void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
 
+u16 nvmet_copy_ns_identifier(struct nvmet_req *req, u8 type, u8 len,
+			     void *id, off_t *off);
   static inline u32 nvmet_rw_len(struct nvmet_req *req)
   {
   	return ((u32)le16_to_cpu(req->cmd->rw.length) + 1) <<
#12 lore raw

PATCH v7 5/14 nvmet-passthru: update KConfig with config passthru option

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

This patch updates KConfig file for the NVMeOF target where we add new option so that user can selectively enable/disable passthru code.

Signed-off-by Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
[logang at deltatee.com: fixed some of the wording in the help message]
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/Kconfig | 10 ++++++++++
 1 file changed, 10 insertions(+)
drivers/nvme/target/Kconfig +10 -0
index d7f48c0fb311..2478cb5a932d 100644
@@ -15,6 +15,16 @@ config NVME_TARGET
 	  To configure the NVMe target you probably want to use the nvmetcli
 	  tool from http://git.infradead.org/users/hch/nvmetcli.git.
 
+config NVME_TARGET_PASSTHRU
+	bool "NVMe Target Passthrough support"
+	depends on NVME_CORE
+	depends on NVME_TARGET
+	help
+	  This enables target side NVMe passthru controller support for the
+	  NVMe Over Fabrics protocol. It allows for hosts to manage and
+	  directly access an actual NVMe controller residing on the target
+	  side, incuding executing Vendor Unique Commands.
+
 config NVME_TARGET_LOOP
 	tristate "NVMe loopback device support"
 	depends on NVME_TARGET
Signature · 1 line
2.20.1
Reviewed-by on v7 5/14
#13 lore raw

Re: [PATCH v7 05/14] nvmet-passthru: update KConfig with config passthru option

Quoted 23 lines · Logan Gunthorpe

From: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>

This patch updates KConfig file for the NVMeOF target where we add new option so that user can selectively enable/disable passthru code.

Signed-off-by Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
[logang at deltatee.com: fixed some of the wording in the help message]
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
  drivers/nvme/target/Kconfig | 10 ++++++++++
  1 file changed, 10 insertions(+)
drivers/nvme/target/Kconfig +3 -0
index d7f48c0fb311..2478cb5a932d 100644
@@ -15,6 +15,16 @@ config NVME_TARGET
  	  To configure the NVMe target you probably want to use the nvmetcli
  	  tool from http://git.infradead.org/users/hch/nvmetcli.git.
 
+config NVME_TARGET_PASSTHRU
+	bool "NVMe Target Passthrough support"
+	depends on NVME_CORE

drivers/nvme/target/Kconfig config NVME_TARGET · line 30

Quoted 10 lines
drivers/nvme/target/Kconfig +7 -0
+	depends on NVME_TARGET
+	help
+	  This enables target side NVMe passthru controller support for the
+	  NVMe Over Fabrics protocol. It allows for hosts to manage and
+	  directly access an actual NVMe controller residing on the target
+	  side, incuding executing Vendor Unique Commands.
+
  config NVME_TARGET_LOOP
  	tristate "NVMe loopback device support"
  	depends on NVME_TARGET

Looks good,

Reviewed-by Max Gurtovoy <maxg at mellanox.com>
#14 lore raw

PATCH v7 6/14 nvmet-passthru: add passthru code to process commands

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

This patch adds passthru command handling capability for the NVMeOF target and exports passthru APIs which are used to integrate passthru code with nvmet-core. We add passthru ns member to the target request to hold the ns reference for respective commands.

The new file io-cmd-passthru.c handles passthru cmd parsing and execution. In the passthru mode, we create a block layer request from the nvmet request and map the data on to the block layer request. For handling the side effects of the passthru admin commands we add two functions similar to the nvme_passthru[start|end]() functions present in the nvme-core. We explicitly blacklist the commands at the time of parsing, which allows us to route the fabric commands through default code path.

We introduce new passthru workqueue similar to the one we have for the file backend for NVMeOF target to execute the NVMe Admin passthru commands.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com> [logang at deltatee.com:
* renamed passthru-cmd.c to io-cmd-passthru.c for consistency
* squashed "update target makefile for passthru"
* squashed "integrate passthru request processing"
* added appropriate CONFIG_NVME_TARGET_PASSTHRU #ifdefs
* pushed passthru_wq into passthrtu.c and introduced nvmet_passthru_init() and nvmet_passthru_destroy() to avoid inline #ifdef mess
* renamed nvmet_passthru_ctrl() to nvmet_req_passthru_ctrl() and provided nvmet_passthr_ctrl() to get the ctrl from a subsys
* fixed failure path in nvmet_passthru_execute_cmd() to ensure we always complete the request (with an error when appropriate)
* restructered out nvmet_passthru_make_request() and nvmet_passthru_execute_cmd() to create nvmet_passthru_map_sg() which makes the code simpler and more readable.
* move call to nvme_find_get_ns() into nvmet_passthru_execute_cmd() to prevent a lockdep error. nvme_find_get_ns() takes a lock and can sleep but nvme_init_req() is called while hctx_lock() is held (in the loop transport) and therefore should not sleep.
* added check in nvmet_passthru_execute_cmd() to ensure we don't violate queue_max_segments or queue_max_hw_sectors.
* added nvmet_passthru_set_mdts() to prevent requests that exceed max_segments
* force setting cmic bit to support multipath for connections to the target
* dropped le16_to_cpu() conversion in nvmet_passthru_req_done() as it's currently already done in nvme_end_request()
* unabbreviated 'VUC' in a comment as it's not a commonly known acronym
* removed unnecessary inline tags on static functions
* minor edits to commit message
]
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
 drivers/nvme/target/Makefile          |   1 +
 drivers/nvme/target/core.c            |  11 +-
 drivers/nvme/target/io-cmd-passthru.c | 572 ++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h           |  46 +++
 4 files changed, 629 insertions(+), 1 deletion(-)
 create mode 100644 drivers/nvme/target/io-cmd-passthru.c
drivers/nvme/target/Makefile +1 -0
index 2b33836f3d3e..bf57799fde63 100644
@@ -11,6 +11,7 @@ obj-$(CONFIG_NVME_TARGET_TCP)		+= nvmet-tcp.o
 
 nvmet-y		+= core.o configfs.o admin-cmd.o fabrics-cmd.o \
 			discovery.o io-cmd-file.o io-cmd-bdev.o
+nvmet-$(CONFIG_NVME_TARGET_PASSTHRU)	+= io-cmd-passthru.o
 nvme-loop-y	+= loop.o
 nvmet-rdma-y	+= rdma.o
 nvmet-fc-y	+= fc.o
drivers/nvme/target/core.c +10 -1
index f7f25bdc4763..50c01b2da568 100644
@@ -895,6 +895,8 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
 	if (unlikely(!req->sq->ctrl))
 		/* will return an error for any Non-connect command: */
 		status = nvmet_parse_connect_cmd(req);
+	else if (nvmet_req_passthru_ctrl(req))
+		status = nvmet_parse_passthru_cmd(req);
 	else if (likely(req->sq->qid != 0))
 		status = nvmet_parse_io_cmd(req);
 	else if (nvme_is_fabrics(req->cmd))
@@ -1462,11 +1464,15 @@ static int __init nvmet_init(void)
 
 	nvmet_ana_group_enabled[NVMET_DEFAULT_ANA_GRPID] = 1;
 
+	error = nvmet_passthru_init();
+	if (error)
+		goto out;
+
 	buffered_io_wq = alloc_workqueue("nvmet-buffered-io-wq",
 			WQ_MEM_RECLAIM, 0);
 	if (!buffered_io_wq) {
 		error = -ENOMEM;
-		goto out;
+		goto out_passthru_destroy;
 	}
 
 	error = nvmet_init_discovery();
@@ -1482,6 +1488,8 @@ static int __init nvmet_init(void)
 	nvmet_exit_discovery();
 out_free_work_queue:
 	destroy_workqueue(buffered_io_wq);
+out_passthru_destroy:
+	nvmet_passthru_destroy();
 out:
 	return error;
 }
@@ -1492,6 +1500,7 @@ static void __exit nvmet_exit(void)
 	nvmet_exit_discovery();
 	ida_destroy(&cntlid_ida);
 	destroy_workqueue(buffered_io_wq);
+	nvmet_passthru_destroy();
 
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_entry) != 1024);
 	BUILD_BUG_ON(sizeof(struct nvmf_disc_rsp_page_hdr) != 1024);
drivers/nvme/target/io-cmd-passthru.c new file +572 -0

572 lines not on this page · raw message

drivers/nvme/target/nvmet.h +46 -0

64 lines not on this page · raw message

Signature · 1 line
2.20.1
#15 lore raw

PATCH v7 7/14 nvmet-passthru: add enable/disable helpers

In reply to Logan Gunthorpe

This patch adds helper functions which are used in the NVMeOF configfs when the user is configuring the passthru subsystem. Here we ensure that only one subsys is assigned to each nvme_ctrl by using an xarray on the cntlid.

[chaitanya.kulkarni at wdc.com: this patch is very roughly based
on a similar one by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
 drivers/nvme/target/core.c            |  8 +++
 drivers/nvme/target/io-cmd-passthru.c | 77 +++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h           | 10 ++++
 3 files changed, 95 insertions(+)
drivers/nvme/target/core.c +8 -0
index 50c01b2da568..2e75968af7f4 100644
@@ -519,6 +519,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 
 	mutex_lock(&subsys->lock);
 	ret = 0;
+
+	if (nvmet_passthru_ctrl(subsys)) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
 	if (ns->enabled)
 		goto out_unlock;
 
@@ -1439,6 +1445,8 @@ static void nvmet_subsys_free(struct kref *ref)
 
 	WARN_ON_ONCE(!list_empty(&subsys->namespaces));
 
+	nvmet_passthru_subsys_free(subsys);
+
 	kfree(subsys->subsysnqn);
 	kfree(subsys);
 }
drivers/nvme/target/io-cmd-passthru.c +77 -0

89 lines not on this page · raw message

drivers/nvme/target/nvmet.h +10 -0
index bd11114ebbb9..aff4db03269d 100644
@@ -230,6 +230,7 @@ struct nvmet_subsys {
 
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
+	char			*passthru_ctrl_path;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
@@ -509,6 +510,9 @@ static inline u32 nvmet_rw_len(struct nvmet_req *req)
 
 int nvmet_passthru_init(void);
 void nvmet_passthru_destroy(void);
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
 u16 nvmet_parse_passthru_cmd(struct nvmet_req *req);
 
 static inline
@@ -526,6 +530,12 @@ static inline int nvmet_passthru_init(void)
 static inline void nvmet_passthru_destroy(void)
 {
 }
+static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+}
+static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+}
 static inline u16 nvmet_parse_passthru_cmd(struct nvmet_req *req)
 {
 	return 0;
Signature · 1 line
2.20.1
#16 lore raw

Re: [PATCH v7 07/14] nvmet-passthru: add enable/disable helpers

Quoted 91 lines · Logan Gunthorpe

This patch adds helper functions which are used in the NVMeOF configfs when the user is configuring the passthru subsystem. Here we ensure that only one subsys is assigned to each nvme_ctrl by using an xarray on the cntlid.

[chaitanya.kulkarni at wdc.com: this patch is very roughly based
on a similar one by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
  drivers/nvme/target/core.c            |  8 +++
  drivers/nvme/target/io-cmd-passthru.c | 77 +++++++++++++++++++++++++++
  drivers/nvme/target/nvmet.h           | 10 ++++
  3 files changed, 95 insertions(+)
drivers/nvme/target/core.c +8 -0
index 50c01b2da568..2e75968af7f4 100644
@@ -519,6 +519,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 
  	mutex_lock(&subsys->lock);
  	ret = 0;
+
+	if (nvmet_passthru_ctrl(subsys)) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
  	if (ns->enabled)
  		goto out_unlock;
 
@@ -1439,6 +1445,8 @@ static void nvmet_subsys_free(struct kref *ref)
 
  	WARN_ON_ONCE(!list_empty(&subsys->namespaces));
 
+	nvmet_passthru_subsys_free(subsys);
+
  	kfree(subsys->subsysnqn);
  	kfree(subsys);
  }
drivers/nvme/target/io-cmd-passthru.c +35 -0
index 46c58fec5608..b199785500ad 100644
@@ -11,6 +11,11 @@
  #include "../host/nvme.h"
  #include "nvmet.h"
 
+/*
+ * xarray to maintain one passthru subsystem per nvme controller.
+ */
+static DEFINE_XARRAY(passthru_subsystems);
+
  static struct workqueue_struct *passthru_wq;
 
  int nvmet_passthru_init(void)
@@ -27,6 +32,78 @@ void nvmet_passthru_destroy(void)
  	destroy_workqueue(passthru_wq);
  }
 
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys)
+{
+	struct nvme_ctrl *ctrl;
+	int ret = -EINVAL;
+	void *old;
+
+	mutex_lock(&subsys->lock);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	if (subsys->nr_namespaces) {
+		pr_info("cannot enable both passthru and regular namespaces for a single subsystem");
+		goto out_unlock;
+	}
+
+	ctrl = nvme_ctrl_get_by_path(subsys->passthru_ctrl_path);
+	if (IS_ERR(ctrl)) {
+		ret = PTR_ERR(ctrl);
+		pr_err("failed to open nvme controller %s\n",
+		       subsys->passthru_ctrl_path);
+
+		goto out_unlock;
+	}
+
+	old = xa_cmpxchg(&passthru_subsystems, ctrl->cntlid, NULL,
+			 subsys, GFP_KERNEL);
+	if (xa_is_err(old)) {
+		ret = xa_err(old);

drivers/nvme/target/io-cmd-passthru.c void nvmet_passthru_destroy(void) · line 74

Quoted 10 lines
drivers/nvme/target/io-cmd-passthru.c +10 -0
+		goto out_put_ctrl;
+	}
+
+	if (old)
+		goto out_put_ctrl;
+
+	subsys->passthru_ctrl = ctrl;
+	ret = 0;
+
+	goto out_unlock;

can we re-arrange the code here ?

it's not so common to see goto in a good flow.

maybe have 1 good flow the goto's will go bellow it as we usually do in this subsystem.

Quoted 70 lines
drivers/nvme/target/io-cmd-passthru.c +32 -0
+
+out_put_ctrl:
+	nvme_put_ctrl(ctrl);
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+
+static void __nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	if (subsys->passthru_ctrl) {
+		xa_erase(&passthru_subsystems, subsys->passthru_ctrl->cntlid);
+		nvme_put_ctrl(subsys->passthru_ctrl);
+	}
+	subsys->passthru_ctrl = NULL;
+}
+
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	mutex_unlock(&subsys->lock);
+}
+
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+	mutex_lock(&subsys->lock);
+	__nvmet_passthru_ctrl_disable(subsys);
+	kfree(subsys->passthru_ctrl_path);
+	mutex_unlock(&subsys->lock);
+}
+
  static void nvmet_passthru_req_complete(struct nvmet_req *req,
  		struct request *rq, u16 status)
  {
drivers/nvme/target/nvmet.h +10 -0
index bd11114ebbb9..aff4db03269d 100644
@@ -230,6 +230,7 @@ struct nvmet_subsys {
 
  #ifdef CONFIG_NVME_TARGET_PASSTHRU
  	struct nvme_ctrl	*passthru_ctrl;
+	char			*passthru_ctrl_path;
  #endif /* CONFIG_NVME_TARGET_PASSTHRU */
  };
 
@@ -509,6 +510,9 @@ static inline u32 nvmet_rw_len(struct nvmet_req *req)
 
  int nvmet_passthru_init(void);
  void nvmet_passthru_destroy(void);
+void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
+int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
+void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
  u16 nvmet_parse_passthru_cmd(struct nvmet_req *req);
 
  static inline
@@ -526,6 +530,12 @@ static inline int nvmet_passthru_init(void)
  static inline void nvmet_passthru_destroy(void)
  {
  }
+static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
+{
+}
+static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
+{
+}
  static inline u16 nvmet_parse_passthru_cmd(struct nvmet_req *req)
  {
  	return 0;
#17 lore raw

Re: [PATCH v7 07/14] nvmet-passthru: add enable/disable helpers

Quoted 102 lines · Max Gurtovoy
Quoted 87 lines · Logan Gunthorpe

This patch adds helper functions which are used in the NVMeOF configfs when the user is configuring the passthru subsystem. Here we ensure that only one subsys is assigned to each nvme_ctrl by using an xarray on the cntlid.

[chaitanya.kulkarni at wdc.com: this patch is very roughly based ? on a similar one by Chaitanya]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>
---

? drivers/nvme/target/core.c??????????? |? 8 +++
? drivers/nvme/target/io-cmd-passthru.c | 77 +++++++++++++++++++++++++++
? drivers/nvme/target/nvmet.h?????????? | 10 ++++

? 3 files changed, 95 insertions(+)

drivers/nvme/target/core.c +6 -0
index 50c01b2da568..2e75968af7f4 100644
@@ -519,6 +519,12 @@ int nvmet_ns_enable(struct nvmet_ns *ns)
 ? ????? mutex_lock(&subsys->lock);
 ????? ret = 0;
+
+??? if (nvmet_passthru_ctrl(subsys)) {
+??????? pr_info("cannot enable both passthru and regular namespaces
 for a single subsystem");
+??????? goto out_unlock;
+??? }
+
 ????? if (ns->enabled)
 ????????? goto out_unlock;
 ? @@ -1439,6 +1445,8 @@ static void nvmet_subsys_free(struct kref *ref)

? ????? WARN_ON_ONCE(!list_empty(&subsys->namespaces)); ? +??? nvmet_passthru_subsys_free(subsys);
+
????? kfree(subsys->subsysnqn);
????? kfree(subsys);
? }

drivers/nvme/target/io-cmd-passthru.c +0 -0
index 46c58fec5608..b199785500ad 100644
@@ -11,6 +11,11 @@

? #include "../host/nvme.h"
? #include "nvmet.h"
? +/*
+ * xarray to maintain one passthru subsystem per nvme controller.
+ */

drivers/nvme/target/io-cmd-passthru.c +2 -0
+static DEFINE_XARRAY(passthru_subsystems);
+

? static struct workqueue_struct *passthru_wq;
? ? int nvmet_passthru_init(void)

drivers/nvme/target/io-cmd-passthru.c +29 -0
@@ -27,6 +32,78 @@ void nvmet_passthru_destroy(void)
 ????? destroy_workqueue(passthru_wq);
 ? }
 ? +int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys)
+{
+??? struct nvme_ctrl *ctrl;
+??? int ret = -EINVAL;
+??? void *old;
+
+??? mutex_lock(&subsys->lock);
+??? if (!subsys->passthru_ctrl_path)
+??????? goto out_unlock;
+??? if (subsys->passthru_ctrl)
+??????? goto out_unlock;
+
+??? if (subsys->nr_namespaces) {
+??????? pr_info("cannot enable both passthru and regular namespaces
 for a single subsystem");
+??????? goto out_unlock;
+??? }
+
+??? ctrl = nvme_ctrl_get_by_path(subsys->passthru_ctrl_path);
+??? if (IS_ERR(ctrl)) {
+??????? ret = PTR_ERR(ctrl);
+??????? pr_err("failed to open nvme controller %s\n",
+?????????????? subsys->passthru_ctrl_path);
+
+??????? goto out_unlock;
+??? }
+
+??? old = xa_cmpxchg(&passthru_subsystems, ctrl->cntlid, NULL,
+???????????? subsys, GFP_KERNEL);
+??? if (xa_is_err(old)) {
+??????? ret = xa_err(old);

drivers/nvme/target/io-cmd-passthru.c void nvmet_passthru_destroy(void) · line 74

Quoted 10 lines
drivers/nvme/target/io-cmd-passthru.c +10 -0
+??????? goto out_put_ctrl;
+??? }
+
+??? if (old)
+??????? goto out_put_ctrl;
+
+??? subsys->passthru_ctrl = ctrl;
+??? ret = 0;
+
+??? goto out_unlock;

can we re-arrange the code here ?

Quoted 4 lines

it's not so common to see goto in a good flow.

maybe have 1 good flow the goto's will go bellow it as we usually do in this subsystem.

OK, queued up for v8.

Thanks,

Logan

#18 lore raw

PATCH v7 8/14 nvmet-core: allow one host per passthru-ctrl

In reply to Logan Gunthorpe

This patch rejects any new connection to the passthru-ctrl if this controller is already connected to a different host. At the time of allocating the controller we check if the subsys associated with the passthru ctrl is already connected to a host and reject it if the hostnqn differs.

Connections from the same host (by hostnqn) are supported to allow for multipath.

[chaitanya.kulkarni at wdc.com: based conceptually on a similar patch but
different implementation]
Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com>
Signed-off-by: Logan Gunthorpe <logang at deltatee.com>

---
 drivers/nvme/target/core.c            |  4 ++++
 drivers/nvme/target/io-cmd-passthru.c | 31 +++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h           |  7 ++++++
 3 files changed, 42 insertions(+)
drivers/nvme/target/core.c +4 -0
index 2e75968af7f4..c655f26db3da 100644
@@ -1278,6 +1278,10 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
 	if (!ctrl->sqs)
 		goto out_free_cqs;
 
+	ret = nvmet_passthru_alloc_ctrl(subsys, hostnqn);
+	if (ret)
+		goto out_free_sqs;
+
 	ret = ida_simple_get(&cntlid_ida,
 			     NVME_CNTLID_MIN, NVME_CNTLID_MAX,
 			     GFP_KERNEL);
drivers/nvme/target/io-cmd-passthru.c +31 -0
index b199785500ad..06a919283cc5 100644
@@ -104,6 +104,37 @@ void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
 	mutex_unlock(&subsys->lock);
 }
 
+int nvmet_passthru_alloc_ctrl(struct nvmet_subsys *subsys,
+			      const char *hostnqn)
+{
+	struct nvmet_ctrl *ctrl;
+
+	/*
+	 * Check here if this subsystem is already connected to the passthru
+	 * ctrl. We allow only one host to connect to a given passthru
+	 * subsystem.
+	 */
+	int rc = 0;
+
+	mutex_lock(&subsys->lock);
+
+	if (!subsys->passthru_ctrl)
+		goto out;
+
+	if (list_empty(&subsys->ctrls))
+		goto out;
+
+	ctrl = list_first_entry(&subsys->ctrls, struct nvmet_ctrl,
+				subsys_entry);
+
+	if (strcmp(hostnqn, ctrl->hostnqn))
+		rc = -ENODEV;
+
+out:
+	mutex_unlock(&subsys->lock);
+	return rc;
+}
+
 static void nvmet_passthru_req_complete(struct nvmet_req *req,
 		struct request *rq, u16 status)
 {
drivers/nvme/target/nvmet.h +7 -0
index aff4db03269d..6436cb990905 100644
@@ -513,6 +513,8 @@ void nvmet_passthru_destroy(void);
 void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys);
 int nvmet_passthru_ctrl_enable(struct nvmet_subsys *subsys);
 void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys);
+int nvmet_passthru_alloc_ctrl(struct nvmet_subsys *subsys,
+			      const char *hostnqn);
 u16 nvmet_parse_passthru_cmd(struct nvmet_req *req);
 
 static inline
@@ -536,6 +538,11 @@ static inline void nvmet_passthru_subsys_free(struct nvmet_subsys *subsys)
 static inline void nvmet_passthru_ctrl_disable(struct nvmet_subsys *subsys)
 {
 }
+static inline int nvmet_passthru_alloc_ctrl(struct nvmet_subsys *subsys,
+					    const char *hostnqn)
+{
+	return 0;
+}
 static inline u16 nvmet_parse_passthru_cmd(struct nvmet_req *req)
 {
 	return 0;
Signature · 1 line
2.20.1
#19 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 5 lines · Logan Gunthorpe

This patch rejects any new connection to the passthru-ctrl if this controller is already connected to a different host. At the time of allocating the controller we check if the subsys associated with the passthru ctrl is already connected to a host and reject it if the hostnqn differs.

This is a big limitation.

Are we plan to enable many front-end ctrl's to connect to the single back-end ctrl in the future ?

I guess it can be incremental to this series.

#20 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 8 lines · Max Gurtovoy
Quoted 5 lines · Logan Gunthorpe

This patch rejects any new connection to the passthru-ctrl if this controller is already connected to a different host. At the time of allocating the controller we check if the subsys associated with the passthru ctrl is already connected to a host and reject it if the hostnqn differs.

Quoted 4 lines

This is a big limitation.

Are we plan to enable many front-end ctrl's to connect to the single back-end ctrl in the future ?

Honestly, I don't know that it's really necessary, but the limitation was requested by Sagi the first time this patch-set was submitted[1] citing unspecified user troubles. If there's consensus to remove the restriction I certainly can.

Logan

[1] http://lists.infradead.org/pipermail/linux-nvme/2018-April/016588.html

#21 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 12 lines · Logan Gunthorpe
Quoted 10 lines · Max Gurtovoy
Quoted 5 lines · Logan Gunthorpe

This patch rejects any new connection to the passthru-ctrl if this controller is already connected to a different host. At the time of allocating the controller we check if the subsys associated with the passthru ctrl is already connected to a host and reject it if the hostnqn differs.

This is a big limitation.

Are we plan to enable many front-end ctrl's to connect to the single back-end ctrl in the future ?

Quoted 8 lines

Honestly, I don't know that it's really necessary, but the limitation was requested by Sagi the first time this patch-set was submitted[1] citing unspecified user troubles. If there's consensus to remove the restriction I certainly can.

Logan

[1] http://lists.infradead.org/pipermail/linux-nvme/2018-April/016588.html

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

can you explain the trouble you meant and how this limitation solve it ?

maintainer
#22 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 10 lines

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

can you explain the trouble you meant and how this limitation solve it ?

Its different to emulate the controller with all its admin commands vs. passing it through to the nvme device.. (think of format nvm)

#23 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 5 lines · Sagi Grimberg
Quoted 5 lines

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Quoted 11 lines
Quoted 5 lines

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

can you explain the trouble you meant and how this limitation solve it ?

Its different to emulate the controller with all its admin commands vs. passing it through to the nvme device.. (think of format nvm)

we don't need to support format command for PT ctrl as we don't support other commands such create_sq/cq.

maintainer
#24 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

can you explain the trouble you meant and how this limitation solve it ?

Its different to emulate the controller with all its admin commands vs. passing it through to the nvme device.. (think of format nvm)

we don't need to support format command for PT ctrl as we don't support other commands such create_sq/cq.

That is just an example, basically every command that we are not aware of we simply passthru to the drive without knowing the implications on a multi-host environment..

maintainer
#25 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 10 lines

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

can you explain the trouble you meant and how this limitation solve it ?

Its different to emulate the controller with all its admin commands vs. passing it through to the nvme device.. (think of format nvm)

Quoted 5 lines
Quoted 1 line

we don't need to support format command for PT ctrl as we don't support other commands such create_sq/cq.

That is just an example, basically every command that we are not aware of we simply passthru to the drive without knowing the implications on a multi-host environment..

If we were to change the logic of nvmet_parse_passthru_admin_cmd to have the default case do nvmet_parse_admin_cmd, and only have the vendor-specific space opcodes do nvmet_passthru_execute_cmd then I could not see at the moment how we can break a multi-host export...

#26 lore raw

Re: [PATCH v7 08/14] nvmet-core: allow one host per passthru-ctrl

Quoted 29 lines · Sagi Grimberg
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 7 lines

I don't understand why we don't limit a regular ctrl to single access and we do it for the PT ctrl.

I guess the block layer helps to sync between multiple access in parallel but we can do it as well.

Quoted 8 lines

Also, let's say you limit the access to this subsystem to 1 user, the bdev is still accessibly for local user and also you can create a different subsystem that will use this device (PT and non-PT ctrl).

Sagi,

can you explain the trouble you meant and how this limitation solve it ?

Its different to emulate the controller with all its admin commands vs. passing it through to the nvme device.. (think of format nvm)

Quoted 6 lines
Quoted 2 lines

we don't need to support format command for PT ctrl as we don't support other commands such create_sq/cq.

That is just an example, basically every command that we are not aware of we simply passthru to the drive without knowing the implications on a multi-host environment..

Quoted 5 lines

If we were to change the logic of nvmet_parse_passthru_admin_cmd to have the default case do nvmet_parse_admin_cmd, and only have the vendor-specific space opcodes do nvmet_passthru_execute_cmd then I could not see at the moment how we can break a multi-host export...

That makes sense. I'll make that change and resend a v8 next week.

Logan

#27 lore raw

PATCH v7 9/14 nvmet-core: don't check the data len for pt-ctrl

In reply to Logan Gunthorpe

From: Chaitanya Kulkarni <[email protected]>

Right now, data_len is calculated before the transfer len after we parse the command, With passthru interface we allow VUCs (Vendor-Unique Commands). In order to make the code simple and compact, instead of assigning the data len or each VUC in the command parse function just use the transfer len as it is. This may result in error if expected data_len != transfer_len.

Signed-off-by: Chaitanya Kulkarni <chaitanya.kulkarni at wdc.com> [logang at deltatee.com:
* added definition of VUC to the commit message and comment
* use nvmet_req_passthru_ctrl() helper seeing we can't dereference

     subsys->passthru_ctrl if CONFIG_NVME_TARGET_PASSTHRU is not set]
Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/core.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/nvme/target/core.c +10 -1
index c655f26db3da..1cda94437fbf 100644
@@ -941,7 +941,16 @@ EXPORT_SYMBOL_GPL(nvmet_req_uninit);
 
 void nvmet_req_execute(struct nvmet_req *req)
 {
-	if (unlikely(req->data_len != req->transfer_len)) {
+	/*
+	 * data_len is calculated before the transfer len after we parse
+	 * the command, With passthru interface we allow VUC (Vendor-Unique
+	 * Commands)'s. In order to make the code simple and compact,
+	 * instead of assinging the dala len for each VUC in the command
+	 * parse function just use the transfer len as it is. This may
+	 * result in error if expected data_len != transfer_len.
+	 */
+	if (!(req->sq->ctrl && nvmet_req_passthru_ctrl(req)) &&
+	    unlikely(req->data_len != req->transfer_len)) {
 		req->error_loc = offsetof(struct nvme_common_command, dptr);
 		nvmet_req_complete(req, NVME_SC_SGL_INVALID_DATA | NVME_SC_DNR);
 	} else
Signature · 1 line
2.20.1
#28 lore raw

PATCH v7 10/14 nvmet-tcp: don't check data_len in nvmet_tcp_map_data()

In reply to Logan Gunthorpe

With passthru, the data_len is no longer guaranteed to be set for all requests. Therefore, we should not check for it to be non-zero. Instead check if the SGL length is zero and map when appropriate.

None of the other transports check data_len which is verified in core code.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/tcp.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/tcp.c +1 -1
index 69b83fa0c76c..db5bc5f444c4 100644
@@ -320,7 +320,7 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)
 	struct nvme_sgl_desc *sgl = &cmd->req.cmd->common.dptr.sgl;
 	u32 len = le32_to_cpu(sgl->length);
 
-	if (!cmd->req.data_len)
+	if (!len)
 		return 0;
 
 	if (sgl->type == ((NVME_SGL_FMT_DATA_DESC << 4) |
Signature · 1 line
2.20.1
#29 lore raw

PATCH v7 11/14 nvmet-configfs: introduce passthru configfs interface

In reply to Logan Gunthorpe

When CONFIG_NVME_TARGET_PASSTHRU as 'passthru' directory will be added to each subsystem. The directory is similar to a namespace and has two attributes: device_path and enable. The user must set the path to the nvme controller's char device and write '1' to enable the subsystem to use passthru.

Any given subsystem is prevented from enabling both a regular namespace and the passthru device. If one is enabled, enabling the other will produce an error.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/configfs.c | 99 ++++++++++++++++++++++++++++++++++
 drivers/nvme/target/nvmet.h    |  1 +
 2 files changed, 100 insertions(+)
drivers/nvme/target/configfs.c +99 -0
index 98613a45bd3b..b15d64c19f58 100644
@@ -615,6 +615,103 @@ static const struct config_item_type nvmet_namespaces_type = {
 	.ct_owner		= THIS_MODULE,
 };
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+
+static ssize_t nvmet_passthru_device_path_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->passthru_ctrl_path);
+}
+
+static ssize_t nvmet_passthru_device_path_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	int ret = -ENOMEM;
+	size_t len;
+
+	mutex_lock(&subsys->lock);
+
+	ret = -EBUSY;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	ret = -EINVAL;
+	len = strcspn(page, "\n");
+	if (!len)
+		goto out_unlock;
+
+	kfree(subsys->passthru_ctrl_path);
+	ret = -ENOMEM;
+	subsys->passthru_ctrl_path = kstrndup(page, len, GFP_KERNEL);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+
+	mutex_unlock(&subsys->lock);
+
+	return count;
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+CONFIGFS_ATTR(nvmet_passthru_, device_path);
+
+static ssize_t nvmet_passthru_enable_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return sprintf(page, "%d\n", subsys->passthru_ctrl ? 1 : 0);
+}
+
+static ssize_t nvmet_passthru_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	bool enable;
+	int ret = 0;
+
+	if (strtobool(page, &enable))
+		return -EINVAL;
+
+	if (enable)
+		ret = nvmet_passthru_ctrl_enable(subsys);
+	else
+		nvmet_passthru_ctrl_disable(subsys);
+
+	return ret ? ret : count;
+}
+CONFIGFS_ATTR(nvmet_passthru_, enable);
+
+static struct configfs_attribute *nvmet_passthru_attrs[] = {
+	&nvmet_passthru_attr_device_path,
+	&nvmet_passthru_attr_enable,
+	NULL,
+};
+
+static const struct config_item_type nvmet_passthru_type = {
+	.ct_attrs		= nvmet_passthru_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+	config_group_init_type_name(&subsys->passthru_group,
+				    "passthru", &nvmet_passthru_type);
+	configfs_add_default_group(&subsys->passthru_group,
+				   &subsys->group);
+}
+
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+}
+
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
 static int nvmet_port_subsys_allow_link(struct config_item *parent,
 		struct config_item *target)
 {
@@ -915,6 +1012,8 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
 	configfs_add_default_group(&subsys->allowed_hosts_group,
 			&subsys->group);
 
+	nvmet_add_passthru_group(subsys);
+
 	return &subsys->group;
 }
 
drivers/nvme/target/nvmet.h +1 -0
index 6436cb990905..f9c593f1305d 100644
@@ -231,6 +231,7 @@ struct nvmet_subsys {
 #ifdef CONFIG_NVME_TARGET_PASSTHRU
 	struct nvme_ctrl	*passthru_ctrl;
 	char			*passthru_ctrl_path;
+	struct config_group	passthru_group;
 #endif /* CONFIG_NVME_TARGET_PASSTHRU */
 };
 
Signature · 1 line
2.20.1
#30 lore raw

Re: [PATCH v7 11/14] nvmet-configfs: introduce passthru configfs interface

Quoted 29 lines · Logan Gunthorpe

When CONFIG_NVME_TARGET_PASSTHRU as 'passthru' directory will be added to each subsystem. The directory is similar to a namespace and has two attributes: device_path and enable. The user must set the path to the nvme controller's char device and write '1' to enable the subsystem to use passthru.

Any given subsystem is prevented from enabling both a regular namespace and the passthru device. If one is enabled, enabling the other will produce an error.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
  drivers/nvme/target/configfs.c | 99 ++++++++++++++++++++++++++++++++++
  drivers/nvme/target/nvmet.h    |  1 +
  2 files changed, 100 insertions(+)
drivers/nvme/target/configfs.c +5 -0
index 98613a45bd3b..b15d64c19f58 100644
@@ -615,6 +615,103 @@ static const struct config_item_type nvmet_namespaces_type = {
  	.ct_owner		= THIS_MODULE,
  };
 
+#ifdef CONFIG_NVME_TARGET_PASSTHRU
+
+static ssize_t nvmet_passthru_device_path_show(struct config_item *item,
+		char *page)
+{

drivers/nvme/target/configfs.c static const struct config_item_type nvmet_namespaces_type = { · line 632

Quoted 10 lines
drivers/nvme/target/configfs.c +10 -0
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return snprintf(page, PAGE_SIZE, "%s\n", subsys->passthru_ctrl_path);
+}
+
+static ssize_t nvmet_passthru_device_path_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	int ret = -ENOMEM;

seems like a redundant initialization.

Quoted 106 lines
drivers/nvme/target/configfs.c +84 -0
+	size_t len;
+
+	mutex_lock(&subsys->lock);
+
+	ret = -EBUSY;
+	if (subsys->passthru_ctrl)
+		goto out_unlock;
+
+	ret = -EINVAL;
+	len = strcspn(page, "\n");
+	if (!len)
+		goto out_unlock;
+
+	kfree(subsys->passthru_ctrl_path);
+	ret = -ENOMEM;
+	subsys->passthru_ctrl_path = kstrndup(page, len, GFP_KERNEL);
+	if (!subsys->passthru_ctrl_path)
+		goto out_unlock;
+
+	mutex_unlock(&subsys->lock);
+
+	return count;
+out_unlock:
+	mutex_unlock(&subsys->lock);
+	return ret;
+}
+CONFIGFS_ATTR(nvmet_passthru_, device_path);
+
+static ssize_t nvmet_passthru_enable_show(struct config_item *item,
+		char *page)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+
+	return sprintf(page, "%d\n", subsys->passthru_ctrl ? 1 : 0);
+}
+
+static ssize_t nvmet_passthru_enable_store(struct config_item *item,
+		const char *page, size_t count)
+{
+	struct nvmet_subsys *subsys = to_subsys(item->ci_parent);
+	bool enable;
+	int ret = 0;
+
+	if (strtobool(page, &enable))
+		return -EINVAL;
+
+	if (enable)
+		ret = nvmet_passthru_ctrl_enable(subsys);
+	else
+		nvmet_passthru_ctrl_disable(subsys);
+
+	return ret ? ret : count;
+}
+CONFIGFS_ATTR(nvmet_passthru_, enable);
+
+static struct configfs_attribute *nvmet_passthru_attrs[] = {
+	&nvmet_passthru_attr_device_path,
+	&nvmet_passthru_attr_enable,
+	NULL,
+};
+
+static const struct config_item_type nvmet_passthru_type = {
+	.ct_attrs		= nvmet_passthru_attrs,
+	.ct_owner		= THIS_MODULE,
+};
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+	config_group_init_type_name(&subsys->passthru_group,
+				    "passthru", &nvmet_passthru_type);
+	configfs_add_default_group(&subsys->passthru_group,
+				   &subsys->group);
+}
+
+#else /* CONFIG_NVME_TARGET_PASSTHRU */
+
+static void nvmet_add_passthru_group(struct nvmet_subsys *subsys)
+{
+}
+
+#endif /* CONFIG_NVME_TARGET_PASSTHRU */
+
  static int nvmet_port_subsys_allow_link(struct config_item *parent,
  		struct config_item *target)
  {
@@ -915,6 +1012,8 @@ static struct config_group *nvmet_subsys_make(struct config_group *group,
  	configfs_add_default_group(&subsys->allowed_hosts_group,
  			&subsys->group);
 
+	nvmet_add_passthru_group(subsys);
+
  	return &subsys->group;
  }
 
drivers/nvme/target/nvmet.h +1 -0
index 6436cb990905..f9c593f1305d 100644
@@ -231,6 +231,7 @@ struct nvmet_subsys {
  #ifdef CONFIG_NVME_TARGET_PASSTHRU
  	struct nvme_ctrl	*passthru_ctrl;
  	char			*passthru_ctrl_path;
+	struct config_group	passthru_group;
  #endif /* CONFIG_NVME_TARGET_PASSTHRU */
  };
 
#31 lore raw

PATCH v7 12/14 block: don't check blk_rq_is_passthrough() in blk_do_io_stat()

In reply to Logan Gunthorpe

Instead of checking blk_rq_is_passthruough() for every call to blk_do_io_stat(), don't set RQF_IO_STAT for passthrough requests. This should be equivalent, and opens the possibility of passthrough requests specifically requesting statistics tracking.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 block/blk-mq.c | 2 +-
 block/blk.h    | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)
block/blk-mq.c +1 -1
index f78d3287dd82..66cb5bfa2ebf 100644
@@ -318,7 +318,7 @@ static struct request *blk_mq_rq_ctx_init(struct blk_mq_alloc_data *data,
 	rq->cmd_flags = op;
 	if (data->flags & BLK_MQ_REQ_PREEMPT)
 		rq->rq_flags |= RQF_PREEMPT;
-	if (blk_queue_io_stat(data->q))
+	if (blk_queue_io_stat(data->q) && !blk_rq_is_passthrough(rq))
 		rq->rq_flags |= RQF_IO_STAT;
 	INIT_LIST_HEAD(&rq->queuelist);
 	INIT_HLIST_NODE(&rq->hash);
block/blk.h +2 -3
index de6b2e146d6e..554efa769bfe 100644
@@ -234,13 +234,12 @@ int blk_dev_init(void);
  *
  *	a) it's attached to a gendisk, and
  *	b) the queue had IO stats enabled when this request was started, and
- *	c) it's a file system request
+ *	c) it's a file system request (RQF_IO_STAT will not be set otherwise)
  */
 static inline bool blk_do_io_stat(struct request *rq)
 {
 	return rq->rq_disk &&
-	       (rq->rq_flags & RQF_IO_STAT) &&
-		!blk_rq_is_passthrough(rq);
+	       (rq->rq_flags & RQF_IO_STAT);
 }
 
 static inline void req_set_nomerge(struct request_queue *q, struct request *req)
Signature · 1 line
2.20.1
#32 lore raw

PATCH v7 13/14 block: call blk_account_io_start() in blk_execute_rq_nowait()

In reply to Logan Gunthorpe

All existing users of blk_execute_rq[_nowait]() are for passthrough commands and will thus be rejected by blk_do_io_stat().

This allows passthrough requests to opt-in to IO accounting by setting RQF_IO_STAT.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 block/blk-exec.c | 2 ++
 1 file changed, 2 insertions(+)
block/blk-exec.c +2 -0
index 1db44ca0f4a6..e20a852ae432 100644
@@ -55,6 +55,8 @@ void blk_execute_rq_nowait(struct request_queue *q, struct gendisk *bd_disk,
 	rq->rq_disk = bd_disk;
 	rq->end_io = done;
 
+	blk_account_io_start(rq, true);
+
 	/*
 	 * don't check dying flag for MQ because the request won't
 	 * be reused after dying flag is set
Signature · 1 line
2.20.1
#33 lore raw

PATCH v7 14/14 nvmet-passthru: support block accounting

In reply to Logan Gunthorpe

Support block disk accounting by setting the RQF_IO_STAT flag and gendisk in the request.

After this change, IO counts will be reflected correctly in /proc/diskstats for drives being used by passthru.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/io-cmd-passthru.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/io-cmd-passthru.c +4 -1
index 06a919283cc5..cb193b434545 100644
@@ -441,6 +441,9 @@ static struct request *nvmet_passthru_blk_make_request(struct nvmet_req *req,
 	if (unlikely(IS_ERR(rq)))
 		return rq;
 
+	if (blk_queue_io_stat(q) && cmd->common.opcode != nvme_cmd_flush)
+		rq->rq_flags |= RQF_IO_STAT;
+
 	if (req->sg_cnt) {
 		ret = nvmet_passthru_map_sg(req, rq);
 		if (unlikely(ret)) {
@@ -505,7 +508,7 @@ static void nvmet_passthru_execute_cmd(struct nvmet_req *req)
 
 	rq->end_io_data = req;
 	if (req->sq->qid != 0) {
-		blk_execute_rq_nowait(rq->q, NULL, rq, 0,
+		blk_execute_rq_nowait(rq->q, ns->disk, rq, 0,
 				      nvmet_passthru_req_done);
 	} else {
 		req->p.rq = rq;
Signature · 1 line
2.20.1