Skip to contentSkip to conversation

Support discovery log change events

PATCH Superseded Sagi Grimberg 6 participants 28 messages Jul 12, 2019 – Sep 9, 2019 3 messages not in the archive NVMe

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Sagi Grimberg

author maintainer
#0 lore raw

v2 cover Support discovery log change events

We want to be able to support discovery log change events automatically without user intervention.

The definition of discovery log change events applies on "persistent" long lived controllers, so first we need to have discovery controllers to stay for a long time and accept kato value.

Then when we do happen to get a discovery log change event on the persistent discovery controller, we simply fire a udev event to user-space to re-query the discovery log page and connect to new subsystems in the fabric.

This works with James' latest nvme-cli patches.

Changes from v1:
- rebase to nvme-5.3
- pass none if trsvcid is uninitialized
- pass NVME_CTRL_NAME instead of NVME_CTRL_INSTANCE

Sagi Grimberg (3):
nvme-fabrics: allow discovery subsystems accept a kato
nvme: enable aen also for discovery controllers
nvme: fire discovery log page change events to userspace

 drivers/nvme/host/core.c    | 39 ++++++++++++++++++++++++++++++++++---
 drivers/nvme/host/fabrics.c | 12 ++----------
 2 files changed, 38 insertions(+), 13 deletions(-)
Signature · 1 line
2.17.1
author maintainer
#1 lore raw

PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato

This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/fabrics.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/nvme/host/fabrics.c +2 -10
index 1994d5b42f94..d0d066307bb4 100644
@@ -381,8 +381,8 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	 * Set keep-alive timeout in seconds granularity (ms * 1000)
 	 * and add a grace period for controller kato enforcement
 	 */
-	cmd.connect.kato = ctrl->opts->discovery_nqn ? 0 :
-		cpu_to_le32((ctrl->kato + NVME_KATO_GRACE) * 1000);
+	cmd.connect.kato = ctrl->kato ?
+		cpu_to_le32((ctrl->kato + NVME_KATO_GRACE) * 1000) : 0;
 
 	if (ctrl->opts->disable_sqflow)
 		cmd.connect.cattr |= NVME_CONNECT_DISABLE_SQFLOW;
@@ -738,13 +738,6 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 				pr_warn("keep_alive_tmo 0 won't execute keep alives!!!\n");
 			}
 			opts->kato = token;
-
-			if (opts->discovery_nqn && opts->kato) {
-				pr_err("Discovery controllers cannot accept KATO != 0\n");
-				ret = -EINVAL;
-				goto out;
-			}
-
 			break;
 		case NVMF_OPT_CTRL_LOSS_TMO:
 			if (match_int(args, &token)) {
@@ -865,7 +858,6 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
 	}
 
 	if (opts->discovery_nqn) {
-		opts->kato = 0;
 		opts->nr_io_queues = 0;
 		opts->nr_write_queues = 0;
 		opts->nr_poll_queues = 0;
Signature · 1 line
2.17.1
#2 lore raw

PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato

Quoted 7 lines · Sagi Grimberg

This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).

Signed-off-by Sagi Grimberg <sagi at grimberg.me>

This looks good to me.

Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>

Thanks,

#3 lore raw

PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato

In reply to Sagi Grimberg
Quoted 6 lines · Sagi Grimberg

This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).

Quoted 6 lines
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/fabrics.c | 12 ++----------
  1 file changed, 2 insertions(+), 10 deletions(-)
Reviewed-by James Smart <james.smart at broadcom.com>
#4 lore raw

PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato

In reply to Sagi Grimberg
Quoted 11 lines · Sagi Grimberg

This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/fabrics.c | 12 ++----------
  1 file changed, 2 insertions(+), 10 deletions(-)
Reviewed-by Hannes Reinecke <hare at suse.com>

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare at suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#5 lore raw

PATCH v2 2/3 nvme: enable aen also for discovery controllers

In reply to Sagi Grimberg

If the controller supports discovery log page change events, we want to enable it on the host side as well.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +7 -2
index 3077cd4d75bf..116c210826c2 100644
@@ -1190,6 +1190,8 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)
 	if (!supported_aens)
 		return;
 
+	queue_work(nvme_wq, &ctrl->async_event_work);
+
 	status = nvme_set_features(ctrl, NVME_FEAT_ASYNC_EVENT, supported_aens,
 			NULL, 0, &result);
 	if (status)
@@ -3746,10 +3748,13 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl)
 	if (ctrl->kato)
 		nvme_start_keep_alive(ctrl);
 
+	if (ctrl->queue_count > 1 ||
+	    (ctrl->ops->flags & NVME_F_FABRICS &&
+	     ctrl->opts->discovery_nqn))
+		nvme_enable_aen(ctrl);
+
 	if (ctrl->queue_count > 1) {
 		nvme_queue_scan(ctrl);
-		nvme_enable_aen(ctrl);
-		queue_work(nvme_wq, &ctrl->async_event_work);
 		nvme_start_queues(ctrl);
 	}
 }
Signature · 1 line
2.17.1
#6 lore raw

PATCH v2 2/3 nvme: enable aen also for discovery controllers

Quoted 4 lines · Sagi Grimberg

If the controller supports discovery log page change events, we want to enable it on the host side as well.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>

This looks good to me.

Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>

Thanks,

#7 lore raw

PATCH v2 2/3 nvme: enable aen also for discovery controllers

In reply to Sagi Grimberg
Quoted 27 lines · Sagi Grimberg

If the controller supports discovery log page change events, we want to enable it on the host side as well.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/core.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +4 -0
index 3077cd4d75bf..116c210826c2 100644
@@ -1190,6 +1190,8 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)
  	if (!supported_aens)
  		return;
 
+	queue_work(nvme_wq, &ctrl->async_event_work);
+
  	status = nvme_set_features(ctrl, NVME_FEAT_ASYNC_EVENT, supported_aens,
  			NULL, 0, &result);
  	if (status)
@@ -3746,10 +3748,13 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl)
  	if (ctrl->kato)
  		nvme_start_keep_alive(ctrl);
 
+	if (ctrl->queue_count > 1 ||
+	    (ctrl->ops->flags & NVME_F_FABRICS &&

drivers/nvme/host/core.c void nvme_start_ctrl(struct nvme_ctrl *ctrl) · line 3760

Quoted 10 lines
drivers/nvme/host/core.c +3 -2
+	     ctrl->opts->discovery_nqn))
+		nvme_enable_aen(ctrl);
+
  	if (ctrl->queue_count > 1) {
  		nvme_queue_scan(ctrl);
-		nvme_enable_aen(ctrl);
-		queue_work(nvme_wq, &ctrl->async_event_work);
  		nvme_start_queues(ctrl);
  	}
  }

Looks fine, but curious why you needed to qualify the discovery_nqn with the FABRICS flag.? I would have assumed discovery_nqn would have been sufficient.

Reviewed-by James Smart <james.smart at broadcom.com>

-- james

#8 lore raw

PATCH v2 2/3 nvme: enable aen also for discovery controllers

In reply to Sagi Grimberg
Quoted 8 lines · Sagi Grimberg

If the controller supports discovery log page change events, we want to enable it on the host side as well.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/core.c | 9 +++++++--
  1 file changed, 7 insertions(+), 2 deletions(-)
Reviewed-by Hannes Reinecke <hare at suse.com>

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare at suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)
author maintainer
#9 lore raw

PATCH v2 3/3 nvme: fire discovery log page change events to userspace

In reply to Sagi Grimberg

Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
 drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +29 -1
index 116c210826c2..76cd3dd8736a 100644
@@ -1180,7 +1180,8 @@ int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count)
 EXPORT_SYMBOL_GPL(nvme_set_queue_count);
 
 #define NVME_AEN_SUPPORTED \
-	(NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | NVME_AEN_CFG_ANA_CHANGE)
+	(NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | NVME_AEN_CFG_ANA_CHANGE | \
+	 NVME_AEN_CFG_DISC_CHANGE)
 
 static void nvme_enable_aen(struct nvme_ctrl *ctrl)
 {
@@ -3612,6 +3613,30 @@ static void nvme_aen_uevent(struct nvme_ctrl *ctrl)
 	kfree(envp[0]);
 }
 
+static void nvme_disc_aen_uevent(struct nvme_ctrl *ctrl)
+{
+	struct nvmf_ctrl_options *opts = ctrl->opts;
+	char *envp[16];
+	int i, envloc = 0;
+
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_EVENT=discovery");
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_CTRL_NAME=%s",
+			dev_name(ctrl->device));
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_TRTYPE=%s", opts->transport);
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_TRADDR=%s", opts->traddr);
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_TRSVCID=%s",
+			opts->trsvcid ?: "none");
+	envp[envloc++] = kasprintf(GFP_KERNEL, "NVME_HOST_TRADDR=%s",
+			opts->host_traddr ?: "none");
+	envp[envloc] = NULL;
+
+	for (i = 0; i < envloc; i++)
+		dev_dbg(ctrl->device, "%s\n", envp[i]);
+	kobject_uevent_env(&ctrl->device->kobj, KOBJ_CHANGE, envp);
+	for (i = 0; i < envloc; i++)
+		kfree(envp[i]);
+}
+
 static void nvme_async_event_work(struct work_struct *work)
 {
 	struct nvme_ctrl *ctrl =
@@ -3702,6 +3727,9 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)
 		queue_work(nvme_wq, &ctrl->ana_work);
 		break;
 #endif
+	case NVME_AER_NOTICE_DISC_CHANGED:
+		nvme_disc_aen_uevent(ctrl);
+		break;
 	default:
 		dev_warn(ctrl->device, "async event result %08x\n", result);
 	}
Signature · 1 line
2.17.1
#10 lore raw

PATCH v2 3/3 nvme: fire discovery log page change events to userspace

Quoted 5 lines · Sagi Grimberg

Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>

Sagi, Thanks for this patch!

Reviewed-by Minwoo Im <minwoo.im.dev at gmail.com>
#11 lore raw

PATCH v2 3/3 nvme: fire discovery log page change events to userspace

In reply to Sagi Grimberg
Quoted 10 lines · Sagi Grimberg

Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++-
  1 file changed, 29 insertions(+), 1 deletion(-)
Reviewed-by James Smart <james.smart at broadcom.com>
#12 lore raw

PATCH v2 3/3 nvme: fire discovery log page change events to userspace

In reply to Sagi Grimberg
Quoted 9 lines · Sagi Grimberg

Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.

Signed-off-by Sagi Grimberg <sagi at grimberg.me>
---
  drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++-
  1 file changed, 29 insertions(+), 1 deletion(-)
Reviewed-by Hannes Reinecke <hare at suse.com>

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
hare at suse.de                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg
GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG N?rnberg)

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 13 lines · Sagi Grimberg
Quoted 10 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

Quoted 7 lines

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

If I look at other callers of kobject_uevent_env none throws in such a huge context.

Quoted 2 lines

Makes sense?

---end quoted text---

#17 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 19 lines · Christoph Hellwig
Quoted 18 lines · Sagi Grimberg
Quoted 10 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

Quoted 5 lines
Quoted 2 lines

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

thanks,

greg k-h

author maintainer
#18 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

#19 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 7 lines · Sagi Grimberg
Quoted 2 lines

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.

author maintainer
#20 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.

Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...

#21 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Yes we do, userspace should use it to order events.  Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

Quoted 8 lines
Quoted 2 lines

The same is generally true for lot of kernel devices.  We could reduce the chance by using the idr cyclic allocator.

Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...

We hit it right and left without the cyclic allocator, but that won't necessarily remove it.

Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token.  The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.

Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.

-- james

author maintainer
#22 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Yes we do, userspace should use it to order events.  Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

Quoted 8 lines
Quoted 2 lines

The same is generally true for lot of kernel devices.  We could reduce the chance by using the idr cyclic allocator.

Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...

Quoted 5 lines

We hit it right and left without the cyclic allocator, but that won't necessarily remove it.

Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token.  The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.

Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.

This unique token is: trtype:traddr:trsvcid:host-traddr ...

#23 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Quoted 17 lines
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines

Yes we do, userspace should use it to order events.  Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

The same is generally true for lot of kernel devices.  We could reduce the chance by using the idr cyclic allocator.

Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...

Quoted 7 lines
Quoted 5 lines

We hit it right and left without the cyclic allocator, but that won't necessarily remove it.

Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token.  The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.

Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.

This unique token is: trtype:traddr:trsvcid:host-traddr ...

well yes :)  though rather verbose.   There is still a minute window as we're comparing values in sysfs, prior to opening the device, so technically something could change in that window between when we checked sysfs and when we open'd.   We can certain check after we open the device to solve that issue.

There is some elegance to a 32-bit token for the controller (can be an incrementing value) passed in the event and used when servicing the event that avoids a bunch of work.

Doing either of these would eliminate what Hannes liked - looking for the discovery controller with those attributes. Although, I don't know that looking for it is all that meaningful.

#24 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 40 lines · James Smart
Quoted 26 lines · Sagi Grimberg
Quoted 26 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Yes we do, userspace should use it to order events.  Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

Quoted 8 lines
Quoted 2 lines

The same is generally true for lot of kernel devices.  We could reduce the chance by using the idr cyclic allocator.

Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...

We hit it right and left without the cyclic allocator, but that won't necessarily remove it.

Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token.  The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.

Quoted 6 lines
Quoted 4 lines

Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.

This unique token is: trtype:traddr:trsvcid:host-traddr ...

well yes :)  though rather verbose.   There is still a minute window as we're comparing values in sysfs, prior to opening the device, so technically something could change in that window between when we checked sysfs and when we open'd.   We can certain check after we open the device to solve that issue.

Quoted 8 lines

There is some elegance to a 32-bit token for the controller (can be an incrementing value) passed in the event and used when servicing the event that avoids a bunch of work.

Doing either of these would eliminate what Hannes liked - looking for the discovery controller with those attributes. Although, I don't know that looking for it is all that meaningful.

yeah, we do have this fundamental problem with uevents; they do refer to a '/dev/nvmeX' device with those parameters, but this device might be long gone (or have been reallocated) by the time the event is processed.

From my POV we have two choices here:
- rely on the transport options to find a matching controller (ignoring the device name) and use that for sending out discovery requests. In the end, it shouldn't really matter device we're using if the transport options are identical. Although I'm not sure for RDMA; here we don't necessarily have a host transport address, so we _might_ send the discovery via the wrong controller in a CMIC enviroment.
- Match the options in nvme-cli, and just discard the event if it doesn't match. Which is some additional coding in nvme-cli and might ran afoul if we ever miss events.

I'd go for the second option; after considering the first introduces far too many conditions rendering debugging impractical.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      Teamlead Storage & Networking
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 247165 (AG München), GF: Felix Imendörffer
#25 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

In reply to Sagi Grimberg
Quoted 25 lines · Sagi Grimberg
Quoted 25 lines
Quoted 24 lines
Quoted 14 lines
Quoted 11 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

Quoted 7 lines

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

Quoted 7 lines
Quoted 2 lines

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.

thanks,

greg k-h

author maintainer
#26 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

Quoted 5 lines
Quoted 2 lines

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

Quoted 8 lines
Quoted 4 lines

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.

Still don't understand how this is ok...

I have /dev/nvme0 represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.

By the time I actually get to do that, /dev/nvme0 represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.

Its a stretch case, but it was raised by people as a potential issue.

#27 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 43 lines · Sagi Grimberg
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines

You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..

We could send the discovery aen with NVME_CTRL_NAME and have then have systemd run something like:

nvme connect-all -d nvme0 --sysfs

and have nvme-cli retrieve all this stuff from sysfs?

Actually that may be a problem.

There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.

Quoted 5 lines
Quoted 2 lines

This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.

Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?

Yes we do, userspace should use it to order events. Does udev not handle that properly today?

Quoted 8 lines
Quoted 4 lines

The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.

So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.

Still don't understand how this is ok...

I have /dev/nvme0 represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.

Quoted 6 lines

By the time I actually get to do that, /dev/nvme0 represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.

Its a stretch case, but it was raised by people as a potential issue.

Ok, and how do you handle this same thing for something like /dev/sda ? (hint, it isn't new, and is something we solved over a decade ago)

If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.

That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.

thanks,

greg k-h

author maintainer
#28 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 17 lines
Quoted 7 lines

Still don't understand how this is ok...

I have /dev/nvme0 represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.

Quoted 6 lines

By the time I actually get to do that, /dev/nvme0 represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.

Its a stretch case, but it was raised by people as a potential issue.

Ok, and how do you handle this same thing for something like /dev/sda ? (hint, it isn't new, and is something we solved over a decade ago)

Quoted 8 lines

If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.

That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.

We could send these variables when adding the device and then validating them using a rich-text-explanatory symlink. Seems slightly backwards to me, but that would work too.

We create the char device using device_add (in nvme_init_subsystem), I didn't find any way to append env variables to that ADD uevent.

Did you mean that we should add another flavor of device_add that accepts char *envp_ext[]?

What exactly is the "standard manner" to pass these variables to the chardev KOBJ_ADD uevent? All other examples I could find use KOBJ_CHANGE to pass private stuff..

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#29 lore raw

Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace

Quoted 21 lines · Sagi Grimberg
Quoted 21 lines
Quoted 7 lines

Still don't understand how this is ok...

I have /dev/nvme0 represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.

Quoted 6 lines

By the time I actually get to do that, /dev/nvme0 represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.

Its a stretch case, but it was raised by people as a potential issue.

Ok, and how do you handle this same thing for something like /dev/sda ? (hint, it isn't new, and is something we solved over a decade ago)

If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.

Quoted 8 lines
Quoted 4 lines

That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.

We could send these variables when adding the device and then validating them using a rich-text-explanatory symlink. Seems slightly backwards to me, but that would work too.

That's the way the driver model is expected to work, instead of having to do crazy device-specific stuff.

Quoted 2 lines

We create the char device using device_add (in nvme_init_subsystem), I didn't find any way to append env variables to that ADD uevent.

You do that in your uevent or dev_uevent callback like all other subsystems. Nothing "new" to do here, again, it's been working fine for everyone else for well over a decade now :)

thanks,

greg k-h

author maintainer
#30 lore raw

v2 cover Support discovery log change events

In reply to Sagi Grimberg
Quoted 8 lines

We want to be able to support discovery log change events automatically without user intervention.

The definition of discovery log change events applies on "persistent" long lived controllers, so first we need to have discovery controllers to stay for a long time and accept kato value.

Quoted 5 lines

Then when we do happen to get a discovery log change event on the persistent discovery controller, we simply fire a udev event to user-space to re-query the discovery log page and connect to new subsystems in the fabric.

This works with James' latest nvme-cli patches.

James, I've applied this to nvme-5.4

Are you sending out the nvme-cli part? If you can make sure that it works with what is in nvme-5.4 it will be great..

Thanks