Support discovery log change events
Sagi Grimberg
maintainerv2 cover Support discovery log change events
We want to be able to support discovery log change events automatically without user intervention.
The definition of discovery log change events applies on "persistent" long lived controllers, so first we need to have discovery controllers to stay for a long time and accept kato value.
Then when we do happen to get a discovery log change event on the persistent discovery controller, we simply fire a udev event to user-space to re-query the discovery log page and connect to new subsystems in the fabric.
This works with James' latest nvme-cli patches.
Changes from v1:
- rebase to nvme-5.3
- pass none if trsvcid is uninitialized
- pass NVME_CTRL_NAME instead of NVME_CTRL_INSTANCE
Sagi Grimberg (3):
nvme-fabrics: allow discovery subsystems accept a kato
nvme: enable aen also for discovery controllers
nvme: fire discovery log page change events to userspace
drivers/nvme/host/core.c | 39 ++++++++++++++++++++++++++++++++++--- drivers/nvme/host/fabrics.c | 12 ++---------- 2 files changed, 38 insertions(+), 13 deletions(-)
Signature · 1 line
2.17.1
Sagi Grimberg
maintainerPATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato
This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).
--- drivers/nvme/host/fabrics.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/nvme/host/fabrics.c +2 -10
@@ -381,8 +381,8 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)* Set keep-alive timeout in seconds granularity (ms * 1000) * and add a grace period for controller kato enforcement */ - cmd.connect.kato = ctrl->opts->discovery_nqn ?0 : -cpu_to_le32 ((ctrl->kato +NVME_KATO_GRACE ) *1000 ); + cmd.connect.kato = ctrl->kato ? +cpu_to_le32 ((ctrl->kato +NVME_KATO_GRACE ) *1000 ) :0 ;if (ctrl->opts->disable_sqflow) cmd.connect.cattr |=NVME_CONNECT_DISABLE_SQFLOW ;
@@ -738,13 +738,6 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,pr_warn ("keep_alive_tmo 0 won't execute keep alives!!!\n" ); } opts->kato = token; - -if (opts->discovery_nqn && opts->kato) { -pr_err ("Discovery controllers cannot accept KATO != 0\n" ); - ret = -EINVAL ; - goto out; - } -break ;case NVMF_OPT_CTRL_LOSS_TMO :if (match_int (args, &token)) {
@@ -865,7 +858,6 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, }if (opts->discovery_nqn) { - opts->kato =0 ; opts->nr_io_queues =0 ; opts->nr_write_queues =0 ; opts->nr_poll_queues =0 ;
Signature · 1 line
2.17.1
PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato
Quoted 7 lines · Sagi Grimberg
This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
This looks good to me.
Thanks,
PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato
In reply to Sagi GrimbergQuoted 6 lines · Sagi Grimberg
This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).
Quoted 6 lines
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/fabrics.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
PATCH v2 1/3 nvme-fabrics: allow discovery subsystems accept a kato
In reply to Sagi GrimbergQuoted 11 lines · Sagi Grimberg
This modifies the behavior of discovery subsystems to accept a kato as a preparation to support discovery log change events. This also means that now every discovery controller will have a default kato value, and for non-persistent connections the host needs to pass in a zero kato value (keep_alive_tmo=0).
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/fabrics.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v2 2/3 nvme: enable aen also for discovery controllers
In reply to Sagi GrimbergIf the controller supports discovery log page change events, we want to enable it on the host side as well.
--- drivers/nvme/host/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +7 -2
@@ -1190,6 +1190,8 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)if (!supported_aens)return ; +queue_work (nvme_wq, &ctrl->async_event_work); + status =nvme_set_features (ctrl,NVME_FEAT_ASYNC_EVENT , supported_aens,NULL ,0 , &result);if (status)
@@ -3746,10 +3748,13 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl)if (ctrl->kato)nvme_start_keep_alive (ctrl); +if (ctrl->queue_count >1 || + (ctrl->ops->flags &NVME_F_FABRICS && + ctrl->opts->discovery_nqn)) +nvme_enable_aen (ctrl); +if (ctrl->queue_count >1 ) {nvme_queue_scan (ctrl); -nvme_enable_aen (ctrl); -queue_work (nvme_wq, &ctrl->async_event_work);nvme_start_queues (ctrl); } }
Signature · 1 line
2.17.1
PATCH v2 2/3 nvme: enable aen also for discovery controllers
Quoted 4 lines · Sagi Grimberg
If the controller supports discovery log page change events, we want to enable it on the host side as well.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
This looks good to me.
Thanks,
PATCH v2 2/3 nvme: enable aen also for discovery controllers
In reply to Sagi GrimbergQuoted 27 lines · Sagi Grimberg
If the controller supports discovery log page change events, we want to enable it on the host side as well.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)drivers/nvme/host/core.c +4
-0@@ -1190,6 +1190,8 @@ static void nvme_enable_aen(struct nvme_ctrl *ctrl)if (!supported_aens)return ; +queue_work (nvme_wq, &ctrl->async_event_work); + status =nvme_set_features (ctrl,NVME_FEAT_ASYNC_EVENT , supported_aens,NULL ,0 , &result);if (status)@@ -3746,10 +3748,13 @@ void nvme_start_ctrl(struct nvme_ctrl *ctrl)if (ctrl->kato)nvme_start_keep_alive (ctrl); + if (ctrl->queue_count > 1 || + (ctrl->ops->flags & NVME_F_FABRICS &&
drivers/nvme/host/core.c void nvme_start_ctrl(struct nvme_ctrl *ctrl) · line 3760
Quoted 10 lines
drivers/nvme/host/core.c +3
-2+ ctrl->opts->discovery_nqn)) +nvme_enable_aen (ctrl); +if (ctrl->queue_count >1 ) {nvme_queue_scan (ctrl); -nvme_enable_aen (ctrl); -queue_work (nvme_wq, &ctrl->async_event_work);nvme_start_queues (ctrl); } }
Looks fine, but curious why you needed to qualify the discovery_nqn with the FABRICS flag.? I would have assumed discovery_nqn would have been sufficient.
-- james
PATCH v2 2/3 nvme: enable aen also for discovery controllers
In reply to Sagi GrimbergQuoted 8 lines · Sagi Grimberg
If the controller supports discovery log page change events, we want to enable it on the host side as well.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Sagi Grimberg
maintainerPATCH v2 3/3 nvme: fire discovery log page change events to userspace
In reply to Sagi GrimbergProvide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.
--- drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +29 -1
@@ -1180,7 +1180,8 @@ int nvme_set_queue_count(struct nvme_ctrl *ctrl, int *count)EXPORT_SYMBOL_GPL (nvme_set_queue_count);#define NVME_AEN_SUPPORTED \ - (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | NVME_AEN_CFG_ANA_CHANGE) + (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | NVME_AEN_CFG_ANA_CHANGE | \ + NVME_AEN_CFG_DISC_CHANGE)static void nvme_enable_aen (struct nvme_ctrl *ctrl) {
@@ -3612,6 +3613,30 @@ static void nvme_aen_uevent(struct nvme_ctrl *ctrl)kfree (envp[0 ]); } +static void nvme_disc_aen_uevent (struct nvme_ctrl *ctrl) +{ +struct nvmf_ctrl_options *opts = ctrl->opts; +char *envp[16 ]; +int i, envloc =0 ; + + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_EVENT=discovery" ); + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_CTRL_NAME=%s" , +dev_name (ctrl->device)); + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_TRTYPE=%s" , opts->transport); + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_TRADDR=%s" , opts->traddr); + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_TRSVCID=%s" , + opts->trsvcid ?:"none" ); + envp[envloc++] =kasprintf (GFP_KERNEL ,"NVME_HOST_TRADDR=%s" , + opts->host_traddr ?:"none" ); + envp[envloc] =NULL ; + +for (i =0 ; i < envloc; i++) +dev_dbg (ctrl->device,"%s\n" , envp[i]); +kobject_uevent_env (&ctrl->device->kobj,KOBJ_CHANGE , envp); +for (i =0 ; i < envloc; i++) +kfree (envp[i]); +} +static void nvme_async_event_work (struct work_struct *work) { struct nvme_ctrl *ctrl =
@@ -3702,6 +3727,9 @@ static void nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)queue_work (nvme_wq, &ctrl->ana_work);break ;#endif +case NVME_AER_NOTICE_DISC_CHANGED : +nvme_disc_aen_uevent (ctrl); +break ;default :dev_warn (ctrl->device,"async event result %08x\n" , result); }
Signature · 1 line
2.17.1
PATCH v2 3/3 nvme: fire discovery log page change events to userspace
Quoted 5 lines · Sagi Grimberg
Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>
Sagi, Thanks for this patch!
PATCH v2 3/3 nvme: fire discovery log page change events to userspace
In reply to Sagi GrimbergQuoted 10 lines · Sagi Grimberg
Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
PATCH v2 3/3 nvme: fire discovery log page change events to userspace
In reply to Sagi GrimbergQuoted 9 lines · Sagi Grimberg
Provide userspace with nvme discovery controller device instance, controller traddr and trsvcid. We'd expect userspace to handle this event by issuing a discovery + connect.
Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/core.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Christoph Hellwig
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 13 lines · Sagi Grimberg
Quoted 10 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
Quoted 7 lines
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
If I look at other callers of kobject_uevent_env none throws in such a huge context.
Quoted 2 lines
Makes sense?
---end quoted text---
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 19 lines · Christoph Hellwig
Quoted 18 lines · Sagi Grimberg
Quoted 10 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
Quoted 5 lines
Quoted 2 lines
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
thanks,
greg k-h
Sagi Grimberg
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
Christoph Hellwig
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 7 lines · Sagi Grimberg
Quoted 2 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Sagi Grimberg
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 9 lines · Sagi Grimberg
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
Quoted 8 lines
Quoted 2 lines
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...
We hit it right and left without the cyclic allocator, but that won't necessarily remove it.
Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token. The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.
Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.
-- james
Sagi Grimberg
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
Quoted 8 lines
Quoted 2 lines
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...
Quoted 5 lines
We hit it right and left without the cyclic allocator, but that won't necessarily remove it.
Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token. The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.
Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.
This unique token is: trtype:traddr:trsvcid:host-traddr ...
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 18 lines · Sagi Grimberg
Quoted 18 lines
Quoted 17 lines
Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...
Quoted 7 lines
Quoted 5 lines
We hit it right and left without the cyclic allocator, but that won't necessarily remove it.
Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token. The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.
Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.
This unique token is: trtype:traddr:trsvcid:host-traddr ...
well yes :) though rather verbose. There is still a minute window as we're comparing values in sysfs, prior to opening the device, so technically something could change in that window between when we checked sysfs and when we open'd. We can certain check after we open the device to solve that issue.
There is some elegance to a 32-bit token for the controller (can be an incrementing value) passed in the event and used when servicing the event that avoids a bunch of work.
Doing either of these would eliminate what Hannes liked - looking for the discovery controller with those attributes. Although, I don't know that looking for it is all that meaningful.
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 40 lines · James Smart
Quoted 26 lines · Sagi Grimberg
Quoted 26 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
Quoted 8 lines
Quoted 2 lines
The same is generally true for lot of kernel devices. We could reduce the chance by using the idr cyclic allocator.
Well, it was raised by Hannes and James, so I'll ask them respond here because I don't mind having it this way. I personally think that this is a better approach than having a cyclic idr allocator. In general, I don't necessarily think that this is a good idea to have cyclic controller enumerations if we don't absolutely have to...
We hit it right and left without the cyclic allocator, but that won't necessarily remove it.
Perhaps we should have had a unique token assigned to the controller, and have the event pass the name and the token. The cli would then, if the token is present, validate it via an ioctl before proceeding with other ioctls.
Quoted 6 lines
Quoted 4 lines
Where all the connection arguments were added we due to the reuse issue and then solving the question of how to verify and/or lookup the desired controller, by using the shotgun approach rather than being very pointed, which is what the name/token would do.
This unique token is: trtype:traddr:trsvcid:host-traddr ...
well yes :) though rather verbose. There is still a minute window as we're comparing values in sysfs, prior to opening the device, so technically something could change in that window between when we checked sysfs and when we open'd. We can certain check after we open the device to solve that issue.
Quoted 8 lines
There is some elegance to a 32-bit token for the controller (can be an incrementing value) passed in the event and used when servicing the event that avoids a bunch of work.
Doing either of these would eliminate what Hannes liked - looking for the discovery controller with those attributes. Although, I don't know that looking for it is all that meaningful.
yeah, we do have this fundamental problem with uevents; they do refer to a '/dev/nvmeX' device with those parameters, but this device might be long gone (or have been reallocated) by the time the event is processed.
From my POV we have two choices here:
- rely on the transport options to find a matching controller (ignoring the device name) and use that for sending out discovery requests. In the end, it shouldn't really matter device we're using if the transport options are identical. Although I'm not sure for RDMA; here we don't necessarily have a host transport address, so we _might_ send the discovery via the wrong controller in a CMIC enviroment.
- Match the options in nvme-cli, and just discard the event if it doesn't match. Which is some additional coding in nvme-cli and might ran afoul if we ever miss events.
I'd go for the second option; after considering the first introduces far too many conditions rendering debugging impractical.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Teamlead Storage & Networking [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 247165 (AG München), GF: Felix Imendörffer
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
In reply to Sagi GrimbergQuoted 25 lines · Sagi Grimberg
Quoted 25 lines
Quoted 24 lines
Quoted 14 lines
Quoted 11 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
Quoted 7 lines
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
Quoted 7 lines
Quoted 2 lines
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.
thanks,
greg k-h
Sagi Grimberg
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
Quoted 5 lines
Quoted 2 lines
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
Quoted 8 lines
Quoted 4 lines
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.
Still don't understand how this is ok...
I have /dev/nvme0 represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.
By the time I actually get to do that, /dev/nvme0 represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.
Its a stretch case, but it was raised by people as a potential issue.
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 43 lines · Sagi Grimberg
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 19 lines
Quoted 19 lines
Quoted 11 lines
You are correct that this information can be derived from sysfs, but the main reason why we add these here, is because in udev rule we can't just go ahead and start looking these up and parsing these..
We could send the discovery aen with
NVME_CTRL_NAMEand have then have systemd run something like:nvme connect-all -d nvme0 --sysfs
and have nvme-cli retrieve all this stuff from sysfs?
Actually that may be a problem.
There could be a hypothetical case where after the event was fired and before it was handled, the discovery controller went away and came back again with a different controller instance, and the old instance is now a different discovery controller.
Quoted 5 lines
Quoted 2 lines
This is why we need this information in the event. And we verify this information in sysfs in nvme-cli.
Well, that must be a usual issue with uevents, right? Don't we usually have a increasing serial number for that or something?
Yes we do, userspace should use it to order events. Does udev not handle that properly today?
Quoted 8 lines
Quoted 4 lines
The problem is not ordering of events, its really about the fact that the chardev can be removed and reallocated for a different controller (could be a completely different discovery controller) by the time that userspace handles the event.
So? You will have gotten the remove and then new addition uevent in order showing you this. So your userspace code knows that something went away and then came back properly so you should be kept in sync.
Still don't understand how this is ok...
I have
/dev/nvme0represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.
Quoted 6 lines
By the time I actually get to do that,
/dev/nvme0represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.Its a stretch case, but it was raised by people as a potential issue.
Ok, and how do you handle this same thing for something like /dev/sda ? (hint, it isn't new, and is something we solved over a decade ago)
If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.
That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.
thanks,
greg k-h
Sagi Grimberg
maintainerRe: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 17 lines
Quoted 7 lines
Still don't understand how this is ok...
I have
/dev/nvme0represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.Quoted 6 lines
By the time I actually get to do that,
/dev/nvme0represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.Its a stretch case, but it was raised by people as a potential issue.
Ok, and how do you handle this same thing for something like
/dev/sda? (hint, it isn't new, and is something we solved over a decade ago)
Quoted 8 lines
If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.
That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.
We could send these variables when adding the device and then validating them using a rich-text-explanatory symlink. Seems slightly backwards to me, but that would work too.
We create the char device using device_add (in nvme_init_subsystem), I didn't find any way to append env variables to that ADD uevent.
Did you mean that we should add another flavor of device_add that accepts char *envp_ext[]?
What exactly is the "standard manner" to pass these variables to the chardev KOBJ_ADD uevent? All other examples I could find use KOBJ_CHANGE to pass private stuff..
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH v2 3/3] nvme: fire discovery log page change events to userspace
Quoted 21 lines · Sagi Grimberg
Quoted 21 lines
Quoted 7 lines
Still don't understand how this is ok...
I have
/dev/nvme0represents a network endpoint that I would discover from, it is raising me an event to do a discovery operation (namely to issue an ioctl to it) so my udev code calls a systemd script.Quoted 6 lines
By the time I actually get to do that,
/dev/nvme0represents now a new network endpoint (where the event is no longer relevant to). I would rather the discovery to explicitly fail than to give me something different, so we pass some arguments that we verify in the operation.Its a stretch case, but it was raised by people as a potential issue.
Ok, and how do you handle this same thing for something like
/dev/sda? (hint, it isn't new, and is something we solved over a decade ago)If you worry about stuff like this, use a persistant device naming scheme and have your device node be pointed to by a symlink. Create that symlink by using the information in the initial 'ADD' uevent.
Quoted 8 lines
Quoted 4 lines
That way, when userspace opens the device node, it "knows" exactly which one it opens. It sounds like you have a bunch of metadata to describe these uniquely, so pass that in the ADD event, not in some other crazy non-standard manner.
We could send these variables when adding the device and then validating them using a rich-text-explanatory symlink. Seems slightly backwards to me, but that would work too.
That's the way the driver model is expected to work, instead of having to do crazy device-specific stuff.
Quoted 2 lines
We create the char device using
device_add(innvme_init_subsystem), I didn't find any way to append env variables to that ADD uevent.
You do that in your uevent or dev_uevent callback like all other subsystems. Nothing "new" to do here, again, it's been working fine for everyone else for well over a decade now :)
thanks,
greg k-h
Sagi Grimberg
maintainerv2 cover Support discovery log change events
In reply to Sagi GrimbergQuoted 8 lines
We want to be able to support discovery log change events automatically without user intervention.
The definition of discovery log change events applies on "persistent" long lived controllers, so first we need to have discovery controllers to stay for a long time and accept kato value.
Quoted 5 lines
Then when we do happen to get a discovery log change event on the persistent discovery controller, we simply fire a udev event to user-space to re-query the discovery log page and connect to new subsystems in the fabric.
This works with James' latest nvme-cli patches.
James, I've applied this to nvme-5.4
Are you sending out the nvme-cli part? If you can make sure that it works with what is in nvme-5.4 it will be great..
Thanks