nvme: Common subsys and controller instances IDA
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
The controller char device name used to be a subset of all its namespace block device names, but that's not true when we name block devices for their parent subsystem. It's long been the case that the numerals appended to our device handles are simply the first unique number available. They do not have any more significance in relations to other devices with simpler numeric prefixes.
However, this disconnect has led to some very unfortunate situations where people assume the old behavior, and accidently have lost data because they issue admin commands to the wrong device. For example, I want to format /dev/nvme0n1 and /dev/nvme0n2, so I'll run:
# nvme format /dev/nvme0
to hit up all namespaces attached to nvme0. The problem is that nvme0's namespaces may not even be those desired namespaces, and now I've lost data on devices I wished to preserve.
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
--- drivers/nvme/host/core.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +2 -4
@@ -80,7 +80,6 @@ EXPORT_SYMBOL_GPL(nvme_reset_wq);struct workqueue_struct *nvme_delete_wq;EXPORT_SYMBOL_GPL (nvme_delete_wq); -static DEFINE_IDA (nvme_subsystems_ida);static LIST_HEAD (nvme_subsystems);static DEFINE_MUTEX (nvme_subsystems_lock);
@@ -2247,7 +2246,7 @@ static void nvme_init_subnqn(struct nvme_subsystem *subsys, struct nvme_ctrl *ctstatic void __nvme_release_subsystem (struct nvme_subsystem *subsys) { -ida_simple_remove (&nvme_subsystems_ida, subsys->instance); +ida_simple_remove (&nvme_instance_ida, subsys->instance);kfree (subsys); }
@@ -2366,7 +2365,7 @@ static int nvme_init_subsystem(struct nvme_ctrl *ctrl, struct nvme_id_ctrl *id) subsys =kzalloc (sizeof (*subsys),GFP_KERNEL );if (!subsys)return -ENOMEM ; - ret =ida_simple_get (&nvme_subsystems_ida,0 ,0 ,GFP_KERNEL ); + ret =ida_simple_get (&nvme_instance_ida,0 ,0 ,GFP_KERNEL );if (ret <0 ) {kfree (subsys);return ret;
@@ -3958,7 +3957,6 @@ static int __init nvme_core_init(void) static void __exit nvme_core_exit(void) { -ida_destroy (&nvme_subsystems_ida);class_destroy (nvme_subsys_class);class_destroy (nvme_class);unregister_chrdev_region (nvme_chr_devt,NVME_MINORS );
Signature · 1 line
2.14.4
Christoph Hellwig
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 5 lines · Keith Busch
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
Hmm. So we'll get:
/dev/nvme0
- chardev subsys X ctl 1
/dev/nvme1n1/dev/nvme1n2
- namespaces for subsys X
/dev/nvme2
- chardev subsys X ctl 2
/dev/nvme3
- chardev subsys Y ctl 1
...
This should work. Not sure it really buys us so much, though.
Btw, I wonder if we should have udev rules for:
/dev/nvme-subsys-$NQN-ctrl1/dev/nvme-subsys-$NQN-ctrl2/dev/nvme-subsys-$NQN-ns1/dev/nvme-subsys-$NQN-ns2
I have to admit that since udev/systemd moved to github I can't be bother to send them patches, though as it would require signing up for that crappy service.
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 28 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
Hmm. So we'll get:
/dev/nvme0
- chardev subsys X ctl 1
/dev/nvme1n1/dev/nvme1n2
- namespaces for subsys X
/dev/nvme2
- chardev subsys X ctl 2
/dev/nvme3
- chardev subsys Y ctl 1...
This should work. Not sure it really buys us so much, though.
Btw, I wonder if we should have udev rules for:
Quoted 9 lines
/dev/nvme-subsys-$NQN-ctrl1/dev/nvme-subsys-$NQN-ctrl2/dev/nvme-subsys-$NQN-ns1/dev/nvme-subsys-$NQN-ns2I have to admit that since udev/systemd moved to github I can't be bother to send them patches, though as it would require signing up for that crappy service.
That rather calls for persistent device names, no?
proposal would be to create symlinks like
/dev/disk/by-id/nvme-subsys-$NQN-ns-$NS
to keep in line with the other symlinks in /dev/disk/by-id/ The controller symlinks don't really fall into this category, though; we could create a 'nvme' subdirectory (much like MD does), and create symlinks in there.
And yes, I'm happy to draft up udev rules once we've come to a conclusion here.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Teamlead Storage & Networking hare at suse.de +49 911 74053 688 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 N?rnberg GF: Felix Imend?rffer, Mary Higgins, Sri Rasiah HRB 21284 (AG N?rnberg)
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 38 lines · Hannes Reinecke
Quoted 28 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
Hmm. So we'll get:
/dev/nvme0
- chardev subsys X ctl 1
/dev/nvme1n1/dev/nvme1n2
- namespaces for subsys X
/dev/nvme2
- chardev subsys X ctl 2
/dev/nvme3
- chardev subsys Y ctl 1...
This should work. Not sure it really buys us so much, though.
Btw, I wonder if we should have udev rules for:
Quoted 9 lines
/dev/nvme-subsys-$NQN-ctrl1/dev/nvme-subsys-$NQN-ctrl2/dev/nvme-subsys-$NQN-ns1/dev/nvme-subsys-$NQN-ns2I have to admit that since udev/systemd moved to github I can't be bother to send them patches, though as it would require signing up for that crappy service.
Quoted 5 lines
That rather calls for persistent device names, no?
proposal would be to create symlinks like
/dev/disk/by-id/nvme-subsys-$NQN-ns-$NS
Hm, the current by-id rules use the namespace's nguid or eui if it exists, and serial + nsid if not. We can add susbsys too.
Quoted 4 lines
to keep in line with the other symlinks in
/dev/disk/by-id/The controller symlinks don't really fall into this category, though; we could create a 'nvme' subdirectory (much like MD does), and create symlinks in there.
Cool, I would also like to see persistent char dev names, but I wasn't sure where they'd go. If we can make our own 'nvme' subdirectory, that'd be great.
Quoted 2 lines
And yes, I'm happy to draft up udev rules once we've come to a conclusion here.
Thanks!
Christoph Hellwig
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 4 lines · Keith Busch
Quoted 1 line
/dev/disk/by-id/nvme-subsys-$NQN-ns-$NSHm, the current by-id rules use the namespace's nguid or eui if it exists, and serial + nsid if not. We can add susbsys too.
Anything that supports the subsystem NQN must support euid, nguid or uuid, so I'm not sure it is worth it.
We still need good human readable names for the character devices, though.
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
In reply to Keith BuschKeith,
Quoted 8 lines
Quoted 4 lines
to keep in line with the other symlinks in
/dev/disk/by-id/The controller symlinks don't really fall into this category, though; we could create a 'nvme' subdirectory (much like MD does), and create symlinks in there.Cool, I would also like to see persistent char dev names, but I wasn't sure where they'd go. If we can make our own 'nvme' subdirectory, that'd be great.
Should be fine. As long as /dev/disk stays intact.
It would be really nice to have a clear indication of subsys/controller/namespace relationships. So much confusion as a result of the current approach...
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Hi Martin,
Quoted 6 lines · Martin K. Petersen
Keith,
Quoted 4 lines
Quoted 4 lines
to keep in line with the other symlinks in
/dev/disk/by-id/The controller symlinks don't really fall into this category, though; we could create a 'nvme' subdirectory (much like MD does), and create symlinks in there.
Quoted 8 lines
Quoted 3 lines
Cool, I would also like to see persistent char dev names, but I wasn't sure where they'd go. If we can make our own 'nvme' subdirectory, that'd be great.
Should be fine. As long as
/dev/diskstays intact.It would be really nice to have a clear indication of subsys/controller/namespace relationships. So much confusion as a result of the current approach...
you can use the nvme-cli. e.g (2 subsystems with 2 ns each exposed from target side):
Host
========
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420
nvme-subsys1 - NQN=testsubsystem_1
\
?+- nvme1 rdma traddr=11.212.140.146 trsvcid=4421
[root at server50 ~]# nvme list-ns /dev/nvme0
[?? 0]:0x1
[?? 1]:0x2
[root at server50 ~]#
[root at server50 ~]# nvme list-ns /dev/nvme1
[?? 0]:0x1
[?? 1]:0x2
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420
?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux
? +- nvme0n2 SN=cf8bbff661502c51 Model=Linux
and we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 14 lines · Max Gurtovoy
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420 ?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux ? +- nvme0n2 SN=cf8bbff661502c51 Model=Linux
Quoted 3 lines
and we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Yes, this is very useful! I'd like to add this as soon as possible. Do you have a patch, or is this just an example? The only change I'd recommend is remove SN and Model since those are controller properties rather than from the namespace.
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 15 lines · Keith Busch
Quoted 14 lines · Max Gurtovoy
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420 ?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux ? +- nvme0n2 SN=cf8bbff661502c51 Model=Linux
Quoted 7 lines
Quoted 3 lines
and we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Yes, this is very useful! I'd like to add this as soon as possible. Do you have a patch, or is this just an example? The only change I'd recommend is remove SN and Model since those are controller properties rather than from the namespace.
This is just an example but I can prepare something in a couple of days.
Let me know what do you prefer (and which attrs to print there)
-Max.
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 20 lines · Max Gurtovoy
Quoted 18 lines · Keith Busch
Quoted 17 lines · Max Gurtovoy
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420 ?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux ? +- nvme0n2 SN=cf8bbff661502c51 Model=Linuxand we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Quoted 8 lines
Quoted 4 lines
Yes, this is very useful! I'd like to add this as soon as possible. Do you have a patch, or is this just an example? The only change I'd recommend is remove SN and Model since those are controller properties rather than from the namespace.
This is just an example but I can prepare something in a couple of days.
Let me know what do you prefer (and which attrs to print there)
Let's just get the namespace handles (ex nvme0n1) displaying with each of their controllers within a subsystem. Showing just this relationship will be immediately helpful. We can can add verbose flags to get more details in the future.
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
In reply to Keith BuschQuoted 15 lines
Quoted 15 lines
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420 ?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux ? +- nvme0n2 SN=cf8bbff661502c51 Model=Linux
Quoted 8 lines
Quoted 3 lines
and we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Yes, this is very useful! I'd like to add this as soon as possible. Do you have a patch, or is this just an example? The only change I'd recommend is remove SN and Model since those are controller properties rather than from the namespace.
This now gives some context to the series Max posted :)
I still don't understand how this is helping more than the existing list-subsys for what you're trying to achieve keith.
Your use-case was "I want to understand what is my controller for /dev/nvme0n1". This information is already available in nvme list subsys. Why do you need the mpath-slave devices listed as well?
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 24 lines · Sagi Grimberg
Quoted 15 lines
Quoted 15 lines
maybe we can improve it and print also the namespace during the "nvme list-subsys" command.
something like:
[root at server50 ~]# nvme list-subsys nvme-subsys0 - NQN=testsubsystem_0
\
?+- nvme0 rdma traddr=11.212.140.146 trsvcid=4420 ?\
? +- nvme0n1 SN=cf8bbff661502c51 Model=Linux ? +- nvme0n2 SN=cf8bbff661502c51 Model=LinuxQuoted 8 lines
Quoted 3 lines
and we'll get subsys/ctrl/ns relations in 1 cmd.
thoughts ?
Yes, this is very useful! I'd like to add this as soon as possible. Do you have a patch, or is this just an example? The only change I'd recommend is remove SN and Model since those are controller properties rather than from the namespace.
Quoted 8 lines
This now gives some context to the series Max posted :)
I still don't understand how this is helping more than the existing list-subsys for what you're trying to achieve keith.
Your use-case was "I want to understand what is my controller for
/dev/nvme0n1". This information is already available in nvme list subsys. Why do you need the mpath-slave devices listed as well?
The current 'list-subsys' does not show us any namespaces, actually. You have to dig through sysfs on your own to see which namespaces belong to which controllers, but Max has added a more user friendly way to view this.
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 9 lines
Quoted 4 lines
Your use-case was "I want to understand what is my controller for
/dev/nvme0n1". This information is already available in nvme list subsys. Why do you need the mpath-slave devices listed as well?The current 'list-subsys' does not show us any namespaces, actually. You have to dig through sysfs on your own to see which namespaces belong to which controllers, but Max has added a more user friendly way to view this.
I was referring to your example in the change log:
"For example,
I want to format /dev/nvme0n1 and /dev/nvme0n2, so I'll run:
# nvme format /dev/nvme0
to hit up all namespaces attached to nvme0. The problem is that nvme0's namespaces may not even be those desired namespaces, and now I've lost data on devices I wished to preserve."
You can still run: nvme list-subsys for each namespace and understand which controller contains each volume so you can format correctly.
I personally think that this display, while more informative, is not intuitive to read, mainly because we don't get a display of the mapping between the mpath and slave devices.
Maybe for starters, when passing a namespace handle to list-subsys, the output should display only the slave devices from each controller.
RFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 20 lines · Sagi Grimberg
Quoted 9 lines
Quoted 4 lines
Your use-case was "I want to understand what is my controller for
/dev/nvme0n1". This information is already available in nvme list subsys. Why do you need the mpath-slave devices listed as well?The current 'list-subsys' does not show us any namespaces, actually. You have to dig through sysfs on your own to see which namespaces belong to which controllers, but Max has added a more user friendly way to view this.
I was referring to your example in the change log:
"For example,
I want to format/dev/nvme0n1and/dev/nvme0n2, so I'll run:? # nvme format
/dev/nvme0to hit up all namespaces attached to nvme0. The problem is that nvme0's namespaces may not even be those desired namespaces, and now I've lost data on devices I wished to preserve."
Quoted 6 lines
You can still run: nvme list-subsys for each namespace and understand which controller contains each volume so you can format correctly.
I personally think that this display, while more informative, is not intuitive to read, mainly because we don't get a display of the mapping between the mpath and slave devices.
Sagi,
my original patch showed the namespaces and not the mpath device since I developed it in non mpath environment.
Afterwards, Keith asked me to test it on mpath env and I changed my V1 to show the mpath device (in case of mpath configuration) and "IO-ble" block device in case of non mpath configuration.
I thought that this was Keith intention but afterwards we agreed to print the "IO-ble" namespace in all the cases (Also Christoph agreed on that).
So I'll update the patchset to show always the "IO-ble" namespace and not mpath slaves that a user can't do anything interested with them.
Quoted 3 lines
Maybe for starters, when passing a namespace handle to list-subsys, the output should display only the slave devices from each controller.
I see "nvme list-subsys" as "nvme list" (without any params passed). I don't see why one will pass namespace handle to it. maybe we can pass subsystem handle/name to show the description of the this specific subsystem:
nvme list-subsys -n nvme-subsys3
or for transport
nvme list-subsys -t tcp
Sagi Grimberg
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 6 lines
Sagi,
my original patch showed the namespaces and not the mpath device since I developed it in non mpath environment.
Quoted 7 lines
Afterwards, Keith asked me to test it on mpath env and I changed my V1 to show the mpath device (in case of mpath configuration) and "IO-ble" block device in case of non mpath configuration.
I thought that this was Keith intention but afterwards we agreed to print the "IO-ble" namespace in all the cases (Also Christoph agreed on that).
What's IO-ble?
Quoted 2 lines
So I'll update the patchset to show always the "IO-ble" namespace and not mpath slaves that a user can't do anything interested with them.
But a namespace in a multipath environment is not associated with a controller. I don;t understand how you can display that without confusing the user even more.
Unless they are not nested under controllers but under subsystems, that is fine, and aligned with the spec.
Quoted 5 lines
Quoted 2 lines
Maybe for starters, when passing a namespace handle to list-subsys, the output should display only the slave devices from each controller.
I see "nvme list-subsys" as "nvme list" (without any params passed). I don't see why one will pass namespace handle to it.
Because the user wants to be able to associate controller(s) with namespaces (which is exactly what keith listed in the change log!).
That user could have simply run: 'nvme list-subsys /dev/nvme0n1' and it would immediately be able to associate the controller to format.
Overall, I don't understand what this display is buying us.
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 10 lines · Sagi Grimberg
Quoted 10 lines
Sagi,
my original patch showed the namespaces and not the mpath device since I developed it in non mpath environment.
Afterwards, Keith asked me to test it on mpath env and I changed my V1 to show the mpath device (in case of mpath configuration) and "IO-ble" block device in case of non mpath configuration.
Quoted 5 lines
Quoted 3 lines
I thought that this was Keith intention but afterwards we agreed to print the "IO-ble" namespace in all the cases (Also Christoph agreed on that).
What's IO-ble?
Just means "not hidden".
Quoted 9 lines
Quoted 2 lines
So I'll update the patchset to show always the "IO-ble" namespace and not mpath slaves that a user can't do anything interested with them.
But a namespace in a multipath environment is not associated with a controller. I don;t understand how you can display that without confusing the user even more.
Unless they are not nested under controllers but under subsystems, that is fine, and aligned with the spec.
The virtual device "head" is associated with a subsystem where its device names comes from, but that head has namespace paths, which are controllers, and this is just showing that topology.
It would not make sense to nest the block device names under the subsystem because the substem may have many paths/controllers, but the namespaces are accessible only under a subset of those paths.
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
In reply to Christoph HellwigQuoted 19 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
Hmm. So we'll get:
/dev/nvme0
- chardev subsys X ctl 1
/dev/nvme1n1/dev/nvme1n2
- namespaces for subsys X
/dev/nvme2
- chardev subsys X ctl 2
Quoted 6 lines
/dev/nvme3
- chardev subsys Y ctl 1...
This should work. Not sure it really buys us so much, though.
Right, it's not much. The only thing I want accomplish with this is to remove any excuse for sending commands to the wrong handle. We can insist that it's wrong
Quoted 10 lines
Btw, I wonder if we should have udev rules for:
/dev/nvme-subsys-$NQN-ctrl1/dev/nvme-subsys-$NQN-ctrl2/dev/nvme-subsys-$NQN-ns1/dev/nvme-subsys-$NQN-ns2I have to admit that since udev/systemd moved to github I can't be bother to send them patches, though as it would require signing up for that crappy service.
I can send it on your behalf if you have a rule in mind.
We do already have /dev/disk/by-{path,id,uuid} rules already, if that's what you're looking for.
We don't have rules for the chardev though. Those devices are skipped in 60-persistent-naming.rules.
Keith Busch
maintainerRFC PATCH 1/1 nvme: Common subsys and controller instances IDA
Quoted 23 lines · Keith Busch
Quoted 22 lines · Christoph Hellwig
Quoted 5 lines · Keith Busch
So here's a solution that no one will like: pull subsystem and controller instances from the same IDA so that there won't be any namespace block devices with a matching controller handle name. While this does nothing to clear up device relationships, this will force the user to think really hard about what they're doing and avoid such mistakes.
Hmm. So we'll get:
/dev/nvme0
- chardev subsys X ctl 1
/dev/nvme1n1/dev/nvme1n2
- namespaces for subsys X
/dev/nvme2
- chardev subsys X ctl 2
/dev/nvme3
- chardev subsys Y ctl 1
Quoted 6 lines
Quoted 3 lines
...
This should work. Not sure it really buys us so much, though.
Right, it's not much. The only thing I want accomplish with this is to remove any excuse for sending commands to the wrong handle. We can insist that it's wrong
[finishing the sentence]
We can insist that it's wrong for anyone to have assumed a chardev is the controller of a specific blkdev, but amount of feedback I've received indicates the current naming is a little too easy for people to shoot themselves in the foot. This new naming will put a stop to such mistakes without breaking anything.