Native multipath across multiple subsystem NQNs
Hello,
We have the need to expose the same namespace (considered the same due to reporting the same unique identifier in "Identify Namespace") from two distinct targets, having different subsystem NQNs. In this case, the native multipath implementation in this driver behaves differently from most other multipath implementations, including DM-multipath and the implementations in other operating systems.
Native multipath: Does not consider the namespaces the same. Two block devices are created, corresponding to the two subsystems in play.
Others: The namespaces are considered the same. One block device is created, backed by paths to both subsystems.
The second behavior is desired in our use case. I want to implement this behavior in the driver. I can think of two approaches, and want some feedback on which one(s) would be acceptable.
1. A "direct" approach: add a parameter, which, when set, will result in the second behavior - paths to namespaces are consolidated purely based on namespace unique ID, without considering the subsystem.
2. The recently ratified TPE4034 Dispersed Namespaces feature provides a means to get the second behavior, and a lot more. It requires both host and target support.
I look forward to your feedback!
Thanks,
Uday
Christoph Hellwig
maintainerThe answer is neither. The fact that NVMe with ANA required namespaces to be be attached to the same subsystem is good and important. TPE 4034 is retrograde and completely broken in this respect and should have never been ratifier. Please fix your storage system to export virtual subsystems like everyone else and don't break the nice NVMe architecture.
Keith Busch
maintainerQuoted 6 lines · Christoph Hellwig
The answer is neither. The fact that NVMe with ANA required namespaces to be be attached to the same subsystem is good and important. TPE 4034 is retrograde and completely broken in this respect and should have never been ratifier. Please fix your storage system to export virtual subsystems like everyone else and don't break the nice NVMe architecture.
If anything, the driver could check for duplicate namespace identifiers across subsystems so that we could warn about or prevent exposing them.
Christoph Hellwig
maintainerQuoted 10 lines · Keith Busch
Quoted 6 lines · Christoph Hellwig
The answer is neither. The fact that NVMe with ANA required namespaces to be be attached to the same subsystem is good and important. TPE 4034 is retrograde and completely broken in this respect and should have never been ratifier. Please fix your storage system to export virtual subsystems like everyone else and don't break the nice NVMe architecture.
If anything, the driver could check for duplicate namespace identifiers across subsystems so that we could warn about or prevent exposing them.
Yes, this has actually been something I've been wanting to do for a long time.
Pulling Randy into this thread.
Quoted 6 lines
The fact that NVMe with ANA required namespaces
to be be attached to the same subsystem is good and important. TPE 4034 is retrograde and completely broken in this respect and should have never been ratifier. Please fix your storage system to export virtual subsystems like everyone else and don't break the nice NVMe architecture.
Supposing we did implement virtual subsystems to handle exposing namespaces on multiple arrays.
In addition to other considerations, migrating a namespace with hot data from a different storage vendor array nondisruptive to client i/o would be difficult to do without having a namespace exist under multiple NQNs. One approach that has been used for SCSI is by multipathing to both targets & a proxy mirroring writes; on cut-over, the other path disconnects. If the filehandle has to change, that is disruptive to the client software.
Unifying different storage arrays behind the same NQN/virtual subsystem requires coordination of nvme ctrl_ids at least, and doing that between different storage vendor arrays is unlikely. Having a mechanism to migrate between different JBOF devices non-disruptively would be helpful regardless of the source/destination vendors. Such devices will probably not have the option of virtual subsystems.
Additionally, the granularity at which NQNs/subsystems are exposed to hosts affects how many connections the host creates with the controller. Each connection has a cost in hardware & software.
In other words, even with implementing virtual subsystems, we still have use for non-disruptively moving a namespace between subsystems. How will this usecase be supported on Linux?
Sincerely,
Randy Jennings
Christoph Hellwig
maintainerQuoted 8 lines · Randy Jennings
In addition to other considerations, migrating a namespace with hot data from a different storage vendor array nondisruptive to client i/o would be difficult to do without having a namespace exist under multiple NQNs. One approach that has been used for SCSI is by multipathing to both targets & a proxy mirroring writes; on cut-over, the other path disconnects. If the filehandle has to change, that is disruptive to the client software.
Quoted 6 lines
Unifying different storage arrays behind the same NQN/virtual subsystem requires coordination of nvme
ctrl_idsat least, and doing that between different storage vendor arrays is unlikely. Having a mechanism to migrate between different JBOF devices non-disruptively would be helpful regardless of the source/destination vendors. Such devices will probably not have the option of virtual subsystems.
Yes, it does require coordination, so please coordinate.
Quoted 3 lines
In other words, even with implementing virtual subsystems, we still have use for non-disruptively moving a namespace between subsystems. How will this usecase be supported on Linux?
You can create a mirror using dm (or md but not as easily), resync and
then cut off a leg. If you want to do that transparently on an
already existing device node you'll need to pick up the block device interposer patchset and help to get it upstream.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Christoph Hellwig
maintainerHi Prashanth,
your mail client completely messed up the quoting so I could not easily find what you actually wrote inbetween the full quote.
Quoted 15 lines · Christoph Hellwig
Quoted 8 lines · Randy Jennings
In addition to other considerations, migrating a namespace with hot data from a different storage vendor array nondisruptive to client i/o would be difficult to do without having a namespace exist under multiple NQNs. One approach that has been used for SCSI is by multipathing to both targets & a proxy mirroring writes; on cut-over, the other path disconnects. If the filehandle has to change, that is disruptive to the client software.
Quoted 12 lines
Quoted 6 lines
Unifying different storage arrays behind the same NQN/virtual subsystem requires coordination of nvme
ctrl_idsat least, and doing that between different storage vendor arrays is unlikely. Having a mechanism to migrate between different JBOF devices non-disruptively would be helpful regardless of the source/destination vendors. Such devices will probably not have the option of virtual subsystems.Yes, it does require coordination, so please coordinate.
Quoted 3 lines
In other words, even with implementing virtual subsystems, we still have use for non-disruptively moving a namespace between subsystems. How will this usecase be supported on Linux?
Implementing the same subsystem NQN on multiple storage arrays imposes excessive architecture/design costs on the target. In addition to implementing the same subsystem NQN, the target has to ensure that all components that make up the virtual subsystem (across multiple arrays) have
the same inventory (same namespaces with matching NSIDs and ANA groups). This needs to be kept consistent even when there are communication failures between those components of the virtual subsystem.
The virtual subsystem would also be required to have the same host access control settings and similar QoS settings etc on all of the components that make up the virtual subsystem. This coordination, in addition to what is required for controller IDs, is extremely complicated when both the arrays belong to the same vendor and next to impossible, if in the future, a multi-vendor solution is developed.
TP 4034 is designed to specifically address these migration/business continuity solutions, while minimizing all of that complexity in the target. So it's quite useful in that aspect.
Can you please shed some light on why you consider TP 4034 retrograde? What issues are you concerned about if Linux nvme were to handle namespaces shared across subsystems?
Prashanth
Christoph Hellwig
maintainerHi Prashanth,
thanks for adding Fred who apparently complained to the NVMe BOD about this issue, and who really needs some shaming from Greg about trying to use corporate means for forcing Linux developers into implementing broken standards.
Quoted 2 lines · Prashanth
Implementing the same subsystem NQN on multiple storage arrays imposes excessive architecture/design costs on the target.
While on the other hand implementing TP4034 forces excessive architecture, design, maintainance and debugging costs on the host. That's why I've told the NVMe technical working group on the very day that this TPAR was brought in that we are not going to support it, and I've also heard feeback from various other host implementators that there is very little interest in it.
Quoted 13 lines
In addition to implementing
the same subsystem NQN, the target has to ensure that all components that make up the virtual subsystem (across multiple arrays) have
the same inventory (same namespaces with matching NSIDs and ANA groups). This needs to be kept consistent even when there are communication failures between those components of the virtual subsystem.
The virtual subsystem would also be required to have the same host access control settings and similar QoS settings etc on all of the components that make up the virtual subsystem. This coordination, in addition to what is required for controller IDs, is extremely complicated when both the arrays belong to the same vendor and next to impossible, if in the future, a multi-vendor solution is developed.
Yes, and that is the whole point.
Quoted 1 line
Can you please shed some light on why you consider TP 4034 retrograde? What issues are you concerned about if Linux nvme were to handle namespaces shared across subsystems?
NVMe has a sensible object model that allows us for easy sanity checking of duplicate ids and more importantly for proper discovery. We do not want to lose that, nor do want to break the sysfs hierachy that directly results from encoding the nvme architecture model in the Linux device model.