dispersed namespaces revisited
Hi all,
there had been quite some discussion on various venues about dispersed namespaces on NVMe and missing linux support. Especially since it looks as if the original specification will not be implemented, yet vendors do view it as a crucial use-case. Which is already supported on other protocols like SCSI just fine, I might add. Even on Linux.
So I would like to have a discussion on where we stand, what the proposals are, and what we can do from the linux side to support the use case.
To add a bit of background:
Dispersed namespaces have been defined to support live migration of data from one subsystem to another. General idea is that the same namespace (as identified by the namespace identifier) might show up on different subsystems. This is already working on SCSI, as dm multipathing will just look as the VPD page identifcation and arrange devices based on that.
For NVMe with native multipathing this currently does not work, as
a) we're identifying namespaces with the numerical NSID
and
b) namespaces are attached to the subsystem, and can only be assembled within that subsystem.
Sure we can always switch back to device-mapper multipathing, but I don't think that's a direction we want to go. (I certainly don't.)
This discussion will be on how do we go from here; changing the spec and/or the implementation is on the table.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Hi Hannes,
the answer is pretty simple: dispersed namespaces are a bad idea and will not be implemented in Linux, and this has been the stance since this was first proposed.
If these vendors want Linux support they can already trivially implement virtual subsystems and are highly encouraged to do so. (the concept of domains actually makes it even simpler than in the beginning).
Quoted 10 lines · Christoph Hellwig
Hi Hannes,
the answer is pretty simple: dispersed namespaces are a bad idea and will not be implemented in Linux, and this has been the stance since this was first proposed.
If these vendors want Linux support they can already trivially implement virtual subsystems and are highly encouraged to do so. (the concept of domains actually makes it even simpler than in the beginning).
Guess what, that's what I have been proposing.
And they have (somewhat) agreed.
I just wanted to use LSF to get everyone on board, and then be able to come with a proposal to NVMexpress which I know will be acceptable from the Linux community.
I have a patchset ready implementing virtual subsystems based on the NS UUID (and thereby putting each namespace into a separate subsystem). At this time it's just a PoC to demonstrate the concept; specification is not even proposed, and so it's hard to code against.
I can send a pointer to my kernel.org branch if you like.
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Quoted 4 lines · Hannes Reinecke
I have a patchset ready implementing virtual subsystems based on the NS UUID (and thereby putting each namespace into a separate subsystem). At this time it's just a PoC to demonstrate the concept; specification is not even proposed, and so it's hard to code against.
Virtual subsystems already work, with one or more namespaces per subsystems. If you need to do any new code you are doing this completely wrong.
Just looks at the Linux target code - we can create subsystems at will, and a sensible configuration will have one per tenant (host or set of closely cooperating hosts). You can trivially add another controller or set of controllers on another piece of hardware from the protocol point of view. You just need to have some way to synchronize the data access just like you do for the dispersed namespace proposal.
I agree with Christoph that the current conception of dispersed namespaces fundamentally breaks the NVMe storage object model.
However, there have been several ideas proffered at FMDS to fix this and I think people are willing to make changes to TP-4034 to redress this problem.
I think this is a topic that needs to be discussed at LSF/MM or ALPSS.
/John
Quoted 25 lines · Hannes Reinecke
Quoted 10 lines · Christoph Hellwig
Hi Hannes,
the answer is pretty simple: dispersed namespaces are a bad idea and will not be implemented in Linux, and this has been the stance since this was first proposed.
If these vendors want Linux support they can already trivially implement virtual subsystems and are highly encouraged to do so. (the concept of domains actually makes it even simpler than in the beginning).
Guess what, that's what I have been proposing.
And they have (somewhat) agreed.I just wanted to use LSF to get everyone on board, and then be able to come with a proposal to NVMexpress which I know will be acceptable from the Linux community.
I have a patchset ready implementing virtual subsystems based on the NS UUID (and thereby putting each namespace into a separate subsystem). At this time it's just a PoC to demonstrate the concept; specification is not even proposed, and so it's hard to code against.
I can send a pointer to my kernel.org branch if you like.
Cheers,
Hannes
Quoted 6 lines · John Meneghini
I agree with Christoph that the current conception of dispersed namespaces fundamentally breaks the NVMe storage object model.
However, there have been several ideas proffered at FMDS to fix this and I think people are willing to make changes to TP-4034 to redress this problem.
I think this is a topic that needs to be discussed at LSF/MM or ALPSS.
NVMeoF has been designed to support more than one "virtual" subsystem behind a single port, and thus use the subsystem as a lean per-tenant container including the ability to scale and migrate it over multiple pieces hardware. And ANA and Domains have made that even easier.
So really what we need here is a clear explanation of why this does not work for the people that scream loud, and why brekaing fundamental NVMe abstractions is the way to go. This is the specific question I've asked since the TPAR was proposed but it has simply been ignored. And no, dumb implementation with a lot of legacy baggage do not count.
The array vendors need to do their homework first.