nvme: Add kernel API for admin command
cover nvme: Add kernel API for admin command
Hello,
This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.
[1] https://github.com/Open-CAS/open-cas-linux
Best regards,
Robert Baldyga
Michal Rakowski (1):
nvme: add API for sending admin commands by bdev
Robert Baldyga (1):
nvme: add API for getting nsid by bdev
drivers/nvme/host/core.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/nvme.h | 5 +++++ 2 files changed, 42 insertions(+)
Signature · 11 lines
2.17.1 -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
PATCH 1/2 nvme: add API for sending admin commands by bdev
From: Michal Rakowski <[email protected]>
Add kernel API function for sending nvme admin commands.
--- drivers/nvme/host/core.c | 23 +++++++++++++++++++++++ include/linux/nvme.h | 3 +++ 2 files changed, 26 insertions(+)
drivers/nvme/host/core.c +23 -0
@@ -812,6 +812,29 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, }EXPORT_SYMBOL_GPL (nvme_submit_sync_cmd); +int nvme_submit_admin_cmd_by_bdev (struct block_device *bdev, +struct nvme_command *c,void *buffer,unsigned int bufflen) +{ +struct nvme_ns *ns; +struct nvme_ctrl *ctrl; +int error; + +if (!bdev && !c) +return -EINVAL ; + + ns = bdev->bd_disk->private_data; + ctrl = ns->ctrl; + + error =nvme_submit_sync_cmd (ctrl->admin_q, c, buffer, bufflen); +if (error) { +dev_warn (ctrl->device,"Admin command failed (%d)\n" , error); +return error; + } + +return error; +} +EXPORT_SYMBOL_GPL (nvme_submit_admin_cmd_by_bdev); +static void *nvme_add_user_metadata (struct bio *bio,void __user *ubuf,unsigned len,u32 seed,bool write) {
include/linux/nvme.h +3 -0
@@ -1392,4 +1392,7 @@ struct nvme_completion {#define NVME_MINOR (ver) (((ver) >> 8) & 0xff)#define NVME_TERTIARY (ver) ((ver) & 0xff) +int nvme_submit_admin_cmd_by_bdev (struct block_device *bdev, +struct nvme_command *c,void *buffer,unsigned int bufflen); +#endif /* _LINUX_NVME_H */
Signature · 11 lines
2.17.1 -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
kbuild test robot
build botRe: [PATCH 1/2] nvme: add API for sending admin commands by bdev
Thank you for the patch! Perhaps something to improve:
Hi Robert,
Thank you for the patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[cannot apply to v5.3 next-20190915]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Robert-Baldyga/nvme-Add-kernel-API-for-admin-command/20190916-134358
config: sparc64-allmodconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=sparc64
If you fix the issue, kindly add following tag
All warnings (new ones prefixed by >>):
In file included from drivers/nvme/host/nvme.h:9:0,
from drivers/nvme/host/fault_inject.c:9:
>> include/linux/nvme.h:1395:42: warning: 'struct block_device' declared inside parameter list will not be visible outside of this definition or declaration
int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
^~~~~~~~~~~~
vim +1395 include/linux/nvme.h
1394
> 1395 int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
1396 struct nvme_command *c, void *buffer, unsigned int bufflen);
1397
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
PATCH 2/2 nvme: add API for getting nsid by bdev
In reply to Robert BaldygaAdd kernel API function for getting nvme namespace id.
--- drivers/nvme/host/core.c | 14 ++++++++++++++ include/linux/nvme.h | 2 ++ 2 files changed, 16 insertions(+)
drivers/nvme/host/core.c +14 -0
@@ -812,6 +812,20 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd, }EXPORT_SYMBOL_GPL (nvme_submit_sync_cmd); +int nvme_get_nsid_by_bdev (struct block_device *bdev,unsigned int *nsid) +{ +struct nvme_ns *ns; + +if (!bdev && !nsid) +return -EINVAL ; + + ns = bdev->bd_disk->private_data; + *nsid = ns->head->ns_id; + +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_nsid_by_bdev); +int nvme_submit_admin_cmd_by_bdev (struct block_device *bdev,struct nvme_command *c,void *buffer,unsigned int bufflen) {
include/linux/nvme.h +2 -0
@@ -1392,6 +1392,8 @@ struct nvme_completion {#define NVME_MINOR (ver) (((ver) >> 8) & 0xff)#define NVME_TERTIARY (ver) ((ver) & 0xff) +int nvme_get_nsid_by_bdev (struct block_device *bdev,unsigned int *nsid); +int nvme_submit_admin_cmd_by_bdev (struct block_device *bdev,struct nvme_command *c,void *buffer,unsigned int bufflen);
Signature · 11 lines
2.17.1 -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
kbuild test robot
build botRe: [PATCH 2/2] nvme: add API for getting nsid by bdev
I love your patch! Yet something to improve:
Hi Robert,
I love your patch! Yet something to improve:
[auto build test ERROR on linus/master]
[cannot apply to v5.3 next-20190915]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Robert-Baldyga/nvme-Add-kernel-API-for-admin-command/20190916-134358
config: sparc64-allmodconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=sparc64
If you fix the issue, kindly add following tag
All error/warnings (new ones prefixed by >>):
In file included from include/linux/linkage.h:7:0,
from include/linux/kernel.h:8, from include/asm-generic/bug.h:18, from arch/sparc/include/asm/bug.h:25, from include/linux/bug.h:5, from include/linux/thread_info.h:12, from arch/sparc/include/asm/current.h:15, from include/linux/sched.h:12, from include/linux/blkdev.h:5, from drivers/nvme/host/core.c:7:
>> drivers/nvme/host/core.c:827:19: error: 'nvme_nsid_by_bdev' undeclared here (not in a function); did you mean 'nvme_get_nsid_by_bdev'?
EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
^
include/linux/export.h:79:16: note: in definition of macro '___EXPORT_SYMBOL'
extern typeof(sym) sym; \
^~~
>> drivers/nvme/host/core.c:827:1: note: in expansion of macro 'EXPORT_SYMBOL_GPL'
EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
^~~~~~~~~~~~~~~~~vim +827 drivers/nvme/host/core.c
814
815 int nvme_get_nsid_by_bdev(struct block_device *bdev, unsigned int *nsid)
816 {
817 struct nvme_ns *ns;
818
819 if (!bdev && !nsid)
820 return -EINVAL;
821
822 ns = bdev->bd_disk->private_data;
823 *nsid = ns->head->ns_id;
824
825 return 0;
826 }
> 827 EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
828
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Christoph Hellwig
maintainerQuoted 7 lines · Robert Baldyga
Hello,
This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like
nvme_admin_identify.
We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.
Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga
Hello,
This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like
nvme_admin_identify.We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.
So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?
Quoted 2 lines
Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?
Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.
If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.
Best regards,
Robert Baldyga
--------------------------------------------------------------------
Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.
Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
Christoph Hellwig
maintainerQuoted 10 lines · Robert
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga
Hello,
This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like
nvme_admin_identify.
Quoted 6 lines
Quoted 2 lines
We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.
So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?
Because you added another badly designed caching layer instead of fixing up one of the 2 to 3 (depending on how you count) in the kernel tree. And yours is amazingly bad even compared to the not very nice one in the tree. It basically spends more efforts on abstracting away abstraction that you wouldn't need without another layer of abstractions. And it does all that pointlessly tied to NVMe through a bunch of layering violations.
Quoted 6 lines
Quoted 3 lines
Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?
Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.
It is not. We will not let random kernel modules directly issue nvme admin command as there is no reason for it. And even if for some weird reason we did we'd certainly not do it for out of tree modules.
Quoted 4 lines
If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.
And the answer is that we are not going to allow that. And we'd not allow other things even if they were not a bad design for out of tree modules.
Quoted 30 lines · Christoph Hellwig
Quoted 10 lines · Robert
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga
Hello,
This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like
nvme_admin_identify.Quoted 6 lines
Quoted 2 lines
We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.
So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?
Because you added another badly designed caching layer instead of fixing up one of the 2 to 3 (depending on how you count) in the kernel tree. And yours is amazingly bad even compared to the not very nice one in the tree. It basically spends more efforts on abstracting away abstraction that you wouldn't need without another layer of abstractions. And it does all that pointlessly tied to NVMe through a bunch of layering violations.
Quoted 4 lines
Quoted 3 lines
Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?
Quoted 6 lines
Quoted 2 lines
Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.
It is not. We will not let random kernel modules directly issue nvme admin command as there is no reason for it. And even if for some weird reason we did we'd certainly not do it for out of tree modules.
Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.
Quoted 8 lines
Quoted 4 lines
If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.
And the answer is that we are not going to allow that. And we'd not allow other things even if they were not a bad design for out of tree modules.
I'm not going to insist on merging changes dedicated for out of tree modules. I'd be happy to get involved in improving kernel support for NVMe so that there would be no need for that changes. What I proposed is a quick fix for the problem that, for sure, can be solved another, better way.
And this is not about being in tree or out of tree - a lot of out of tree modules are doing well without any special care from the upstream. This is about lack of support for certain features. If you have any proposal how we should solve this problem in a better way than I proposed in my humble patchset, I will gladly submit new patches.
Best regards,
Robert Baldyga
Keith Busch
maintainerQuoted 7 lines · Robert
Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.
I think your quickest path to supporting such a format is to consider each part separately, then bounce and interleave/unmix the data + metadata at another layer that understands how the data needs to be laid out in host memory. Something like this RFC here:
http://lists.infradead.org/pipermail/linux-nvme/2018-February/015844.html
It appears connecting to infradead lists is a bit flaky at the moment, so not sure if you'll be able to read the above link right now.
Anyway, the RFC would have needed a bit of work to be considered, like using a mempool for the purpose, but it does generically make such formats usable through the block stack so maybe try flushing out that idea.
Christoph Hellwig
maintainerQuoted 16 lines · Keith Busch
Quoted 7 lines · Robert
Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.
I think your quickest path to supporting such a format is to consider each part separately, then bounce and interleave/unmix the data + metadata at another layer that understands how the data needs to be laid out in host memory. Something like this RFC here:
http://lists.infradead.org/pipermail/linux-nvme/2018-February/015844.html
Quoted 7 lines
It appears connecting to infradead lists is a bit flaky at the moment, so not sure if you'll be able to read the above link right now.
Anyway, the RFC would have needed a bit of work to be considered, like using a mempool for the purpose, but it does generically make such formats usable through the block stack so maybe try flushing out that idea.
Even if we had a use case for that the bounce buffer is just too ugly to live. And I'm really sick and tired of Intel wasting our time for their out of tree monster given that they haven't even tried helping to improve the in-kernel write caching layers.
Keith Busch
maintainerQuoted 4 lines · Christoph Hellwig
Even if we had a use case for that the bounce buffer is just too ugly to live. And I'm really sick and tired of Intel wasting our time for their out of tree monster given that they haven't even tried helping to improve the in-kernel write caching layers.
Right, I don't have any stake in that out-of-tree caching either. I am more just interested in trying to get Linux to generically reach as many NVMe spec features as possible, and extended formats have been in-spec since the beginning of nvme.
And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.
Christoph Hellwig
maintainerQuoted 4 lines · Keith Busch
And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.
Well, what is the point? If people really want to use metadata they should just buy a drive supporting the separate metadata pointer. In fact I haven't had to deal with a drive that only supports interleaved metadata so far given how awkward that is to deal with. But based on the discussions here it seems Intel is stupid enough to ship such a thing.
Christoph,
Quoted 10 lines · Keith Busch
Quoted 4 lines
And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.
Well, what is the point? If people really want to use metadata they should just buy a drive supporting the separate metadata pointer. In fact I haven't had to deal with a drive that only supports interleaved metadata so far given how awkward that is to deal with.
Yep. There's a reason we did DIX...
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering