Skip to contentSkip to conversation

nvme: Add kernel API for admin command

PATCH In review Robert Baldyga 5 participants 14 messages Sep 13, 2019 – Sep 19, 2019 NVMe

lore
Share & tools
lore
#0 lore raw

cover nvme: Add kernel API for admin command

Hello,

This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.

[1] https://github.com/Open-CAS/open-cas-linux

Best regards,
Robert Baldyga

Michal Rakowski (1):
nvme: add API for sending admin commands by bdev

Robert Baldyga (1):
nvme: add API for getting nsid by bdev

 drivers/nvme/host/core.c | 37 +++++++++++++++++++++++++++++++++++++
 include/linux/nvme.h     |  5 +++++
 2 files changed, 42 insertions(+)
Signature · 11 lines
2.17.1

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
#1 lore raw

PATCH 1/2 nvme: add API for sending admin commands by bdev

From: Michal Rakowski <[email protected]>

Add kernel API function for sending nvme admin commands.

---
 drivers/nvme/host/core.c | 23 +++++++++++++++++++++++
 include/linux/nvme.h     |  3 +++
 2 files changed, 26 insertions(+)
drivers/nvme/host/core.c +23 -0
index d3d6b7bd6903..06f917f391c4 100644
@@ -812,6 +812,29 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 }
 EXPORT_SYMBOL_GPL(nvme_submit_sync_cmd);
 
+int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
+		struct nvme_command *c, void *buffer, unsigned int bufflen)
+{
+	struct nvme_ns *ns;
+	struct nvme_ctrl *ctrl;
+	int error;
+
+	if (!bdev && !c)
+		return -EINVAL;
+
+	ns = bdev->bd_disk->private_data;
+	ctrl = ns->ctrl;
+
+	error = nvme_submit_sync_cmd(ctrl->admin_q, c, buffer, bufflen);
+	if (error) {
+		dev_warn(ctrl->device, "Admin command failed (%d)\n", error);
+		return error;
+	}
+
+	return error;
+}
+EXPORT_SYMBOL_GPL(nvme_submit_admin_cmd_by_bdev);
+
 static void *nvme_add_user_metadata(struct bio *bio, void __user *ubuf,
 		unsigned len, u32 seed, bool write)
 {
include/linux/nvme.h +3 -0
index 01aa6a6c241d..6f26c5654514 100644
@@ -1392,4 +1392,7 @@ struct nvme_completion {
 #define NVME_MINOR(ver)		(((ver) >> 8) & 0xff)
 #define NVME_TERTIARY(ver)	((ver) & 0xff)
 
+int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
+		struct nvme_command *c, void *buffer, unsigned int bufflen);
+
 #endif /* _LINUX_NVME_H */
Signature · 11 lines
2.17.1

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
#2 lore raw

Re: [PATCH 1/2] nvme: add API for sending admin commands by bdev

Thank you for the patch! Perhaps something to improve:

Hi Robert,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[cannot apply to v5.3 next-20190915]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Robert-Baldyga/nvme-Add-kernel-API-for-admin-command/20190916-134358
config: sparc64-allmodconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=sparc64

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

In file included from drivers/nvme/host/nvme.h:9:0,
from drivers/nvme/host/fault_inject.c:9:

>> include/linux/nvme.h:1395:42: warning: 'struct block_device' declared inside parameter list will not be visible outside of this definition or declaration

int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
^~~~~~~~~~~~

vim +1395 include/linux/nvme.h

1394

> 1395	int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,

1396 struct nvme_command *c, void *buffer, unsigned int bufflen);
1397

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#3 lore raw

PATCH 2/2 nvme: add API for getting nsid by bdev

In reply to Robert Baldyga

Add kernel API function for getting nvme namespace id.

---
 drivers/nvme/host/core.c | 14 ++++++++++++++
 include/linux/nvme.h     |  2 ++
 2 files changed, 16 insertions(+)
drivers/nvme/host/core.c +14 -0
index 06f917f391c4..35d121cd2378 100644
@@ -812,6 +812,20 @@ int nvme_submit_sync_cmd(struct request_queue *q, struct nvme_command *cmd,
 }
 EXPORT_SYMBOL_GPL(nvme_submit_sync_cmd);
 
+int nvme_get_nsid_by_bdev(struct block_device *bdev, unsigned int *nsid)
+{
+	struct nvme_ns *ns;
+
+	if (!bdev && !nsid)
+		return -EINVAL;
+
+	ns = bdev->bd_disk->private_data;
+	*nsid = ns->head->ns_id;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
+
 int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
 		struct nvme_command *c, void *buffer, unsigned int bufflen)
 {
include/linux/nvme.h +2 -0
index 6f26c5654514..c54e210ad271 100644
@@ -1392,6 +1392,8 @@ struct nvme_completion {
 #define NVME_MINOR(ver)		(((ver) >> 8) & 0xff)
 #define NVME_TERTIARY(ver)	((ver) & 0xff)
 
+int nvme_get_nsid_by_bdev(struct block_device *bdev, unsigned int *nsid);
+
 int nvme_submit_admin_cmd_by_bdev(struct block_device *bdev,
 		struct nvme_command *c, void *buffer, unsigned int bufflen);
 
Signature · 11 lines
2.17.1

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.
#4 lore raw

Re: [PATCH 2/2] nvme: add API for getting nsid by bdev

I love your patch! Yet something to improve:

Hi Robert,

I love your patch! Yet something to improve:

[auto build test ERROR on linus/master]
[cannot apply to v5.3 next-20190915]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Robert-Baldyga/nvme-Add-kernel-API-for-admin-command/20190916-134358
config: sparc64-allmodconfig (attached as .config)
compiler: sparc64-linux-gcc (GCC) 7.4.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=7.4.0 make.cross ARCH=sparc64

If you fix the issue, kindly add following tag

All error/warnings (new ones prefixed by >>):

In file included from include/linux/linkage.h:7:0,
from include/linux/kernel.h:8, from include/asm-generic/bug.h:18, from arch/sparc/include/asm/bug.h:25, from include/linux/bug.h:5, from include/linux/thread_info.h:12, from arch/sparc/include/asm/current.h:15, from include/linux/sched.h:12, from include/linux/blkdev.h:5, from drivers/nvme/host/core.c:7:

>> drivers/nvme/host/core.c:827:19: error: 'nvme_nsid_by_bdev' undeclared here (not in a function); did you mean 'nvme_get_nsid_by_bdev'?
    EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
                      ^
   include/linux/export.h:79:16: note: in definition of macro '___EXPORT_SYMBOL'
     extern typeof(sym) sym;      \
                   ^~~
>> drivers/nvme/host/core.c:827:1: note: in expansion of macro 'EXPORT_SYMBOL_GPL'
    EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);
    ^~~~~~~~~~~~~~~~~

vim +827 drivers/nvme/host/core.c

814
815 int nvme_get_nsid_by_bdev(struct block_device *bdev, unsigned int *nsid)
816 {
817 struct nvme_ns *ns;
818
819 if (!bdev && !nsid)
820 return -EINVAL;
821
822 ns = bdev->bd_disk->private_data;
823 *nsid = ns->head->ns_id;
824
825 return 0;
826 }

 > 827	EXPORT_SYMBOL_GPL(nvme_nsid_by_bdev);

828

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#5 lore raw
In reply to Robert Baldyga
Quoted 7 lines · Robert Baldyga

Hello,

This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.

We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.

Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?

#6 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga

Hello,

This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.

We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.

So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?

Quoted 2 lines

Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?

Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.

If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.

Best regards,
Robert Baldyga
--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.

#7 lore raw
Quoted 10 lines · Robert
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga

Hello,

This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.

Quoted 6 lines
Quoted 2 lines

We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.

So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?

Because you added another badly designed caching layer instead of fixing up one of the 2 to 3 (depending on how you count) in the kernel tree. And yours is amazingly bad even compared to the not very nice one in the tree. It basically spends more efforts on abstracting away abstraction that you wouldn't need without another layer of abstractions. And it does all that pointlessly tied to NVMe through a bunch of layering violations.

Quoted 6 lines
Quoted 3 lines

Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?

Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.

It is not. We will not let random kernel modules directly issue nvme admin command as there is no reason for it. And even if for some weird reason we did we'd certainly not do it for out of tree modules.

Quoted 4 lines

If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.

And the answer is that we are not going to allow that. And we'd not allow other things even if they were not a bad design for out of tree modules.

#8 lore raw
Quoted 30 lines · Christoph Hellwig
Quoted 10 lines · Robert
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Robert Baldyga

Hello,

This patchset adds two functions providing kernel to kernel API for submiting NVMe admin commands. This is for use of NVMe-aware block device drivers stacking on top of NVMe drives. An example of such driver is Open CAS Linux [1] which uses NVMe extended LBA formats and thus needs to issue commands like nvme_admin_identify.

Quoted 6 lines
Quoted 2 lines

We never add functionality for out of tree crap. And this shit really is a bunch of crap, so it is unlikely to ever be merged.

So that modules which are by design out of tree have to hack around lack of API allowing to use functionality implemented by driver. Don't you think that this is what actually produces crap?

Because you added another badly designed caching layer instead of fixing up one of the 2 to 3 (depending on how you count) in the kernel tree. And yours is amazingly bad even compared to the not very nice one in the tree. It basically spends more efforts on abstracting away abstraction that you wouldn't need without another layer of abstractions. And it does all that pointlessly tied to NVMe through a bunch of layering violations.

Quoted 4 lines
Quoted 3 lines

Why can't intel sometimes actually do something useful for a change instead of piling junk over junk?

Quoted 6 lines
Quoted 2 lines

Proposed API is equally useful for both in tree and out of tree modules, so I find your comment unrelated.

It is not. We will not let random kernel modules directly issue nvme admin command as there is no reason for it. And even if for some weird reason we did we'd certainly not do it for out of tree modules.

Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.

Quoted 8 lines
Quoted 4 lines

If you don't like the way it's done, we can look for alternatives. The point is to allow other drivers use NVMe admin commands, which is currently not possible as neither the block layer nor the nvme driver provides sufficient API.

And the answer is that we are not going to allow that. And we'd not allow other things even if they were not a bad design for out of tree modules.

I'm not going to insist on merging changes dedicated for out of tree modules. I'd be happy to get involved in improving kernel support for NVMe so that there would be no need for that changes. What I proposed is a quick fix for the problem that, for sure, can be solved another, better way.

And this is not about being in tree or out of tree - a lot of out of tree modules are doing well without any special care from the upstream. This is about lack of support for certain features. If you have any proposal how we should solve this problem in a better way than I proposed in my humble patchset, I will gladly submit new patches.

Best regards,
Robert Baldyga

maintainer
#9 lore raw
Quoted 7 lines · Robert

Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.

I think your quickest path to supporting such a format is to consider each part separately, then bounce and interleave/unmix the data + metadata at another layer that understands how the data needs to be laid out in host memory. Something like this RFC here:

http://lists.infradead.org/pipermail/linux-nvme/2018-February/015844.html

It appears connecting to infradead lists is a bit flaky at the moment, so not sure if you'll be able to read the above link right now.

Anyway, the RFC would have needed a bit of work to be considered, like using a mempool for the purpose, but it does generically make such formats usable through the block stack so maybe try flushing out that idea.

#10 lore raw
Quoted 16 lines · Keith Busch
Quoted 7 lines · Robert

Ok, fair enough. We want to keep things hidden behind certain layers, and that's definitely a good thing. But there is a problem with these layers - they do not expose all the features. For example AFAIK there is no clear way to use 512+8 format via block layer (nor even a way to get know that bdev is formatted to particular lbaf). It's impossible to use it without layering violations, which nobody sees as a perfect solution, but it is the only one that works.

I think your quickest path to supporting such a format is to consider each part separately, then bounce and interleave/unmix the data + metadata at another layer that understands how the data needs to be laid out in host memory. Something like this RFC here:

http://lists.infradead.org/pipermail/linux-nvme/2018-February/015844.html

Quoted 7 lines

It appears connecting to infradead lists is a bit flaky at the moment, so not sure if you'll be able to read the above link right now.

Anyway, the RFC would have needed a bit of work to be considered, like using a mempool for the purpose, but it does generically make such formats usable through the block stack so maybe try flushing out that idea.

Even if we had a use case for that the bounce buffer is just too ugly to live. And I'm really sick and tired of Intel wasting our time for their out of tree monster given that they haven't even tried helping to improve the in-kernel write caching layers.

maintainer
#11 lore raw
Quoted 4 lines · Christoph Hellwig

Even if we had a use case for that the bounce buffer is just too ugly to live. And I'm really sick and tired of Intel wasting our time for their out of tree monster given that they haven't even tried helping to improve the in-kernel write caching layers.

Right, I don't have any stake in that out-of-tree caching either. I am more just interested in trying to get Linux to generically reach as many NVMe spec features as possible, and extended formats have been in-spec since the beginning of nvme.

And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.

#12 lore raw
Quoted 4 lines · Keith Busch

And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.

Well, what is the point? If people really want to use metadata they should just buy a drive supporting the separate metadata pointer. In fact I haven't had to deal with a drive that only supports interleaved metadata so far given how awkward that is to deal with. But based on the discussions here it seems Intel is stupid enough to ship such a thing.

#13 lore raw

Christoph,

Quoted 10 lines · Keith Busch
Quoted 4 lines

And yes, that bouncing is really nasty, but it's really only needed for PRP, so maybe let's just not ignore that transfer mode and support extended metadata iff the controller supports SGLs. We just need a special SGL setup routine to weave the data and metadata.

Well, what is the point? If people really want to use metadata they should just buy a drive supporting the separate metadata pointer. In fact I haven't had to deal with a drive that only supports interleaved metadata so far given how awkward that is to deal with.

Yep. There's a reason we did DIX...

Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering