nvme: add cse, ds, ms, nsze and nuse to sysfs
RFC 1/1 nvme: add cse, ds, ms, nsze and nuse to sysfs
libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.
As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.
---
As discussed during ALPPS, these here are the missing attribures which libnvme is still looking up via commands. I've tested this with a modified libnvme and didn't observe any ioctls anymore.
I'm pretty sure the naming is a bit off for the variables. Not really sure if we want to stick to the spec naming sceme or have our own one, e.g. 'nsze' vs 'capacity'.
Also getting a pointer to the nvme_ns data structure is a bit strange (dev_to_nvme_ns). This stip is necessary as many of the ns attributes are in nvme_ns. Shouldn't these per path values not all be the same and thus couldn't these be in nvme_ns_head? Anyway, just not sure who to deal with this. So any pointers highly welcomed!
Cheers,
Daniel
drivers/nvme/host/core.c | 2 ++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 72 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+)
drivers/nvme/host/core.c +2 -0
@@ -2029,6 +2029,8 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,blk_mq_freeze_queue (ns->disk->queue); lbaf =nvme_lbaf_index (id->flbas); ns->lba_shift = id->lbaf[lbaf].ds; + ns->nsze =le64_to_cpu (id->nsze); + ns->nuse =le64_to_cpu (id->nuse);nvme_set_queue_limits (ns->ctrl, ns->queue);nvme_configure_metadata (ns, id);
drivers/nvme/host/nvme.h +2 -0
@@ -487,6 +487,8 @@ struct nvme_ns {struct nvme_ns_head *head;int lba_shift; +u64 nsze; +u64 nuse;u16 ms;u16 pi_size;u16 sgs;
drivers/nvme/host/sysfs.c +72 -0
84 lines not on this page · raw message
Signature · 1 line
2.43.0
Keith Busch
maintainerQuoted 7 lines · Daniel Wagner
libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.
As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.
The id namespace 'nuse' field can be quite volatile: it can change on any write or discard command, so caching it may quickly get out of sync with the actual value.
Quoted 5 lines · Keith Busch
Quoted 4 lines · Daniel Wagner
libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.
Quoted 7 lines
Quoted 3 lines
As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.
The id namespace 'nuse' field can be quite volatile: it can change on any write or discard command, so caching it may quickly get out of sync with the actual value.
libnvme itself is also cashing this value and exposes it via the nvme_ns_get_lba_util() getter. I'd say libnvme shouldn't cache it either. Instead the function should just issue the ns command to report the current nuse value.
I'll drop the nuse sysfs entry.
Unfortunately, 'nvme list' is using the 'nuse' field for showing the currently used space. I was hoping to get 'nvme list' working without issuing any commands.
Christoph Hellwig
maintainerQuoted 10 lines · Daniel Wagner
libnvme itself is also cashing this value and exposes it via the
nvme_ns_get_lba_util()getter. I'd say libnvme shouldn't cache it either. Instead the function should just issue the ns command to report the current nuse value.I'll drop the nuse sysfs entry.
Unfortunately, 'nvme list' is using the 'nuse' field for showing the currently used space. I was hoping to get 'nvme list' working without issuing any commands.
I'd be ok with implementing nuse in a way where we issue an identify command to read it, but rate limit the calls to something reasonable. I think the kernel can do that much better than userspace because it can keep that state a lot better.
Christoph Hellwig
maintainerQuoted 6 lines · Daniel Wagner
Also getting a pointer to the
nvme_nsdata structure is a bit strange (dev_to_nvme_ns).
This stip is necessary as many of the ns attributes are innvme_ns. Shouldn't these per path values not all be the same and thus couldn't these be innvme_ns_head? Anyway, just not sure who to deal with this. So any pointers highly welcomed!
Yes, they probably should be in the ns_head.
drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 138
Quoted 10 lines
drivers/nvme/host/sysfs.c +10
-0+list_for_each_entry (ctrl, &subsys->ctrls, subsys_entry) { +down_read (&ctrl->namespaces_rwsem); +list_for_each_entry (ns, &ctrl->namespaces, list) { + ret = ns; +break ; + } +up_read (&ctrl->namespaces_rwsem); + } +return ret; + }
.. I also don't think this would even be safe as-is as we dont hold a reference to the ns after dropping the lock.
drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 153
Quoted 13 lines
drivers/nvme/host/sysfs.c +13
-0+static ssize_t csi_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +return sysfs_emit (buf,"%d\n" ,dev_to_ns_head (dev)->ids.csi); +} +static DEVICE_ATTR_RO (csi); + +static ssize_t lba_ds_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct nvme_ns *ns =dev_to_nvme_ns (dev); + +return sysfs_emit (buf,"%d\n" , ns->lba_shift);
lba_ds is a bit of an odd name. And I also don't think we even need this, because it really is just a weird enconding for the logical block size already exposed by the block layer.
drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 164
Quoted 8 lines
drivers/nvme/host/sysfs.c +8
-0+static ssize_t lba_ms_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct nvme_ns *ns =dev_to_nvme_ns (dev); + +return sysfs_emit (buf,"%d\n" , ns->ms); +} +static DEVICE_ATTR_RO (lba_ms);
I'd probably spell out metadata_size, or probably even better metadata_bytes to match the unit postfixes elsewhere in the block code.
drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 173
Quoted 9 lines
drivers/nvme/host/sysfs.c +9
-0+ +static ssize_t nsze_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct nvme_ns *ns =dev_to_nvme_ns (dev); + +return sysfs_emit (buf,"%llu\n" , ns->nsze); +} +static DEVICE_ATTR_RO (nsze);
This is just the normal size of the block device we already export.
Keith Busch
maintainerQuoted 12 lines · Christoph Hellwig
drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 164
Quoted 8 lines · Daniel Wagner
drivers/nvme/host/sysfs.c +8
-0+static ssize_t lba_ms_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct nvme_ns *ns =dev_to_nvme_ns (dev); + +return sysfs_emit (buf,"%d\n" , ns->ms); +} +static DEVICE_ATTR_RO (lba_ms);I'd probably spell out
metadata_size, or probably even bettermetadata_bytesto match the unit postfixes elsewhere in the block code.
Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.
Christoph Hellwig
maintainerQuoted 7 lines · Keith Busch
Quoted 2 lines
I'd probably spell out
metadata_size, or probably even bettermetadata_bytesto match the unit postfixes elsewhere in the block code.Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.
Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
Keith Busch
maintainerQuoted 11 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 2 lines
I'd probably spell out
metadata_size, or probably even bettermetadata_bytesto match the unit postfixes elsewhere in the block code.Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.
Christoph Hellwig
maintainerQuoted 5 lines · Keith Busch
Quoted 5 lines
Quoted 4 lines
Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.
Quoted 7 lines
Quoted 2 lines
Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.
Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?
Keith Busch
maintainerQuoted 9 lines · Christoph Hellwig
Quoted 8 lines · Keith Busch
Quoted 7 lines
Quoted 4 lines
Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
Quoted 7 lines
Quoted 4 lines
nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.
Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?
tuple_size is the same value as metadata size regardless of PI usage. See nvme_init_integrity() for how this driver sets it:
integrity.tuple_size = ns->ms;
Quoted 9 lines · Keith Busch
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 7 lines
Quoted 4 lines
Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
'nvme list' is just listening the block size and the meta size in the 'Format' field. So nothing really crazy going on:
Usage Format
-------------------------- ----------------
343.33 GB / 512.11 GB 512 B + 0 B
nvme-cli commands like 'nmve ns-id' etc will always issue a command so that is not a concern. It's just the libnvme nvme_scan_topology() call which should stop issuing any commands.
I'll add the missing tuple_size to the integrity sysfs dir in this case.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.
Yes, a fallback will exist. There is no need to break existing users.
In summary, the only missing entries are
- csi
- tuple_size
- nuse
Sagi Grimberg
maintainerQuoted 27 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Should this even be an nvme specific attribute? I thought we should have
blk-integrity.creport its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?
'nvme list' is just listening the block size and the meta size in the 'Format' field. So nothing really crazy going on:
Usage Format -------------------------- ---------------- 343.33 GB / 512.11 GB 512 B + 0 Bnvme-cli commands like 'nmve ns-id' etc will always issue a command so that is not a concern. It's just the libnvme
nvme_scan_topology()call which should stop issuing any commands.I'll add the missing
tuple_sizeto the integrity sysfs dir in this case.Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.
Quoted 7 lines
Yes, a fallback will exist. There is no need to break existing users.
In summary, the only missing entries are
- csi
-tuple_size
- nuse
I agree with the comments made, especially the one made by Christoph that these values should be added to the nshead.
Christoph Hellwig
maintainerQuoted 8 lines · Keith Busch
Quoted 3 lines
Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?
tuple_sizeis the same value as metadata size regardless of PI usage. Seenvme_init_integrity()for how this driver sets it:integrity.tuple_size = ns->ms;
Yes, for the case where we actually support integrity in the kernel for a given device. But if the device has a metadata size larger than the PI size we still support it, and just let the device strip/insert the PI. And if nvme-cli wants to report detailed information about the namespace it probably needs to report the actual metadata size as the tuple size won't be reported given that we're never initializing the kernel PI support.
Keith Busch
maintainerQuoted 8 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 3 lines
Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?
tuple_sizeis the same value as metadata size regardless of PI usage. Seenvme_init_integrity()for how this driver sets it:
Quoted 6 lines
Quoted 1 line
integrity.tuple_size = ns->ms;Yes, for the case where we actually support integrity in the kernel for a given device. But if the device has a metadata size larger than the PI size we still support it, and just let the device strip/insert the PI.
I'm pretty sure that isn't right. We already support PI regardless of the metadata size as long as the PI field is in the first 8 bytes. Strip/insert doesn't even work if metadata is larger than a PI field. For any metadata case where PI isn't used, the driver requests allocating an empty buffer for the purpose.
Quoted 4 lines
And if nvme-cli wants to report detailed information about the namespace it probably needs to report the actual metadata size as the tuple size won't be reported given that we're never initializing the kernel PI support.
I don't understand. For any namespace with a metadata size, even if the namespace format doesn't have PI support, we still register an "integrity" profile with no-ops to get that unused buffer just so the block layer can access the format. We alyways set the tuple_size to the namespace metadata-size so the kernel buffer is correctly sized.
This all works as long as the metadata is separate (not extended) and kernel has CONFIG_BLK_DEV_INTEGRITY.