Skip to contentSkip to conversation

nvme: add cse, ds, ms, nsze and nuse to sysfs

RFC Superseded Daniel Wagner 4 participants 14 messages Nov 27, 2023 – Nov 28, 2023 NVMe

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

RFC 1/1 nvme: add cse, ds, ms, nsze and nuse to sysfs

libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.

As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.

---

As discussed during ALPPS, these here are the missing attribures which libnvme is still looking up via commands. I've tested this with a modified libnvme and didn't observe any ioctls anymore.

I'm pretty sure the naming is a bit off for the variables. Not really sure if we want to stick to the spec naming sceme or have our own one, e.g. 'nsze' vs 'capacity'.

Also getting a pointer to the nvme_ns data structure is a bit strange (dev_to_nvme_ns). This stip is necessary as many of the ns attributes are in nvme_ns. Shouldn't these per path values not all be the same and thus couldn't these be in nvme_ns_head? Anyway, just not sure who to deal with this. So any pointers highly welcomed!

Cheers,
Daniel

 drivers/nvme/host/core.c  |  2 ++
 drivers/nvme/host/nvme.h  |  2 ++
 drivers/nvme/host/sysfs.c | 72 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
drivers/nvme/host/core.c +2 -0
index 80673ea63fea..f100ee241bd7 100644
@@ -2029,6 +2029,8 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns,
 	blk_mq_freeze_queue(ns->disk->queue);
 	lbaf = nvme_lbaf_index(id->flbas);
 	ns->lba_shift = id->lbaf[lbaf].ds;
+	ns->nsze = le64_to_cpu(id->nsze);
+	ns->nuse = le64_to_cpu(id->nuse);
 	nvme_set_queue_limits(ns->ctrl, ns->queue);
 
 	nvme_configure_metadata(ns, id);
drivers/nvme/host/nvme.h +2 -0
index f35647c470af..97652bf2c787 100644
@@ -487,6 +487,8 @@ struct nvme_ns {
 	struct nvme_ns_head *head;
 
 	int lba_shift;
+	u64 nsze;
+	u64 nuse;
 	u16 ms;
 	u16 pi_size;
 	u16 sgs;
drivers/nvme/host/sysfs.c +72 -0

84 lines not on this page · raw message

Signature · 1 line
2.43.0
maintainer
#1 lore raw
Quoted 7 lines · Daniel Wagner

libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.

As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.

The id namespace 'nuse' field can be quite volatile: it can change on any write or discard command, so caching it may quickly get out of sync with the actual value.

#2 lore raw
Quoted 5 lines · Keith Busch
Quoted 4 lines · Daniel Wagner

libnvme is using the sysfs for enumarating the nvme resources. Though there are few missing attritbutes in the sysfs. For these libnvme issues commands during discovering.

Quoted 7 lines
Quoted 3 lines

As the kernel already knows all these attributes and we would like to avoid libnvme to issue commands all the time, expose these missing attributes.

The id namespace 'nuse' field can be quite volatile: it can change on any write or discard command, so caching it may quickly get out of sync with the actual value.

libnvme itself is also cashing this value and exposes it via the nvme_ns_get_lba_util() getter. I'd say libnvme shouldn't cache it either. Instead the function should just issue the ns command to report the current nuse value.

I'll drop the nuse sysfs entry.

Unfortunately, 'nvme list' is using the 'nuse' field for showing the currently used space. I was hoping to get 'nvme list' working without issuing any commands.

#3 lore raw
Quoted 10 lines · Daniel Wagner

libnvme itself is also cashing this value and exposes it via the nvme_ns_get_lba_util() getter. I'd say libnvme shouldn't cache it either. Instead the function should just issue the ns command to report the current nuse value.

I'll drop the nuse sysfs entry.

Unfortunately, 'nvme list' is using the 'nuse' field for showing the currently used space. I was hoping to get 'nvme list' working without issuing any commands.

I'd be ok with implementing nuse in a way where we issue an identify command to read it, but rate limit the calls to something reasonable. I think the kernel can do that much better than userspace because it can keep that state a lot better.

#4 lore raw
In reply to Daniel Wagner
Quoted 6 lines · Daniel Wagner

Also getting a pointer to the nvme_ns data structure is a bit strange (dev_to_nvme_ns).
This stip is necessary as many of the ns attributes are in nvme_ns. Shouldn't these per path values not all be the same and thus couldn't these be in nvme_ns_head? Anyway, just not sure who to deal with this. So any pointers highly welcomed!

Yes, they probably should be in the ns_head.

drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 138

Quoted 10 lines
drivers/nvme/host/sysfs.c +10 -0
+		list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) {
+			down_read(&ctrl->namespaces_rwsem);
+			list_for_each_entry(ns, &ctrl->namespaces, list) {
+				ret = ns;
+				break;
+			}
+			up_read(&ctrl->namespaces_rwsem);
+		}
+		return ret;
+	}

.. I also don't think this would even be safe as-is as we dont hold a reference to the ns after dropping the lock.

drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 153

Quoted 13 lines
drivers/nvme/host/sysfs.c +13 -0
+static ssize_t csi_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	return sysfs_emit(buf, "%d\n", dev_to_ns_head(dev)->ids.csi);
+}
+static DEVICE_ATTR_RO(csi);
+
+static ssize_t lba_ds_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns *ns = dev_to_nvme_ns(dev);
+
+	return sysfs_emit(buf, "%d\n", ns->lba_shift);

lba_ds is a bit of an odd name. And I also don't think we even need this, because it really is just a weird enconding for the logical block size already exposed by the block layer.

drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 164

Quoted 8 lines
drivers/nvme/host/sysfs.c +8 -0
+static ssize_t lba_ms_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns *ns = dev_to_nvme_ns(dev);
+
+	return sysfs_emit(buf, "%d\n", ns->ms);
+}
+static DEVICE_ATTR_RO(lba_ms);

I'd probably spell out metadata_size, or probably even better metadata_bytes to match the unit postfixes elsewhere in the block code.

drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 173

Quoted 9 lines
drivers/nvme/host/sysfs.c +9 -0
+
+static ssize_t nsze_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns *ns = dev_to_nvme_ns(dev);
+
+	return sysfs_emit(buf, "%llu\n", ns->nsze);
+}
+static DEVICE_ATTR_RO(nsze);

This is just the normal size of the block device we already export.

maintainer
#5 lore raw
Quoted 12 lines · Christoph Hellwig

drivers/nvme/host/sysfs.c static ssize_t nsid_show(struct device *dev, struct device_attribute *attr, · line 164

Quoted 8 lines · Daniel Wagner
drivers/nvme/host/sysfs.c +8 -0
+static ssize_t lba_ms_show(struct device *dev, struct device_attribute *attr,
+		char *buf)
+{
+	struct nvme_ns *ns = dev_to_nvme_ns(dev);
+
+	return sysfs_emit(buf, "%d\n", ns->ms);
+}
+static DEVICE_ATTR_RO(lba_ms);

I'd probably spell out metadata_size, or probably even better metadata_bytes to match the unit postfixes elsewhere in the block code.

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

#6 lore raw
Quoted 7 lines · Keith Busch
Quoted 2 lines

I'd probably spell out metadata_size, or probably even better metadata_bytes to match the unit postfixes elsewhere in the block code.

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

maintainer
#7 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 2 lines

I'd probably spell out metadata_size, or probably even better metadata_bytes to match the unit postfixes elsewhere in the block code.

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.

#8 lore raw
Quoted 5 lines · Keith Busch
Quoted 5 lines
Quoted 4 lines

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Quoted 7 lines
Quoted 2 lines

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.

Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?

maintainer
#9 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines · Keith Busch
Quoted 7 lines
Quoted 4 lines

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

Quoted 7 lines
Quoted 4 lines

nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.

Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?

tuple_size is the same value as metadata size regardless of PI usage. See nvme_init_integrity() for how this driver sets it:

	integrity.tuple_size = ns->ms;
#10 lore raw
Quoted 9 lines · Keith Busch
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 7 lines
Quoted 4 lines

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

'nvme list' is just listening the block size and the meta size in the 'Format' field. So nothing really crazy going on:

Usage Format
-------------------------- ----------------
343.33 GB / 512.11 GB 512 B + 0 B

nvme-cli commands like 'nmve ns-id' etc will always issue a command so that is not a concern. It's just the libnvme nvme_scan_topology() call which should stop issuing any commands.

I'll add the missing tuple_size to the integrity sysfs dir in this case.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.

Yes, a fallback will exist. There is no need to break existing users.

In summary, the only missing entries are

- csi
- tuple_size
- nuse

maintainer
#11 lore raw
Quoted 27 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

Should this even be an nvme specific attribute? I thought we should have blk-integrity.c report its 'tuple_size' attribute instead. That should work as long as we're not dealing with extended metadata at least, but that's kind of a special format that doesn't have block layer support.

Reporting the tuple size is a good idea. But is that enough for the existing nvme-cli use case?

'nvme list' is just listening the block size and the meta size in the 'Format' field. So nothing really crazy going on:

   Usage                      Format
   -------------------------- ----------------
   343.33  GB / 512.11  GB    512   B +  0 B

nvme-cli commands like 'nmve ns-id' etc will always issue a command so that is not a concern. It's just the libnvme nvme_scan_topology() call which should stop issuing any commands.

I'll add the missing tuple_size to the integrity sysfs dir in this case.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

nvme-cli currently queries with admin passthrough identify command, so adding a new attribute won't break that. I assume Daniel would have it fallback to that same command for backward compatibilty if a desired sysfs attribute doesn't exist.

Quoted 7 lines

Yes, a fallback will exist. There is no need to break existing users.

In summary, the only missing entries are

- csi
- tuple_size
- nuse

I agree with the comments made, especially the one made by Christoph that these values should be added to the nshead.

#12 lore raw
In reply to Keith Busch
Quoted 8 lines · Keith Busch
Quoted 3 lines

Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?

tuple_size is the same value as metadata size regardless of PI usage. See nvme_init_integrity() for how this driver sets it:

	integrity.tuple_size = ns->ms;

Yes, for the case where we actually support integrity in the kernel for a given device. But if the device has a metadata size larger than the PI size we still support it, and just let the device strip/insert the PI. And if nvme-cli wants to report detailed information about the namespace it probably needs to report the actual metadata size as the tuple size won't be reported given that we're never initializing the kernel PI support.

maintainer
#13 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Keith Busch
Quoted 3 lines

Yes. But does it care about the tuple size, or the actual size of the metadata field even if is bigger than the PI tuple?

tuple_size is the same value as metadata size regardless of PI usage. See nvme_init_integrity() for how this driver sets it:

Quoted 6 lines
Quoted 1 line
	integrity.tuple_size = ns->ms;

Yes, for the case where we actually support integrity in the kernel for a given device. But if the device has a metadata size larger than the PI size we still support it, and just let the device strip/insert the PI.

I'm pretty sure that isn't right. We already support PI regardless of the metadata size as long as the PI field is in the first 8 bytes. Strip/insert doesn't even work if metadata is larger than a PI field. For any metadata case where PI isn't used, the driver requests allocating an empty buffer for the purpose.

Quoted 4 lines

And if nvme-cli wants to report detailed information about the namespace it probably needs to report the actual metadata size as the tuple size won't be reported given that we're never initializing the kernel PI support.

I don't understand. For any namespace with a metadata size, even if the namespace format doesn't have PI support, we still register an "integrity" profile with no-ops to get that unused buffer just so the block layer can access the format. We alyways set the tuple_size to the namespace metadata-size so the kernel buffer is correctly sized.

This all works as long as the metadata is separate (not extended) and kernel has CONFIG_BLK_DEV_INTEGRITY.