nvme: align io queue count with allocted nvme_queue in nvme_probe
PATCH v3 1/1 nvme: align io queue count with allocted nvme_queue in nvme_probe
Since the commit 147b27e4bd0 "nvme-pci: allocate device queues storage space at probe" nvme_alloc_queue will not alloc struct nvme_queue any more.
If user change write/poll_queues to larger than the number of
allocated queue in nvme_probe, nvme_alloc_queue will touch
the memory out of boundary.
This patch add nr_allocated_queues for struct nvme_dev to record how many queues alloctaed in nvme_probe, then nvme driver will not use more queues than nr_allocated_queues when user update queue count and do a controller reset.
Since global module parameter can be changed at rumtime, so it's not safe to use these two parameter directly in the following functions:nvme_dbbuf_dma_allocnvme_dbbuf_dma_freenvme_calc_irq_setsnvme_setup_io_queues
This patch also add nr_write_queues, nr_poll_queues for struct nvme_dev and io_queues_reload for struct nvme_ctrl, that allow per-controller reload module parmater when reset controller. The nvme driver will not reload module parameter(write/poll_queues) by default when reset controller. If user want to reload them, they should enable it by echo 1 > /sys/block/<nvme_disk>/device/io_queues_reload.
By now, nvme pci driver allow user change io queue count for each type(write, read, poll) within nr_allocated_queue, that's to say, if user want to change queue dynamically by reset controller, they should setup io queues as many as possiable when load nvme module, and then tune io queue count for each type.
---
Changes since V2:
* fix typo in commit message.
* don't show /sys/block/<nvme_disk>/device/io_queues_reload for fabric.
Changes since V1:
* don't use module parameter nvme_dbbuf_dma_free, nvme_dbbuf_dma_alloc and nvme_calc_irq_sets.
* add per-controller sysfs file io_queues_reload to enable/disable reload global module parameter.
drivers/nvme/host/core.c | 31 ++++++++++++++++++++++ drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/pci.c | 55 +++++++++++++++++++++++----------------- 3 files changed, 64 insertions(+), 23 deletions(-)
drivers/nvme/host/core.c +31 -0
@@ -3357,6 +3357,34 @@ static ssize_t nvme_sysfs_show_address(struct device *dev, } static DEVICE_ATTR(address, S_IRUGO, nvme_sysfs_show_address, NULL); +static ssize_t nvme_sysfs_io_queues_reload_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return snprintf (buf,PAGE_SIZE ,"%d\n" , + ctrl->io_queues_reload ?1 :0 ); +} + +static ssize_t nvme_sysfs_io_queues_reload_store (struct device *dev, +struct device_attribute *attr,const char *buf, +size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +bool val; + +if (kstrtobool (buf, &val)) +return -EINVAL ; + ctrl->io_queues_reload = val; + +return count; +} + +static DEVICE_ATTR(io_queues_reload, S_IRUGO | S_IWUSR, + nvme_sysfs_io_queues_reload_show, + nvme_sysfs_io_queues_reload_store); +static struct attribute *nvme_dev_attrs[] = { &dev_attr_reset_controller.attr, &dev_attr_rescan_controller.attr,
@@ -3374,6 +3402,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_sqsize.attr, &dev_attr_hostnqn.attr, &dev_attr_hostid.attr, + &dev_attr_io_queues_reload.attr, NULL };
@@ -3391,6 +3420,8 @@ static umode_t nvme_dev_attrs_are_visible(struct kobject *kobj,return 0 ;if (a == &dev_attr_hostid.attr && !ctrl->opts)return 0 ; +if (a == &dev_attr_io_queues_reload.attr && ctrl->opts) +return 0 ;return a->mode; }
drivers/nvme/host/nvme.h +1 -0
@@ -260,6 +260,7 @@ struct nvme_ctrl {struct work_struct fw_act_work;unsigned long events;bool created; +bool io_queues_reload;#ifdef CONFIG_NVME_MULTIPATH /* asymmetric namespace access: */
drivers/nvme/host/pci.c +32 -23
121 lines not on this page · raw message
Signature · 7 lines
2.18.1 _______________________________________________ linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
I don't really thing we need the new sysfs attribute. Comments below:
drivers/nvme/host/pci.c static bool __nvme_disable_io_queues(struct nvme_dev *dev, u8 opcode); · line 94
Quoted 5 lines
struct
nvme_dev{drivers/nvme/host/pci.c +3
-0struct nvme_queue *queues; +int nr_allocated_queue; +int nr_write_queues; +int nr_poll_queues;
Can you keep these at the end of the structure given that they aren't needed in the fast path? Also any reason these can't be unsigned?
drivers/nvme/host/pci.c static int nvme_setup_io_queues(struct nvme_dev *dev) · line 2069
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+/* reload io queue count from module paramters write/poll_queues */ +if (dev->ctrl.io_queues_reload) {
So instead of the sysfs attribute, what about just executing this code unconditionally?
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 12 lines · Christoph Hellwig
I don't really thing we need the new sysfs attribute. Comments below:
drivers/nvme/host/pci.c static bool __nvme_disable_io_queues(struct nvme_dev *dev, u8 opcode); · line 94
Quoted 5 lines
struct nvme_dev { struct nvme_queue *queues;drivers/nvme/host/pci.c +3
-0+int nr_allocated_queue; +int nr_write_queues; +int nr_poll_queues;Can you keep these at the end of the structure given that they aren't needed in the fast path? Also any reason these can't be unsigned?
OK, move them to the end of struct, and use unsigned int.
Quoted 5 lines
drivers/nvme/host/pci.c static int nvme_setup_io_queues(struct nvme_dev *dev) · line 2069
Quoted 2 lines
drivers/nvme/host/pci.c +2
-0+/* reload io queue count from module paramters write/poll_queues */ +if (dev->ctrl.io_queues_reload) {So instead of the sysfs attribute, what about just executing this code unconditionally?
I think is's ok, these parameters are only used for pcie.
Quoted 5 lines
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme