Introduce new max-queue-size configuration
Subsystems (2)
v3 cover Introduce new max-queue-size configuration
Hi Christoph/Sagi/Keith,
This patch series is mainly for adding an interface for a user to configure the maximal queue size for fabrics via port configfs. Using this interface a user will be able to better control the system and HW resources.
Also, I've increased the maximal queue depth for RDMA controllers to be 256 after request from Guixin Liu. This new value will be valid only for controllers that don't support PI.
While developing this feature I've made some minor cleanups as well.
Changes from v2:
- collected Reviewed-by signatures (Sagi and Christoph)
- added local variable to simplify code in patch 6/8 (Christoph)
Changes from v1:
- collected Reviewed-by signatures (Sagi and Guixin Liu)
- removed the patches that unify fabric host and target max/min/default queue size definitions (Sagi)
- align MQES and SQ size according to the NVMe Spec (patch 2/8)
Max Gurtovoy (8):
nvme-rdma: move NVME_RDMA_IP_PORT from common file
nvmet: compare mqes and sqsize only for IO SQ
nvmet: set maxcmd to be per controller
nvmet: set ctrl pi_support cap before initializing cap reg
nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
nvme-rdma: clamp queue size according to ctrl cap
nvmet: introduce new max queue size configuration entry
nvmet-rdma: set max_queue_size for RDMA transport
drivers/nvme/host/rdma.c | 14 ++++++++++---- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 18 ++++++++++++++++-- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/fabrics-cmd.c | 5 ++--- drivers/nvme/target/nvmet.h | 6 ++++-- drivers/nvme/target/passthru.c | 2 +- drivers/nvme/target/rdma.c | 10 ++++++++++ include/linux/nvme-rdma.h | 6 +++++- include/linux/nvme.h | 2 -- 11 files changed, 78 insertions(+), 17 deletions(-)
Signature · 1 line
2.18.1
PATCH v3 1/8 nvme-rdma: move NVME_RDMA_IP_PORT from common file
The correct place for this definition is the nvme rdma header file and not the common nvme header file.
--- include/linux/nvme-rdma.h | 2 ++ include/linux/nvme.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/nvme-rdma.h +2 -0
@@ -6,6 +6,8 @@#ifndef _LINUX_NVME_RDMA_H#define _LINUX_NVME_RDMA_H +#define NVME_RDMA_IP_PORT 4420 +#define NVME_RDMA_MAX_QUEUE_SIZE 128 enum nvme_rdma_cm_fmt {
include/linux/nvme.h +0 -2
@@ -23,8 +23,6 @@#define NVME_DISC_SUBSYS_NAME "nqn.2014-08.org.nvmexpress.discovery" -#define NVME_RDMA_IP_PORT 4420 -#define NVME_NSID_ALL 0xffffffffenum nvme_subsys_type {
Signature · 1 line
2.18.1
PATCH v3 2/8 nvmet: compare mqes and sqsize only for IO SQ
In reply to Max GurtovoyAccording to the NVMe Spec:
"
MQES: This field indicates the maximum individual queue size that the controller supports. For NVMe over PCIe implementations, this value applies to the I/O Submission Queues and I/O Completion Queues that the host creates. For NVMe over Fabrics implementations, this value applies to only the I/O Submission Queues that the host creates.
"
Align the target code to compare mqes and sqsize as mentioned in the NVMe Spec.
--- drivers/nvme/target/fabrics-cmd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/fabrics-cmd.c +2 -1
@@ -157,7 +157,8 @@ static u16 nvmet_install_queue(struct nvmet_ctrl *ctrl, struct nvmet_req *req)return NVME_SC_CMD_SEQ_ERROR |NVME_SC_DNR ; } -if (sqsize > mqes) { +/* for fabrics, this value applies to only the I/O Submission Queues */ +if (qid && sqsize > mqes) {pr_warn ("sqsize %u is larger than MQES supported %u cntlid %d\n" , sqsize, mqes, ctrl->cntlid); req->error_loc = offsetof(struct nvmf_connect_command , sqsize);
Signature · 1 line
2.18.1
PATCH v3 3/8 nvmet: set maxcmd to be per controller
In reply to Max GurtovoyThis is a preparation for having a dynamic configuration of max queue size for a controller. Make sure that the maxcmd field stays the same as the MQES (+1) value as we do today.
--- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/nvmet.h | 2 +- drivers/nvme/target/passthru.c | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/admin-cmd.c +1 -1
@@ -428,7 +428,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->cqes = (0x4 <<4 ) |0x4 ;/* no enforcement soft-limit for maxcmd - pick arbitrary high value */ - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl)); id->nn =cpu_to_le32 (NVMET_MAX_NAMESPACES ); id->mnan =cpu_to_le32 (NVMET_MAX_NAMESPACES );
drivers/nvme/target/discovery.c +1 -1
@@ -282,7 +282,7 @@ static void nvmet_execute_disc_identify(struct nvmet_req *req) id->lpa = (1 <<2 );/* no enforcement soft-limit for maxcmd - pick arbitrary high value */ - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl)); id->sgls =cpu_to_le32 (1 <<0 );/* we always support SGLs */ if (ctrl->ops->flags &NVMF_KEYED_SGLS )
drivers/nvme/target/nvmet.h +1 -1
@@ -545,7 +545,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,#define NVMET_QUEUE_SIZE 1024#define NVMET_NR_QUEUES 128 -#define NVMET_MAX_CMD NVMET_QUEUE_SIZE +#define NVMET_MAX_CMD (ctrl) (NVME_CAP_MQES(ctrl->cap) + 1)/* * Nice round number that makes a list of nsids fit into a page.
drivers/nvme/target/passthru.c +1 -1
@@ -132,7 +132,7 @@ static u16 nvmet_passthru_override_id_ctrl(struct nvmet_req *req) id->sqes =min_t (__u8, ((0x6 <<4 ) |0x6 ), id->sqes); id->cqes =min_t (__u8, ((0x4 <<4 ) |0x4 ), id->cqes); - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl));/* don't support fuse commands */ id->fuses =0 ;
Signature · 1 line
2.18.1
PATCH v3 4/8 nvmet: set ctrl pi_support cap before initializing cap reg
In reply to Max GurtovoyThis is a preparation for setting the maximal queue size of a controller that supports PI.
--- drivers/nvme/target/core.c | 1 + drivers/nvme/target/fabrics-cmd.c | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/target/core.c +1 -0
@@ -1411,6 +1411,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,kref_init (&ctrl->ref); ctrl->subsys = subsys; + ctrl->pi_support = ctrl->port->pi_enable && ctrl->subsys->pi_support;nvmet_init_cap (ctrl);WRITE_ONCE (ctrl->aen_enabled,NVMET_AEN_CFG_OPTIONAL );
drivers/nvme/target/fabrics-cmd.c +0 -2
@@ -252,8 +252,6 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)if (status) goto out; - ctrl->pi_support = ctrl->port->pi_enable && ctrl->subsys->pi_support; -uuid_copy (&ctrl->hostid, &d->hostid); ret =nvmet_setup_auth (ctrl);
Signature · 1 line
2.18.1
PATCH v3 5/8 nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
In reply to Max GurtovoyThis definition will be used by controllers that are configured with metadata support. For now, both regular and metadata controllers have the same maximal queue size but later commit will increase the maximal queue size for regular RDMA controllers to 256.
We'll keep the maximal queue size for metadata controllers to be 128 since there are more resources that are needed for metadata operations and 128 is the optimal size found for metadata controllers base on testing.
--- drivers/nvme/target/rdma.c | 2 ++ include/linux/nvme-rdma.h | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +2 -0
@@ -2015,6 +2015,8 @@ static u8 nvmet_rdma_get_mdts(const struct nvmet_ctrl *ctrl)static u16 nvmet_rdma_get_max_queue_size (const struct nvmet_ctrl *ctrl) { +if (ctrl->pi_support) +return NVME_RDMA_MAX_METADATA_QUEUE_SIZE ;return NVME_RDMA_MAX_QUEUE_SIZE ; }
include/linux/nvme-rdma.h +2 -1
@@ -8,7 +8,8 @@#define NVME_RDMA_IP_PORT 4420 -#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_METADATA_QUEUE_SIZE 128enum nvme_rdma_cm_fmt {NVME_RDMA_CM_FMT_1_0 =0x0 ,
Signature · 1 line
2.18.1
PATCH v3 6/8 nvme-rdma: clamp queue size according to ctrl cap
In reply to Max GurtovoyIf a controller is configured with metadata support, clamp the maximal queue size to be 128 since there are more resources that are needed for metadata operations. Otherwise, clamp it to 256.
--- drivers/nvme/host/rdma.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/nvme/host/rdma.c +10 -4
@@ -1005,6 +1005,7 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) {int ret;bool changed; +u16 max_queue_size; ret =nvme_rdma_configure_admin_queue (ctrl, new);if (ret)
@@ -1029,11 +1030,16 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1); } -if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_QUEUE_SIZE ) { +if (ctrl->ctrl.max_integrity_segments) + max_queue_size =NVME_RDMA_MAX_METADATA_QUEUE_SIZE ; +else + max_queue_size =NVME_RDMA_MAX_QUEUE_SIZE ; + +if (ctrl->ctrl.sqsize +1 > max_queue_size) {dev_warn (ctrl->ctrl.device, -"ctrl sqsize %u > max queue size %u, clamping down\n" , - ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_QUEUE_SIZE ); - ctrl->ctrl.sqsize =NVME_RDMA_MAX_QUEUE_SIZE -1 ; +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 , max_queue_size); + ctrl->ctrl.sqsize = max_queue_size -1 ; }if (ctrl->ctrl.sqsize +1 > ctrl->ctrl.maxcmd) {
Signature · 1 line
2.18.1
Re: [PATCH 6/8] nvme-rdma: clamp queue size according to ctrl cap
Looks good:
PATCH v3 7/8 nvmet: introduce new max queue size configuration entry
In reply to Max GurtovoyUsing this port configuration, one will be able to set the maximal queue size to be used for any controller that will be associated to the configured port.
The default value stayed 1024 but each transport will be able to set the its own values before enabling the port.
Introduce lower limit of 16 for minimal queue depth (same as we use in the host fabrics drivers).
--- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 17 +++++++++++++++-- drivers/nvme/target/nvmet.h | 4 +++- 3 files changed, 46 insertions(+), 3 deletions(-)
drivers/nvme/target/configfs.c +28 -0
@@ -272,6 +272,32 @@ static ssize_t nvmet_param_inline_data_size_store(struct config_item *item,CONFIGFS_ATTR (nvmet_, param_inline_data_size); +static ssize_t nvmet_param_max_queue_size_show (struct config_item *item, +char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); + +return snprintf (page,PAGE_SIZE ,"%d\n" , port->max_queue_size); +} + +static ssize_t nvmet_param_max_queue_size_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +int ret; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + ret =kstrtoint (page,0 , &port->max_queue_size); +if (ret) { +pr_err ("Invalid value '%s' for max_queue_size\n" , page); +return -EINVAL ; + } +return count; +} + +CONFIGFS_ATTR (nvmet_, param_max_queue_size); +#ifdef CONFIG_BLK_DEV_INTEGRITY static ssize_t nvmet_param_pi_enable_show (struct config_item *item,char *page)
@@ -1856,6 +1882,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_trtype, &nvmet_attr_addr_tsas, &nvmet_attr_param_inline_data_size, + &nvmet_attr_param_max_queue_size, #ifdef CONFIG_BLK_DEV_INTEGRITY &nvmet_attr_param_pi_enable, #endif
@@ -1914,6 +1941,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,INIT_LIST_HEAD (&port->subsystems);INIT_LIST_HEAD (&port->referrals); port->inline_data_size =-1 ;/* < 0 == let the transport choose */ + port->max_queue_size =-1 ;/* < 0 == let the transport choose */ port->disc_addr.portid =cpu_to_le16 (portid); port->disc_addr.adrfam =NVMF_ADDR_FAMILY_MAX ;
drivers/nvme/target/core.c +15 -2
@@ -358,6 +358,18 @@ int nvmet_enable_port(struct nvmet_port *port)if (port->inline_data_size <0 ) port->inline_data_size =0 ; +/* +* If the transport didn't set the max_queue_size properly, then clamp +* it to the target limits. Also set default values in case the +* transport didn't set it at all. +*/ +if (port->max_queue_size <0 ) + port->max_queue_size =NVMET_MAX_QUEUE_SIZE ; +else + port->max_queue_size =clamp_t (int, port->max_queue_size, +NVMET_MIN_QUEUE_SIZE , +NVMET_MAX_QUEUE_SIZE ); + port->enabled = true; port->tr_ops = ops;return 0 ;
@@ -1223,9 +1235,10 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl) ctrl->cap |= (15ULL <<24 );/* maximum queue entries supported: */ if (ctrl->ops->get_max_queue_size) - ctrl->cap |= ctrl->ops->get_max_queue_size (ctrl) -1 ; + ctrl->cap |=min_t (u16, ctrl->ops->get_max_queue_size (ctrl), + ctrl->port->max_queue_size) -1 ;else - ctrl->cap |=NVMET_QUEUE_SIZE -1 ; + ctrl->cap |= ctrl->port->max_queue_size -1 ;if (nvmet_is_passthru_subsys (ctrl->subsys))nvmet_passthrough_override_cap (ctrl);
drivers/nvme/target/nvmet.h +3 -1
@@ -163,6 +163,7 @@ struct nvmet_port {void *priv;bool enabled;int inline_data_size; +int max_queue_size;const struct nvmet_fabrics_ops *tr_ops;bool pi_enable; };
@@ -543,7 +544,8 @@ void nvmet_subsys_disc_changed(struct nvmet_subsys *subsys,void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page); -#define NVMET_QUEUE_SIZE 1024 +#define NVMET_MIN_QUEUE_SIZE 16 +#define NVMET_MAX_QUEUE_SIZE 1024#define NVMET_NR_QUEUES 128#define NVMET_MAX_CMD (ctrl) (NVME_CAP_MQES(ctrl->cap) + 1)
Signature · 1 line
2.18.1
PATCH v3 8/8 nvmet-rdma: set max_queue_size for RDMA transport
In reply to Max GurtovoyA new port configuration was added to set max_queue_size. Clamp user configuration to RDMA transport limits.
Increase the maximal queue size of RDMA controllers from 128 to 256 (the default size stays 128 same as before).
--- drivers/nvme/target/rdma.c | 8 ++++++++ include/linux/nvme-rdma.h | 3 ++- 2 files changed, 10 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +8 -0
@@ -1956,6 +1956,14 @@ static int nvmet_rdma_add_port(struct nvmet_port *nport) nport->inline_data_size =NVMET_RDMA_MAX_INLINE_DATA_SIZE ; } +if (nport->max_queue_size <0 ) { + nport->max_queue_size =NVME_RDMA_DEFAULT_QUEUE_SIZE ; + }else if (nport->max_queue_size >NVME_RDMA_MAX_QUEUE_SIZE ) { +pr_warn ("max_queue_size %u is too large, reducing to %u\n" , + nport->max_queue_size,NVME_RDMA_MAX_QUEUE_SIZE ); + nport->max_queue_size =NVME_RDMA_MAX_QUEUE_SIZE ; + } + ret =inet_pton_with_scope (&init_net, af, nport->disc_addr.traddr, nport->disc_addr.trsvcid, &port->addr);if (ret) {
include/linux/nvme-rdma.h +2 -1
@@ -8,8 +8,9 @@#define NVME_RDMA_IP_PORT 4420 -#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_QUEUE_SIZE 256#define NVME_RDMA_MAX_METADATA_QUEUE_SIZE 128 +#define NVME_RDMA_DEFAULT_QUEUE_SIZE 128enum nvme_rdma_cm_fmt {NVME_RDMA_CM_FMT_1_0 =0x0 ,
Signature · 1 line
2.18.1
Hi Keith,
This series was fully reviewed by Sagi and Christoph for 6.8 but we probably somehow missed taking it.
can you please take it for the next 6.9 merge window and/or merge it to nvme-6.9 ?
Quoted 45 lines · Max Gurtovoy
Hi Christoph/Sagi/Keith,
This patch series is mainly for adding an interface for a user to configure the maximal queue size for fabrics via port configfs. Using this interface a user will be able to better control the system and HW resources.Also, I've increased the maximal queue depth for RDMA controllers to be 256 after request from Guixin Liu. This new value will be valid only for controllers that don't support PI.
While developing this feature I've made some minor cleanups as well.
Changes from v2:
- collected Reviewed-by signatures (Sagi and Christoph)
- added local variable to simplify code in patch 6/8 (Christoph)Changes from v1:
- collected Reviewed-by signatures (Sagi and Guixin Liu)
- removed the patches that unify fabric host and target max/min/default queue size definitions (Sagi)
- align MQES and SQ size according to the NVMe Spec (patch 2/8)Max Gurtovoy (8):
nvme-rdma: moveNVME_RDMA_IP_PORTfrom common file
nvmet: compare mqes and sqsize only for IO SQ
nvmet: set maxcmd to be per controller
nvmet: set ctrlpi_supportcap before initializing cap reg
nvme-rdma: introduceNVME_RDMA_MAX_METADATA_QUEUE_SIZEdefinition
nvme-rdma: clamp queue size according to ctrl cap
nvmet: introduce new max queue size configuration entry
nvmet-rdma: setmax_queue_sizefor RDMA transportdrivers/nvme/host/rdma.c | 14 ++++++++++---- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 18 ++++++++++++++++-- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/fabrics-cmd.c | 5 ++--- drivers/nvme/target/nvmet.h | 6 ++++-- drivers/nvme/target/passthru.c | 2 +- drivers/nvme/target/rdma.c | 10 ++++++++++ include/linux/nvme-rdma.h | 6 +++++- include/linux/nvme.h | 2 -- 11 files changed, 78 insertions(+), 17 deletions(-)
Keith Busch
maintainerThanks, queued up for nvme-6.9.