Introduce new max-queue-size configuration
Subsystems (2)
v2 cover Introduce new max-queue-size configuration
Hi Christoph/Sagi/Keith,
This patch series is mainly for adding an interface for a user to configure the maximal queue size for fabrics via port configfs. Using this interface a user will be able to better control the system and HW resources.
Also, I've increased the maximal queue depth for RDMA controllers to be 256 after request from Guixin Liu. This new value will be valid only for controllers that don't support PI.
While developing this feature I've made some minor cleanups as well.
Changes from v1:
- collected Reviewed-by signatures (Sagi and Guixin Liu)
- removed the patches that unify fabric host and target max/min/default queue size definitions (Sagi)
- align MQES and SQ size according to the NVMe Spec (patch 2/8)
Max Gurtovoy (8):
nvme-rdma: move NVME_RDMA_IP_PORT from common file
nvmet: compare mqes and sqsize only for IO SQ
nvmet: set maxcmd to be per controller
nvmet: set ctrl pi_support cap before initializing cap reg
nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
nvme-rdma: clamp queue size according to ctrl cap
nvmet: introduce new max queue size configuration entry
nvmet-rdma: set max_queue_size for RDMA transport
drivers/nvme/host/rdma.c | 19 ++++++++++++++----- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 18 ++++++++++++++++-- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/fabrics-cmd.c | 5 ++--- drivers/nvme/target/nvmet.h | 6 ++++-- drivers/nvme/target/passthru.c | 2 +- drivers/nvme/target/rdma.c | 10 ++++++++++ include/linux/nvme-rdma.h | 6 +++++- include/linux/nvme.h | 2 -- 11 files changed, 82 insertions(+), 18 deletions(-)
Signature · 1 line
2.18.1
PATCH v2 1/8 nvme-rdma: move NVME_RDMA_IP_PORT from common file
The correct place for this definition is the nvme rdma header file and not the common nvme header file.
--- include/linux/nvme-rdma.h | 2 ++ include/linux/nvme.h | 2 -- 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/nvme-rdma.h +2 -0
@@ -6,6 +6,8 @@#ifndef _LINUX_NVME_RDMA_H#define _LINUX_NVME_RDMA_H +#define NVME_RDMA_IP_PORT 4420 +#define NVME_RDMA_MAX_QUEUE_SIZE 128 enum nvme_rdma_cm_fmt {
include/linux/nvme.h +0 -2
@@ -23,8 +23,6 @@#define NVME_DISC_SUBSYS_NAME "nqn.2014-08.org.nvmexpress.discovery" -#define NVME_RDMA_IP_PORT 4420 -#define NVME_NSID_ALL 0xffffffffenum nvme_subsys_type {
Signature · 1 line
2.18.1
Re: [PATCH 1/8] nvme-rdma: move NVME_RDMA_IP_PORT from common file
Looks good:
PATCH v2 2/8 nvmet: compare mqes and sqsize only for IO SQ
In reply to Max GurtovoyAccording to the NVMe Spec:
"
MQES: This field indicates the maximum individual queue size that the controller supports. For NVMe over PCIe implementations, this value applies to the I/O Submission Queues and I/O Completion Queues that the host creates. For NVMe over Fabrics implementations, this value applies to only the I/O Submission Queues that the host creates.
"
Align the target code to compare mqes and sqsize as mentioned in the NVMe Spec.
--- drivers/nvme/target/fabrics-cmd.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/nvme/target/fabrics-cmd.c +2 -1
@@ -157,7 +157,8 @@ static u16 nvmet_install_queue(struct nvmet_ctrl *ctrl, struct nvmet_req *req)return NVME_SC_CMD_SEQ_ERROR |NVME_SC_DNR ; } -if (sqsize > mqes) { +/* for fabrics, this value applies to only the I/O Submission Queues */ +if (qid && sqsize > mqes) {pr_warn ("sqsize %u is larger than MQES supported %u cntlid %d\n" , sqsize, mqes, ctrl->cntlid); req->error_loc = offsetof(struct nvmf_connect_command , sqsize);
Signature · 1 line
2.18.1
Re: [PATCH 2/8] nvmet: compare mqes and sqsize only for IO SQ
Re: [PATCH 2/8] nvmet: compare mqes and sqsize only for IO SQ
In reply to Max GurtovoyLooks good:
PATCH v2 3/8 nvmet: set maxcmd to be per controller
In reply to Max GurtovoyThis is a preparation for having a dynamic configuration of max queue size for a controller. Make sure that the maxcmd field stays the same as the MQES (+1) value as we do today.
--- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/nvmet.h | 2 +- drivers/nvme/target/passthru.c | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/admin-cmd.c +1 -1
@@ -428,7 +428,7 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->cqes = (0x4 <<4 ) |0x4 ;/* no enforcement soft-limit for maxcmd - pick arbitrary high value */ - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl)); id->nn =cpu_to_le32 (NVMET_MAX_NAMESPACES ); id->mnan =cpu_to_le32 (NVMET_MAX_NAMESPACES );
drivers/nvme/target/discovery.c +1 -1
@@ -282,7 +282,7 @@ static void nvmet_execute_disc_identify(struct nvmet_req *req) id->lpa = (1 <<2 );/* no enforcement soft-limit for maxcmd - pick arbitrary high value */ - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl)); id->sgls =cpu_to_le32 (1 <<0 );/* we always support SGLs */ if (ctrl->ops->flags &NVMF_KEYED_SGLS )
drivers/nvme/target/nvmet.h +1 -1
@@ -545,7 +545,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,#define NVMET_QUEUE_SIZE 1024#define NVMET_NR_QUEUES 128 -#define NVMET_MAX_CMD NVMET_QUEUE_SIZE +#define NVMET_MAX_CMD (ctrl) (NVME_CAP_MQES(ctrl->cap) + 1)/* * Nice round number that makes a list of nsids fit into a page.
drivers/nvme/target/passthru.c +1 -1
@@ -132,7 +132,7 @@ static u16 nvmet_passthru_override_id_ctrl(struct nvmet_req *req) id->sqes =min_t (__u8, ((0x6 <<4 ) |0x6 ), id->sqes); id->cqes =min_t (__u8, ((0x4 <<4 ) |0x4 ), id->cqes); - id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD ); + id->maxcmd =cpu_to_le16 (NVMET_MAX_CMD (ctrl));/* don't support fuse commands */ id->fuses =0 ;
Signature · 1 line
2.18.1
Re: [PATCH 3/8] nvmet: set maxcmd to be per controller
Looks good:
PATCH v2 4/8 nvmet: set ctrl pi_support cap before initializing cap reg
In reply to Max GurtovoyThis is a preparation for setting the maximal queue size of a controller that supports PI.
--- drivers/nvme/target/core.c | 1 + drivers/nvme/target/fabrics-cmd.c | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/target/core.c +1 -0
@@ -1411,6 +1411,7 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,kref_init (&ctrl->ref); ctrl->subsys = subsys; + ctrl->pi_support = ctrl->port->pi_enable && ctrl->subsys->pi_support;nvmet_init_cap (ctrl);WRITE_ONCE (ctrl->aen_enabled,NVMET_AEN_CFG_OPTIONAL );
drivers/nvme/target/fabrics-cmd.c +0 -2
@@ -252,8 +252,6 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)if (status) goto out; - ctrl->pi_support = ctrl->port->pi_enable && ctrl->subsys->pi_support; -uuid_copy (&ctrl->hostid, &d->hostid); ret =nvmet_setup_auth (ctrl);
Signature · 1 line
2.18.1
Re: [PATCH 4/8] nvmet: set ctrl pi_support cap before initializing cap reg
Looks good:
PATCH v2 5/8 nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
In reply to Max GurtovoyThis definition will be used by controllers that are configured with metadata support. For now, both regular and metadata controllers have the same maximal queue size but later commit will increase the maximal queue size for regular RDMA controllers to 256.
We'll keep the maximal queue size for metadata controllers to be 128 since there are more resources that are needed for metadata operations and 128 is the optimal size found for metadata controllers base on testing.
--- drivers/nvme/target/rdma.c | 2 ++ include/linux/nvme-rdma.h | 3 ++- 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +2 -0
@@ -2002,6 +2002,8 @@ static u8 nvmet_rdma_get_mdts(const struct nvmet_ctrl *ctrl)static u16 nvmet_rdma_get_max_queue_size (const struct nvmet_ctrl *ctrl) { +if (ctrl->pi_support) +return NVME_RDMA_MAX_METADATA_QUEUE_SIZE ;return NVME_RDMA_MAX_QUEUE_SIZE ; }
include/linux/nvme-rdma.h +2 -1
@@ -8,7 +8,8 @@#define NVME_RDMA_IP_PORT 4420 -#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_METADATA_QUEUE_SIZE 128enum nvme_rdma_cm_fmt {NVME_RDMA_CM_FMT_1_0 =0x0 ,
Signature · 1 line
2.18.1
Re: [PATCH 5/8] nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
Re: [PATCH 5/8] nvme-rdma: introduce NVME_RDMA_MAX_METADATA_QUEUE_SIZE definition
In reply to Max GurtovoyLooks good:
PATCH v2 6/8 nvme-rdma: clamp queue size according to ctrl cap
In reply to Max GurtovoyIf a controller is configured with metadata support, clamp the maximal queue size to be 128 since there are more resources that are needed for metadata operations. Otherwise, clamp it to 256.
--- drivers/nvme/host/rdma.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
drivers/nvme/host/rdma.c +14 -5
@@ -1029,11 +1029,20 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1); } -if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_QUEUE_SIZE ) { -dev_warn (ctrl->ctrl.device, -"ctrl sqsize %u > max queue size %u, clamping down\n" , - ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_QUEUE_SIZE ); - ctrl->ctrl.sqsize =NVME_RDMA_MAX_QUEUE_SIZE -1 ; +if (ctrl->ctrl.max_integrity_segments) { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_METADATA_QUEUE_SIZE ) { +dev_warn (ctrl->ctrl.device, +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_METADATA_QUEUE_SIZE ); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_METADATA_QUEUE_SIZE -1 ; + } + }else { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_QUEUE_SIZE ) { +dev_warn (ctrl->ctrl.device, +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_QUEUE_SIZE ); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_QUEUE_SIZE -1 ; + } }if (ctrl->ctrl.sqsize +1 > ctrl->ctrl.maxcmd) {
Signature · 1 line
2.18.1
Re: [PATCH 6/8] nvme-rdma: clamp queue size according to ctrl cap
Christoph Hellwig
maintainerRe: [PATCH 6/8] nvme-rdma: clamp queue size according to ctrl cap
In reply to Max GurtovoyQuoted 22 lines · Max Gurtovoy
If a controller is configured with metadata support, clamp the maximal queue size to be 128 since there are more resources that are needed for metadata operations. Otherwise, clamp it to 256.
Reviewed-by Israel Rukshin <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- drivers/nvme/host/rdma.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)drivers/nvme/host/rdma.c +4
-0@@ -1029,11 +1029,20 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1); } +if (ctrl->ctrl.max_integrity_segments) { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_METADATA_QUEUE_SIZE ) { + dev_warn(ctrl->ctrl.device, + "ctrl sqsize %u > max queue size %u, clamping down\n",
drivers/nvme/host/rdma.c static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) · line 1045
Quoted 10 lines
drivers/nvme/host/rdma.c +10
-0+ ctrl->ctrl.sqsize + 1, NVME_RDMA_MAX_METADATA_QUEUE_SIZE); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_METADATA_QUEUE_SIZE -1 ; + } + }else { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_QUEUE_SIZE ) { +dev_warn (ctrl->ctrl.device, +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_QUEUE_SIZE ); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_QUEUE_SIZE -1 ; + }
Can you just add a local max_queue_size variable instead of duplicating all this?
Re: [PATCH 6/8] nvme-rdma: clamp queue size according to ctrl cap
Quoted 28 lines · Christoph Hellwig
Quoted 27 lines · Max Gurtovoy
If a controller is configured with metadata support, clamp the maximal queue size to be 128 since there are more resources that are needed for metadata operations. Otherwise, clamp it to 256.
Reviewed-by Israel Rukshin <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--- drivers/nvme/host/rdma.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)drivers/nvme/host/rdma.c +9
-0@@ -1029,11 +1029,20 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) ctrl->ctrl.opts->queue_size, ctrl->ctrl.sqsize + 1); } +if (ctrl->ctrl.max_integrity_segments) { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_METADATA_QUEUE_SIZE ) { +dev_warn (ctrl->ctrl.device, +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_METADATA_QUEUE_SIZE ); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_METADATA_QUEUE_SIZE -1 ; + } + }else { +if (ctrl->ctrl.sqsize +1 >NVME_RDMA_MAX_QUEUE_SIZE ) {
Quoted 9 lines
drivers/nvme/host/rdma.c static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new) · line 1045
Quoted 5 lines
drivers/nvme/host/rdma.c +5
-0+dev_warn (ctrl->ctrl.device, +"ctrl sqsize %u > max queue size %u, clamping down\n" , + ctrl->ctrl.sqsize +1 ,NVME_RDMA_MAX_QUEUE_SIZE ); + ctrl->ctrl.sqsize =NVME_RDMA_MAX_QUEUE_SIZE -1 ; + }Can you just add a local
max_queue_sizevariable instead of duplicating all this?
Sure.
something like:
if (ctrl->ctrl.max_integrity_segments)
max_queue_size = NVME_RDMA_MAX_METADATA_QUEUE_SIZE;
else
max_queue_size = NVME_RDMA_MAX_QUEUE_SIZE;
if (ctrl->ctrl.sqsize + 1 > max_queue_size) {
dev_warn(ctrl->ctrl.device,
"ctrl sqsize %u > max queue size %u, clamping
down\n",
ctrl->ctrl.sqsize + 1, max_queue_size);
ctrl->ctrl.sqsize = max_queue_size - 1;PATCH v2 7/8 nvmet: introduce new max queue size configuration entry
In reply to Max GurtovoyUsing this port configuration, one will be able to set the maximal queue size to be used for any controller that will be associated to the configured port.
The default value stayed 1024 but each transport will be able to set the its own values before enabling the port.
Introduce lower limit of 16 for minimal queue depth (same as we use in the host fabrics drivers).
--- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 17 +++++++++++++++-- drivers/nvme/target/nvmet.h | 4 +++- 3 files changed, 46 insertions(+), 3 deletions(-)
drivers/nvme/target/configfs.c +28 -0
@@ -272,6 +272,32 @@ static ssize_t nvmet_param_inline_data_size_store(struct config_item *item,CONFIGFS_ATTR (nvmet_, param_inline_data_size); +static ssize_t nvmet_param_max_queue_size_show (struct config_item *item, +char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); + +return snprintf (page,PAGE_SIZE ,"%d\n" , port->max_queue_size); +} + +static ssize_t nvmet_param_max_queue_size_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +int ret; + +if (nvmet_is_port_enabled (port, __func__)) +return -EACCES ; + ret =kstrtoint (page,0 , &port->max_queue_size); +if (ret) { +pr_err ("Invalid value '%s' for max_queue_size\n" , page); +return -EINVAL ; + } +return count; +} + +CONFIGFS_ATTR (nvmet_, param_max_queue_size); +#ifdef CONFIG_BLK_DEV_INTEGRITY static ssize_t nvmet_param_pi_enable_show (struct config_item *item,char *page)
@@ -1856,6 +1882,7 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_trtype, &nvmet_attr_addr_tsas, &nvmet_attr_param_inline_data_size, + &nvmet_attr_param_max_queue_size, #ifdef CONFIG_BLK_DEV_INTEGRITY &nvmet_attr_param_pi_enable, #endif
@@ -1914,6 +1941,7 @@ static struct config_group *nvmet_ports_make(struct config_group *group,INIT_LIST_HEAD (&port->subsystems);INIT_LIST_HEAD (&port->referrals); port->inline_data_size =-1 ;/* < 0 == let the transport choose */ + port->max_queue_size =-1 ;/* < 0 == let the transport choose */ port->disc_addr.portid =cpu_to_le16 (portid); port->disc_addr.adrfam =NVMF_ADDR_FAMILY_MAX ;
drivers/nvme/target/core.c +15 -2
@@ -358,6 +358,18 @@ int nvmet_enable_port(struct nvmet_port *port)if (port->inline_data_size <0 ) port->inline_data_size =0 ; +/* +* If the transport didn't set the max_queue_size properly, then clamp +* it to the target limits. Also set default values in case the +* transport didn't set it at all. +*/ +if (port->max_queue_size <0 ) + port->max_queue_size =NVMET_MAX_QUEUE_SIZE ; +else + port->max_queue_size =clamp_t (int, port->max_queue_size, +NVMET_MIN_QUEUE_SIZE , +NVMET_MAX_QUEUE_SIZE ); + port->enabled = true; port->tr_ops = ops;return 0 ;
@@ -1223,9 +1235,10 @@ static void nvmet_init_cap(struct nvmet_ctrl *ctrl) ctrl->cap |= (15ULL <<24 );/* maximum queue entries supported: */ if (ctrl->ops->get_max_queue_size) - ctrl->cap |= ctrl->ops->get_max_queue_size (ctrl) -1 ; + ctrl->cap |=min_t (u16, ctrl->ops->get_max_queue_size (ctrl), + ctrl->port->max_queue_size) -1 ;else - ctrl->cap |=NVMET_QUEUE_SIZE -1 ; + ctrl->cap |= ctrl->port->max_queue_size -1 ;if (nvmet_is_passthru_subsys (ctrl->subsys))nvmet_passthrough_override_cap (ctrl);
drivers/nvme/target/nvmet.h +3 -1
@@ -163,6 +163,7 @@ struct nvmet_port {void *priv;bool enabled;int inline_data_size; +int max_queue_size;const struct nvmet_fabrics_ops *tr_ops;bool pi_enable; };
@@ -543,7 +544,8 @@ void nvmet_subsys_disc_changed(struct nvmet_subsys *subsys,void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page); -#define NVMET_QUEUE_SIZE 1024 +#define NVMET_MIN_QUEUE_SIZE 16 +#define NVMET_MAX_QUEUE_SIZE 1024#define NVMET_NR_QUEUES 128#define NVMET_MAX_CMD (ctrl) (NVME_CAP_MQES(ctrl->cap) + 1)
Signature · 1 line
2.18.1
Sagi Grimberg
maintainerRe: [PATCH 7/8] nvmet: introduce new max queue size configuration entry
|Reviewed-by: Sagi Grimberg <[email protected]>|
Re: [PATCH 7/8] nvmet: introduce new max queue size configuration entry
In reply to Max GurtovoyLooks good:
PATCH v2 8/8 nvmet-rdma: set max_queue_size for RDMA transport
In reply to Max GurtovoyA new port configuration was added to set max_queue_size. Clamp user configuration to RDMA transport limits.
Increase the maximal queue size of RDMA controllers from 128 to 256 (the default size stays 128 same as before).
--- drivers/nvme/target/rdma.c | 8 ++++++++ include/linux/nvme-rdma.h | 3 ++- 2 files changed, 10 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +8 -0
@@ -1943,6 +1943,14 @@ static int nvmet_rdma_add_port(struct nvmet_port *nport) nport->inline_data_size =NVMET_RDMA_MAX_INLINE_DATA_SIZE ; } +if (nport->max_queue_size <0 ) { + nport->max_queue_size =NVME_RDMA_DEFAULT_QUEUE_SIZE ; + }else if (nport->max_queue_size >NVME_RDMA_MAX_QUEUE_SIZE ) { +pr_warn ("max_queue_size %u is too large, reducing to %u\n" , + nport->max_queue_size,NVME_RDMA_MAX_QUEUE_SIZE ); + nport->max_queue_size =NVME_RDMA_MAX_QUEUE_SIZE ; + } + ret =inet_pton_with_scope (&init_net, af, nport->disc_addr.traddr, nport->disc_addr.trsvcid, &port->addr);if (ret) {
include/linux/nvme-rdma.h +2 -1
@@ -8,8 +8,9 @@#define NVME_RDMA_IP_PORT 4420 -#define NVME_RDMA_MAX_QUEUE_SIZE 128 +#define NVME_RDMA_MAX_QUEUE_SIZE 256#define NVME_RDMA_MAX_METADATA_QUEUE_SIZE 128 +#define NVME_RDMA_DEFAULT_QUEUE_SIZE 128enum nvme_rdma_cm_fmt {NVME_RDMA_CM_FMT_1_0 =0x0 ,
Signature · 1 line
2.18.1
Re: [PATCH 8/8] nvmet-rdma: set max_queue_size for RDMA transport
Re: [PATCH 8/8] nvmet-rdma: set max_queue_size for RDMA transport
In reply to Max GurtovoyLooks good:
Quoted 1 line · Max Gurtovoy
Hi Christoph/Sagi/Keith,
Hi Christoph/Keith,
are we considering taking this series into nvme-6.8 ? Most of it was reviewed by Sagi, the rest is pretty trivial
Quoted 40 lines
This patch series is mainly for adding an interface for a user to configure the maximal queue size for fabrics via port configfs. Using this interface a user will be able to better control the system and HW resources.
Also, I've increased the maximal queue depth for RDMA controllers to be 256 after request from Guixin Liu. This new value will be valid only for controllers that don't support PI.
While developing this feature I've made some minor cleanups as well.
Changes from v1:
- collected Reviewed-by signatures (Sagi and Guixin Liu)
- removed the patches that unify fabric host and target max/min/default queue size definitions (Sagi)
- align MQES and SQ size according to the NVMe Spec (patch 2/8)Max Gurtovoy (8):
nvme-rdma: moveNVME_RDMA_IP_PORTfrom common file
nvmet: compare mqes and sqsize only for IO SQ
nvmet: set maxcmd to be per controller
nvmet: set ctrlpi_supportcap before initializing cap reg
nvme-rdma: introduceNVME_RDMA_MAX_METADATA_QUEUE_SIZEdefinition
nvme-rdma: clamp queue size according to ctrl cap
nvmet: introduce new max queue size configuration entry
nvmet-rdma: setmax_queue_sizefor RDMA transportdrivers/nvme/host/rdma.c | 19 ++++++++++++++----- drivers/nvme/target/admin-cmd.c | 2 +- drivers/nvme/target/configfs.c | 28 ++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 18 ++++++++++++++++-- drivers/nvme/target/discovery.c | 2 +- drivers/nvme/target/fabrics-cmd.c | 5 ++--- drivers/nvme/target/nvmet.h | 6 ++++-- drivers/nvme/target/passthru.c | 2 +- drivers/nvme/target/rdma.c | 10 ++++++++++ include/linux/nvme-rdma.h | 6 +++++- include/linux/nvme.h | 2 -- 11 files changed, 82 insertions(+), 18 deletions(-)