sqsize fixes
Subsystems (2)
v3 cover sqsize fixes
This patch series is based on making sure sqsize is defined as a zero-based value throughout the code, per NVMe-over-Fabrics spec.
Changes from v2:
- patch 5, adding 1 to hrsqsize in target dropped
- using ...opts->queue_depth to set various queue sizes over sqsize.
Changes from v1:
- moved +1/+2 to outside le16_to_cpu() in patch 3 and 5
Changes from v0:
- found all the sqsize dependencies and adjusted them accordingly
- nvmf_connect_admin_queue() always uses NVMF_AQ_DEPTH for sqsize
- final patch to adjust hrqsize only, so the series can be easily tested w/hrqsize == hrsqsize (patches 1-4) and hrqsize == hrsqsize+1 (patch 5)
Jay Freyensee (4):
nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
fabrics: define admin sqsize min default, per spec
nvme-rdma: fix sqsize/hsqsize per spec
nvme-loop: set sqsize to 0-based value, per spec
drivers/nvme/host/fabrics.c | 9 ++++++++- drivers/nvme/host/rdma.c | 20 +++++++++++++++----- drivers/nvme/target/loop.c | 4 ++-- drivers/nvme/target/rdma.c | 8 ++++---- 4 files changed, 29 insertions(+), 12 deletions(-)
Signature · 1 line
2.7.4
PATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
The host will be sending sqsize 0-based values,
the target need to be adjusted as well.
--- drivers/nvme/target/rdma.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/rdma.c +4 -4
@@ -1004,11 +1004,11 @@ nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn, queue->host_qid =le16_to_cpu (req->qid);/* -* req->hsqsize corresponds to our recv queue size -* req->hrqsize corresponds to our send queue size +* req->hsqsize corresponds to our recv queue size plus 1 +* req->hrqsize corresponds to our send queue size plus 1 */ - queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize); + queue->recv_queue_size =le16_to_cpu (req->hsqsize) +1 ; + queue->send_queue_size =le16_to_cpu (req->hrqsize) +1 ;if (!queue->host_qid && queue->recv_queue_size >NVMF_AQ_DEPTH )return NVME_RDMA_CM_INVALID_HSQSIZE ;
Signature · 1 line
2.7.4
Christoph Hellwig
maintainerPATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
drivers/nvme/target/rdma.c nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn, · line 1011
Quoted 10 lines · Jay Freyensee
drivers/nvme/target/rdma.c +4
-4/* -* req->hsqsize corresponds to our recv queue size -* req->hrqsize corresponds to our send queue size +* req->hsqsize corresponds to our recv queue size plus 1 +* req->hrqsize corresponds to our send queue size plus 1 */ - queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize); + queue->recv_queue_size =le16_to_cpu (req->hsqsize) +1 ; + queue->send_queue_size =le16_to_cpu (req->hrqsize) +1 ;
I brought this up on the nvme-technical list and the consensus is that hrqsize doesn't use the one off notation. hsqsize refers to the sqsize which is marked as "0's based", while hrqsize only has a short and not very meaningful explanation, which implies that it's '1's based' in NVMe terms (which, btw I think are utterly misleading).
Sagi Grimberg
maintainerPATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
Quoted 9 lines · Jay Freyensee
Quoted 8 lines
drivers/nvme/target/rdma.c +2
-4/* -* req->hsqsize corresponds to our recv queue size -* req->hrqsize corresponds to our send queue size +* req->hsqsize corresponds to our recv queue size plus 1 +* req->hrqsize corresponds to our send queue size plus 1 */ - queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize);
Quoted 8 lines
drivers/nvme/target/rdma.c nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn, · line 1011
Quoted 2 lines
drivers/nvme/target/rdma.c +2
-0+ queue->recv_queue_size =le16_to_cpu (req->hsqsize) +1 ; + queue->send_queue_size =le16_to_cpu (req->hrqsize) +1 ;I brought this up on the nvme-technical list and the consensus is that hrqsize doesn't use the one off notation. hsqsize refers to the sqsize which is marked as "0's based", while hrqsize only has a short and not very meaningful explanation, which implies that it's '1's based' in NVMe terms (which, btw I think are utterly misleading).
Yea... the host needs to set hrqsize to opts->queue_size and the target should take it as is...
PATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
Quoted 10 lines · Jay Freyensee
? /*
(quoted diff) +2
-2- ?* req->hsqsize corresponds to our recv queue size - ?* req->hrqsize corresponds to our send queue size + ?* req->hsqsize corresponds to our recv queue size plus 1 + ?* req->hrqsize corresponds to our send queue size plus 1? ?*/
drivers/nvme/target/rdma.c +1
-2- queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize); + queue->recv_queue_size =le16_to_cpu (req->hsqsize) +1 ;
Quoted 8 lines
drivers/nvme/target/rdma.c nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn, · line 1011
Quoted 1 line
drivers/nvme/target/rdma.c +1
-0+ queue->send_queue_size =le16_to_cpu (req->hrqsize) +1 ;I brought this up on the nvme-technical list and the consensus is that hrqsize doesn't use the one off notation.??hsqsize refers to the sqsize which is marked as "0's based", while hrqsize only has a short and not very meaningful explanation, which implies that it's '1's based' in NVMe terms (which, btw I think are utterly misleading).
OK, so what is the final verdict then? ?Resurrect patch five again and resubmit the series? ?It makes hrqsize 1-based on both the host and target.
http://lists.infradead.org/pipermail/linux-nvme/2016-August/005804.html
I interpret the 'explanation' it can be either solution and I don't see an advantage for either so we should just pick one.
Sagi Grimberg
maintainerPATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
drivers/nvme/target/rdma.c +3
-4/* -* req->hsqsize corresponds to our recv queue size -* req->hrqsize corresponds to our send queue size +* req->hsqsize corresponds to our recv queue size plus 1 +* req->hrqsize corresponds to our send queue size plus 1 */ - queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize); + queue->recv_queue_size =le16_to_cpu (req->hsqsize) +1 ;Quoted 8 lines
drivers/nvme/target/rdma.c nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn, · line 1011
Quoted 1 line
drivers/nvme/target/rdma.c +1
-0+ queue->send_queue_size =le16_to_cpu (req->hrqsize) +1 ;I brought this up on the nvme-technical list and the consensus is that hrqsize doesn't use the one off notation. hsqsize refers to the sqsize which is marked as "0's based", while hrqsize only has a short and not very meaningful explanation, which implies that it's '1's based' in NVMe terms (which, btw I think are utterly misleading).
Quoted 4 lines
OK, so what is the final verdict then? Resurrect patch five again and resubmit the series? It makes hrqsize 1-based on both the host and target.
http://lists.infradead.org/pipermail/linux-nvme/2016-August/005804.html
I think you can just make the target take hrqsize as is in patch 1 and have the host send hrqsize = queue->queue_size.
One more respin should do it (sorry for the exhaustion ;))
PATCH v3 1/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
Quoted 33 lines · Sagi Grimberg
Quoted 26 lines
Quoted 18 lines
Quoted 16 lines
? /*
(quoted diff) +1
-2- ?* req->hsqsize corresponds to our recv queue size - ?* req->hrqsize corresponds to our send queue size + ?* req->hsqsize corresponds to our recv queue sizeplus 1
(quoted diff) +1
-0+ ?* req->hrqsize corresponds to our send queue sizeplus 1
? ?*/drivers/nvme/target/rdma.c +1
-2- queue->recv_queue_size =le16_to_cpu (req->hsqsize); - queue->send_queue_size =le16_to_cpu (req->hrqsize); + queue->recv_queue_size = le16_to_cpu(req->hsqsize) +1;
(quoted diff) +1
-0+ queue->send_queue_size = le16_to_cpu(req->hrqsize) +Quoted 1 line
1;
Quoted 7 lines
I brought this up on the nvme-technical list and the consensus is that hrqsize doesn't use the one off notation.??hsqsize refers to the sqsize which is marked as "0's based", while hrqsize only has a short and not very meaningful explanation, which implies that
it's '1's based' in NVMe terms (which, btw I think are utterly misleading).Quoted 6 lines
OK, so what is the final verdict then???Resurrect patch five again and
resubmit the series???It makes hrqsize 1-based on both the host and target.
http://lists.infradead.org/pipermail/linux-nvme/2016-August/005804.
html
Quoted 4 lines
I think you can just make the target take hrqsize as is in patch 1 and have the host send hrqsize =
queue->queue_size.One more respin should do it (sorry for the exhaustion ;))
No worries, doesn't help when the spec isn't clear on something :-).
PATCH v3 2/4 fabrics: define admin sqsize min default, per spec
In reply to Jay FreyenseeUpon admin queue connect(), the rdma qp was being set based on NVMF_AQ_DEPTH. However, the fabrics layer was using the sqsize field value set for I/O queues for the admin queue, which threw the nvme layer and rdma layer off-whack:
root at fedora23-fabrics-host1 nvmf]# dmesg
[ 3507.798642] nvme_fabrics: nvmf_connect_admin_queue():admin sqsize being sent is: 128 [ 3507.798858] nvme nvme0: creating 16 I/O queues. [ 3507.896407] nvme nvme0: new ctrl: NQN "nullside-nqn", addr
192.168.1.3:4420
Thus, to have a different admin queue value, we use NVMF_AQ_DEPTH for connect() and RDMA private data
as the minimum depth specified in the NVMe-over-Fabrics 1.0 spec.
--- drivers/nvme/host/fabrics.c | 9 ++++++++- drivers/nvme/host/rdma.c | 13 +++++++++++-- 2 files changed, 19 insertions(+), 3 deletions(-)
drivers/nvme/host/fabrics.c +8 -1
@@ -363,7 +363,14 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl) cmd.connect.opcode = nvme_fabrics_command; cmd.connect.fctype = nvme_fabrics_type_connect; cmd.connect.qid =0 ; - cmd.connect.sqsize =cpu_to_le16 (ctrl->sqsize); + +/* +* fabrics spec sets a minimum of depth 32 for admin queue, +* so set the queue with this depth always until +* justification otherwise. +*/ + cmd.connect.sqsize =cpu_to_le16 (NVMF_AQ_DEPTH -1 ); +/* * Set keep-alive timeout in seconds granularity (ms * 1000) * and add a grace period for controller kato enforcement
drivers/nvme/host/rdma.c +11 -2
@@ -1284,8 +1284,17 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue) priv.recfmt =cpu_to_le16 (NVME_RDMA_CM_FMT_1_0 ); priv.qid =cpu_to_le16 (nvme_rdma_queue_idx (queue)); - priv.hrqsize =cpu_to_le16 (queue->queue_size); - priv.hsqsize =cpu_to_le16 (queue->queue_size); +/* +* set the admin queue depth to the minimum size +* specified by the Fabrics standard. +*/ +if (priv.qid ==0 ) { + priv.hrqsize =cpu_to_le16 (NVMF_AQ_DEPTH -1 ); + priv.hsqsize =cpu_to_le16 (NVMF_AQ_DEPTH -1 ); + }else { + priv.hrqsize =cpu_to_le16 (queue->queue_size); + priv.hsqsize =cpu_to_le16 (queue->queue_size); + } ret =rdma_connect (queue->cm_id, ¶m);if (ret) {
Signature · 1 line
2.7.4
PATCH v3 3/4 nvme-rdma: fix sqsize/hsqsize per spec
In reply to Jay FreyenseePer NVMe-over-Fabrics 1.0 spec, sqsize is represented as a 0-based value.
Also per spec, the RDMA binding values shall be set to sqsize, which makes hsqsize 0-based values.
Thus, the sqsize during NVMf connect() is now:
[root at fedora23-fabrics-host1 for-48]# dmesg
[ 318.720645] nvme_fabrics: nvmf_connect_admin_queue(): sqsize for admin queue: 31 [ 318.720884] nvme nvme0: creating 16 I/O queues. [ 318.810114] nvme_fabrics: nvmf_connect_io_queue(): sqsize for i/o
queue: 127
--- drivers/nvme/host/rdma.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
drivers/nvme/host/rdma.c +6 -5
@@ -649,7 +649,8 @@ static int nvme_rdma_init_io_queues(struct nvme_rdma_ctrl *ctrl)int i, ret;for (i =1 ; i < ctrl->queue_count; i++) { - ret =nvme_rdma_init_queue (ctrl, i, ctrl->ctrl.sqsize); + ret =nvme_rdma_init_queue (ctrl, i, + ctrl->ctrl.opts->queue_size);if (ret) {dev_info (ctrl->ctrl.device,"failed to initialize i/o queue: %d\n" , ret);
@@ -1292,8 +1293,8 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue) priv.hrqsize =cpu_to_le16 (NVMF_AQ_DEPTH -1 ); priv.hsqsize =cpu_to_le16 (NVMF_AQ_DEPTH -1 ); }else { - priv.hrqsize =cpu_to_le16 (queue->queue_size); - priv.hsqsize =cpu_to_le16 (queue->queue_size); + priv.hrqsize =cpu_to_le16 (queue->ctrl->ctrl.sqsize); + priv.hsqsize =cpu_to_le16 (queue->ctrl->ctrl.sqsize); } ret =rdma_connect (queue->cm_id, ¶m);
@@ -1818,7 +1819,7 @@ static int nvme_rdma_create_io_queues(struct nvme_rdma_ctrl *ctrl)memset (&ctrl->tag_set,0 ,sizeof (ctrl->tag_set)); ctrl->tag_set.ops = &nvme_rdma_mq_ops; - ctrl->tag_set.queue_depth = ctrl->ctrl.sqsize; + ctrl->tag_set.queue_depth = ctrl->ctrl.opts->queue_size; ctrl->tag_set.reserved_tags =1 ;/* fabric connect */ ctrl->tag_set.numa_node =NUMA_NO_NODE ; ctrl->tag_set.flags =BLK_MQ_F_SHOULD_MERGE ;
@@ -1916,7 +1917,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,spin_lock_init (&ctrl->lock); ctrl->queue_count = opts->nr_io_queues +1 ;/* +1 for admin queue */ - ctrl->ctrl.sqsize = opts->queue_size; + ctrl->ctrl.sqsize = opts->queue_size -1 ; ctrl->ctrl.kato = opts->kato; ret = -ENOMEM ;
Signature · 1 line
2.7.4
PATCH v3 4/4 nvme-loop: set sqsize to 0-based value, per spec
In reply to Jay Freyensee--- drivers/nvme/target/loop.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/nvme/target/loop.c +2 -2
@@ -558,7 +558,7 @@ static int nvme_loop_create_io_queues(struct nvme_loop_ctrl *ctrl)memset (&ctrl->tag_set,0 ,sizeof (ctrl->tag_set)); ctrl->tag_set.ops = &nvme_loop_mq_ops; - ctrl->tag_set.queue_depth = ctrl->ctrl.sqsize; + ctrl->tag_set.queue_depth = ctrl->ctrl.opts->queue_size; ctrl->tag_set.reserved_tags =1 ;/* fabric connect */ ctrl->tag_set.numa_node =NUMA_NO_NODE ; ctrl->tag_set.flags =BLK_MQ_F_SHOULD_MERGE ;
@@ -622,7 +622,7 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev, ret = -ENOMEM ; - ctrl->ctrl.sqsize = opts->queue_size; + ctrl->ctrl.sqsize = opts->queue_size -1 ; ctrl->ctrl.kato = opts->kato; ctrl->queues = kcalloc(opts->nr_io_queues + 1, sizeof(*ctrl->queues),
Signature · 1 line
2.7.4