Skip to contentSkip to conversation

sqsize zero-based fixes

RFC PATCH New Jay Freyensee 3 participants 13 messages Aug 11, 2016

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover sqsize zero-based fixes

These patches are based on NVMe-over-Fabrics spec clarifcations by Dave Minturn, one of the spec authors.
In summary, sqsize and hsqsize are 0-based
values, and hrqsize cannot be smaller than hsqsize+1.

There is an issue with these patches I haven't worked out yet, thus the reason for RFC. The keep-alive timer is going off then causes a kernel crash and I haven't worked out exactly why yet, but I'm thinking
it has something to do with sqsize now being 0-based but the RDMA implemenation layer's queue_size being 1's based.

Anyways, feedback appreciated as I continue
to work on how to fix this appropriately.

Jay Freyensee (4):
fabrics: define admin sqsize min default, per spec
nvme-rdma: fix sqsize/hsqsize/hrqsize per spec
nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize
nvme-loop: set sqsize to 0-based value, per spec

 drivers/nvme/host/fabrics.c | 16 +++++++++++++++-
 drivers/nvme/host/nvme.h    |  1 +
 drivers/nvme/host/rdma.c    | 19 ++++++++++++++++---
 drivers/nvme/target/loop.c  |  2 +-
 drivers/nvme/target/rdma.c  |  8 ++++----
 5 files changed, 37 insertions(+), 9 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

RFC PATCH 1/4 fabrics: define admin sqsize min default, per spec

Upon admin queue connect(), the rdma qp was being set based on NVMF_AQ_DEPTH. However, the fabrics layer was using the sqsize field value set for I/O queues for the admin queue, which through the nvme layer and rdma layer off-whack:

root at fedora23-fabrics-host1 nvmf]# dmesg

[ 3507.798642] nvme_fabrics: nvmf_connect_admin_queue():admin sqsize
being sent is: 128
[ 3507.798858] nvme nvme0: creating 16 I/O queues.
[ 3507.896407] nvme nvme0: new ctrl: NQN "nullside-nqn", addr

192.168.1.3:4420

Thus, to have a different admin queue value (which the fabrics spec states the minimum depth for a fabrics admin queue is 32 via the ASQSZ definition), we need also a new variable to hold the sqsize for admin fabrics queue. This also allows fabric implementation layers to set an admin sqsize greater than the specification-defined minimum, which the fabrics spec allows.

Reported-by Daniel Verkamp <daniel.verkamp at intel.com>
Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>
---
 drivers/nvme/host/fabrics.c | 16 +++++++++++++++-
 drivers/nvme/host/nvme.h    |  1 +
 2 files changed, 16 insertions(+), 1 deletion(-)
drivers/nvme/host/fabrics.c +15 -1
index dc99676..cfc3607 100644
@@ -363,7 +363,14 @@ int nvmf_connect_admin_queue(struct nvme_ctrl *ctrl)
 	cmd.connect.opcode = nvme_fabrics_command;
 	cmd.connect.fctype = nvme_fabrics_type_connect;
 	cmd.connect.qid = 0;
-	cmd.connect.sqsize = cpu_to_le16(ctrl->sqsize);
+
+	/*
+	 * fabrics spec sets a minimum of depth 32 for admin queue, but
+	 * fabric implementation layer may choose to define something deeper.
+	 */
+	cmd.connect.sqsize = (ctrl->admin_sqsize < NVMF_AQ_DEPTH-1) ?
+			     cpu_to_le16(NVMF_AQ_DEPTH-1) :
+			     cpu_to_le16(ctrl->admin_sqsize);
 	/*
 	 * Set keep-alive timeout in seconds granularity (ms * 1000)
 	 * and add a grace period for controller kato enforcement
@@ -833,6 +840,13 @@ static ssize_t nvmf_dev_write(struct file *file, const char __user *ubuf,
 
 	seq_file->private = ctrl;
 
+	/*
+	 * set admin_sqsize a minimum default value
+	 * but allow fabric implementation modules
+	 * to define a larger sqsize, if desired
+	 */
+	ctrl->admin_sqsize = NVMF_AQ_DEPTH-1;
+
 out_unlock:
 	mutex_unlock(&nvmf_dev_mutex);
 	kfree(buf);
drivers/nvme/host/nvme.h +1 -0
index ab18b78..32577a7 100644
@@ -137,6 +137,7 @@ struct nvme_ctrl {
 	struct delayed_work ka_work;
 
 	/* Fabrics only */
+	u16 admin_sqsize;
 	u16 sqsize;
 	u32 ioccsz;
 	u32 iorcsz;
Signature · 1 line
2.7.4
maintainer
#2 lore raw

RFC PATCH 1/4 fabrics: define admin sqsize min default, per spec

Quoted 10 lines · Jay Freyensee

Upon admin queue connect(), the rdma qp was being set based on NVMF_AQ_DEPTH. However, the fabrics layer was using the sqsize field value set for I/O queues for the admin queue, which through the nvme layer and rdma layer off-whack:

root at fedora23-fabrics-host1 nvmf]# dmesg

[ 3507.798642] nvme_fabrics: nvmf_connect_admin_queue():admin sqsize
being sent is: 128
[ 3507.798858] nvme nvme0: creating 16 I/O queues.
[ 3507.896407] nvme nvme0: new ctrl: NQN "nullside-nqn", addr
Quoted 8 lines

192.168.1.3:4420

Thus, to have a different admin queue value (which the fabrics spec states the minimum depth for a fabrics admin queue is 32 via the ASQSZ definition), we need also a new variable to hold the sqsize for admin fabrics queue. This also allows fabric implementation layers to set an admin sqsize greater than the specification-defined minimum, which the fabrics spec allows.

We only need a variable if we actually have someone making use of it with the non-default value. I honestly don't see the reason to keep it in a variable when no one is making real use of it.

If you would have exposed it to user-space then it would make better sense (although I'm not a fan of adding not very useful configs to user-space).

#3 lore raw

RFC PATCH 1/4 fabrics: define admin sqsize min default, per spec

Quoted 7 lines · Sagi Grimberg

it with the non-default value. I honestly don't see the reason to keep it in a variable when no one is making real use of it.

If you would have exposed it to user-space then it would make better sense (although I'm not a fan of adding not very useful configs to user-space).

I agree - until we have a good use case for large admin queues we should keep things simple. And I'm still pissed about the idiotic addition of the admin queue size to the discovery records in the last moment. It's entirely contrary to the disccovery service abstraction we build, and pointless as well. So the more we can ignore it, the better.

#4 lore raw

RFC PATCH 1/4 fabrics: define admin sqsize min default, per spec

Quoted 10 lines · Christoph Hellwig
Quoted 8 lines · Sagi Grimberg

it with the non-default value. I honestly don't see the reason to keep it in a variable when no one is making real use of it.

If you would have exposed it to user-space then it would make better sense (although I'm not a fan of adding not very useful configs to user-space).

Quoted 6 lines

I agree - until we have a good use case for large admin queues we should keep things simple.??And I'm still pissed about the idiotic addition of the admin queue size to the discovery records in the last moment.??It's entirely contrary to the disccovery service abstraction we build, and pointless as well.??So the more we can ignore it, the better.

OK, I have no problems with that. ?So #define set the admin sqsize in nvmf_connect_admin_queue() OK?

#5 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

In reply to Jay Freyensee

Per NVMe-over-Fabrics 1.0 spec, sqsize is represented as a 0-based value.

Also per spec, the RDMA binding values shall be set to sqsize, which makes hsqsize 0-based values.

Also per spec, but not very clear, is hrqsize is +1 of hsqsize.

Thus, the sqsize during NVMf connect() is now:

[root at fedora23-fabrics-host1 for-48]# dmesg

[  318.720645] nvme_fabrics: nvmf_connect_admin_queue(): sqsize for
admin queue: 31
[  318.720884] nvme nvme0: creating 16 I/O queues.
[  318.810114] nvme_fabrics: nvmf_connect_io_queue(): sqsize for i/o

queue: 127

Reported-by Daniel Verkamp <daniel.verkamp at intel.com>
Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>
---
 drivers/nvme/host/rdma.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
drivers/nvme/host/rdma.c +16 -3
index 3e3ce2b..8be64f1 100644
@@ -1284,8 +1284,21 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue)
 
 	priv.recfmt = cpu_to_le16(NVME_RDMA_CM_FMT_1_0);
 	priv.qid = cpu_to_le16(nvme_rdma_queue_idx(queue));
-	priv.hrqsize = cpu_to_le16(queue->queue_size);
-	priv.hsqsize = cpu_to_le16(queue->queue_size);
+
+	/*
+	 * On one end, the fabrics spec is pretty clear that
+	 * hsqsize variables shall be set to the value of sqsize,
+	 * which is a 0-based number. What is confusing is the value for
+	 * hrqsize.  After clarification from NVMe spec committee member,
+	 * the minimum value of hrqsize is hsqsize+1.
+	 */
+	if (priv.qid == 0) {
+		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize);
+		priv.hrqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize+1);
+	} else {
+		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize);
+		priv.hrqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize+1);
+	}
 
 	ret = rdma_connect(queue->cm_id, &param);
 	if (ret) {
@@ -1907,7 +1920,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	spin_lock_init(&ctrl->lock);
 
 	ctrl->queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */
-	ctrl->ctrl.sqsize = opts->queue_size;
+	ctrl->ctrl.sqsize = opts->queue_size-1;
 	ctrl->ctrl.kato = opts->kato;
 
 	ret = -ENOMEM;
Signature · 1 line
2.7.4
maintainer
#6 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

Quoted 39 lines · Jay Freyensee

Per NVMe-over-Fabrics 1.0 spec, sqsize is represented as a 0-based value.

Also per spec, the RDMA binding values shall be set to sqsize, which makes hsqsize 0-based values.

Also per spec, but not very clear, is hrqsize is +1 of hsqsize.

Thus, the sqsize during NVMf connect() is now:

[root at fedora23-fabrics-host1 for-48]# dmesg

[  318.720645] nvme_fabrics: nvmf_connect_admin_queue(): sqsize for
admin queue: 31
[  318.720884] nvme nvme0: creating 16 I/O queues.
[  318.810114] nvme_fabrics: nvmf_connect_io_queue(): sqsize for i/o

queue: 127

Reported-by Daniel Verkamp <daniel.verkamp at intel.com>
Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>
---
 drivers/nvme/host/rdma.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
drivers/nvme/host/rdma.c +5 -2
index 3e3ce2b..8be64f1 100644
@@ -1284,8 +1284,21 @@ static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue)
 
 	priv.recfmt = cpu_to_le16(NVME_RDMA_CM_FMT_1_0);
 	priv.qid = cpu_to_le16(nvme_rdma_queue_idx(queue));
-	priv.hrqsize = cpu_to_le16(queue->queue_size);
-	priv.hsqsize = cpu_to_le16(queue->queue_size);
+
+	/*
+	 * On one end, the fabrics spec is pretty clear that
+	 * hsqsize variables shall be set to the value of sqsize,
+	 * which is a 0-based number. What is confusing is the value for

drivers/nvme/host/rdma.c static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue) · line 1301

Quoted 10 lines
drivers/nvme/host/rdma.c +10 -0
+	 * hrqsize.  After clarification from NVMe spec committee member,
+	 * the minimum value of hrqsize is hsqsize+1.
+	 */
+	if (priv.qid == 0) {
+		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize);
+		priv.hrqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize+1);
+	} else {
+		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize);
+		priv.hrqsize = cpu_to_le16(queue->ctrl->ctrl.sqsize+1);
+	}

Huh? (scratch...) using priv.hrqsize = priv.hsqsize+1 is pointless.

We expose to the block layer X and we send to the target X-1 and the target does X+1 (looks goofy, but ok). We also size our RDMA send/recv according to X so why on earth would we want to tell the target we have a recv queue of size X+1

#7 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

Quoted 54 lines · Sagi Grimberg
Quoted 52 lines · Jay Freyensee

Per NVMe-over-Fabrics 1.0 spec, sqsize is represented as a 0-based value.

Also per spec, the RDMA binding values shall be set to sqsize, which makes hsqsize 0-based values.

Also per spec, but not very clear, is hrqsize is +1 of hsqsize.

Thus, the sqsize during NVMf connect() is now:

[root at fedora23-fabrics-host1 for-48]# dmesg [??318.720645] nvme_fabrics: nvmf_connect_admin_queue(): sqsize for admin queue: 31
[??318.720884] nvme nvme0: creating 16 I/O queues. [??318.810114] nvme_fabrics: nvmf_connect_io_queue(): sqsize for i/o
queue: 127

Reported-by Daniel Verkamp <daniel.verkamp at intel.com>
Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>

---
?drivers/nvme/host/rdma.c | 19 ++++++++++++++++--- ?1 file changed, 16 insertions(+), 3 deletions(-)

drivers/nvme/host/rdma.c +5 -2
index 3e3ce2b..8be64f1 100644
@@ -1284,8 +1284,21 @@ static int nvme_rdma_route_resolved(struct
 nvme_rdma_queue *queue)
 
 ?	priv.recfmt = cpu_to_le16(NVME_RDMA_CM_FMT_1_0);
 ?	priv.qid = cpu_to_le16(nvme_rdma_queue_idx(queue));
-	priv.hrqsize = cpu_to_le16(queue->queue_size);
-	priv.hsqsize = cpu_to_le16(queue->queue_size);
+
+	/*
+	?* On one end, the fabrics spec is pretty clear that
+	?* hsqsize variables shall be set to the value of sqsize,
+	?* which is a 0-based number. What is confusing is the

value for

drivers/nvme/host/rdma.c +1 -0
+	?* hrqsize.??After clarification from NVMe spec committee

member,

drivers/nvme/host/rdma.c +4 -0
+	?* the minimum value of hrqsize is hsqsize+1.
+	?*/
+	if (priv.qid == 0) {
+		priv.hsqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.admin_sqsize);

drivers/nvme/host/rdma.c +1 -0
+		priv.hrqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.admin_sqsize+1);

Quoted 8 lines

drivers/nvme/host/rdma.c static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue) · line 1301

Quoted 6 lines
drivers/nvme/host/rdma.c +2 -0
+	} else {
+		priv.hsqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.sqsize);

drivers/nvme/host/rdma.c +1 -0
+		priv.hrqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.sqsize+1);

drivers/nvme/host/rdma.c +1 -0
+	}

Huh? (scratch...) using priv.hrqsize = priv.hsqsize+1 is pointless.

It may be pointless, but Dave said that is the current interpretation of the NVMe-over-Fabrics spec (which I don't really understand either).

Quoted 5 lines

We expose to the block layer X and we send to the target X-1 and the target does X+1 (looks goofy, but ok). We also size our RDMA send/recv according to X so why on earth would we want to tell the target we have a recv queue of size X+1

Could be the reason I see kato timeouts then kernel crashing...

#8 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

In reply to Jay Freyensee

I think having different notations for hsqsize vs hrqsize is highly confusing, and I've asked for a clarification of the hrqsize definition which is ambious at it's best at the moment. But independent of that a few more comments:

drivers/nvme/host/rdma.c static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, · line 1923

Quoted 6 lines
drivers/nvme/host/rdma.c +1 -1
@@ -1907,7 +1920,7 @@ static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev,
 	spin_lock_init(&ctrl->lock);
 
 	ctrl->queue_count = opts->nr_io_queues + 1; /* +1 for admin queue */
-	ctrl->ctrl.sqsize = opts->queue_size;
+	ctrl->ctrl.sqsize = opts->queue_size-1;

We should keep our own sqsize in normal notation in core, and just convert to strange notations on the wire. Especially as we use it to driver all the resource allocation as Sagi said.

drivers/nvme/host/rdma.c static int nvme_rdma_route_resolved(struct nvme_rdma_queue *queue) · line 1297

Quoted 3 lines
drivers/nvme/host/rdma.c +3 -0
+	if (priv.qid == 0) {
+		priv.hsqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize);
+		priv.hrqsize = cpu_to_le16(queue->ctrl->ctrl.admin_sqsize+1);

Based on that priv.hsqsize should have a - 1 here and hrqsize should be kept as-is. Also always use spaces around the operators.

And while we're at it - the fix to use the separate AQ values should go into the first patch.

#9 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

Quoted 22 lines · Christoph Hellwig

I think having different notations for hsqsize vs hrqsize is highly confusing, and I've asked for a clarification of the hrqsize definition
which is ambious at it's best at the moment.??But independent of that a few more comments:

drivers/nvme/host/rdma.c static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, · line 1923

Quoted 9 lines
(quoted diff) +0 -0
@@ -1907,7 +1920,7 @@ static struct nvme_ctrl
 *nvme_rdma_create_ctrl(struct device *dev,
 ?	spin_lock_init(&ctrl->lock);

?
? ctrl->queue_count = opts->nr_io_queues + 1; /* +1 for
admin queue */

drivers/nvme/host/rdma.c +1 -1
-	ctrl->ctrl.sqsize = opts->queue_size;
+	ctrl->ctrl.sqsize = opts->queue_size-1;

We should keep our own sqsize in normal notation in core, and just convert to strange notations on the wire.??Especially as we use it to driver all the resource allocation as Sagi said.

Quoted 1 line
Quoted 8 lines
Quoted 5 lines
(quoted diff) +2 -0
+	if (priv.qid == 0) {
+		priv.hsqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.admin_sqsize);

(quoted diff) +1 -0
+		priv.hrqsize = cpu_to_le16(queue->ctrl-
Quoted 1 line

ctrl.admin_sqsize+1);

Based on that priv.hsqsize should have a - 1 here??and hrqsize should be kept as-is.??Also always use spaces around the operators.

Hmmm, not sure I understand. ?In this current patch series implementation, admin_sqsize will be NVMF_AQ_DEPTH-1 (31), so it doesn't need a -1.

Then you are saying do NOT add hrqsize? ?Thus, hsqsize == hrqsize??

Quoted 3 lines

And while we're at it - the fix to use the separate AQ values should go into the first patch.

It will get re-worked anyways based on the interest to not have a variable at the moment at someone's disposal to create variable-sized admin queue depths.

#10 lore raw

RFC PATCH 2/4 nvme-rdma: fix sqsize/hsqsize/hrqsize per spec

In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig

I think having different notations for hsqsize vs hrqsize is highly confusing, and I've asked for a clarification of the hrqsize definition
which is ambious at it's best at the moment.??But independent of that a few more comments:

Quoted 1 line
Quoted 12 lines

drivers/nvme/host/rdma.c static struct nvme_ctrl *nvme_rdma_create_ctrl(struct device *dev, · line 1923

Quoted 8 lines
(quoted diff) +0 -0
@@ -1907,7 +1920,7 @@ static struct nvme_ctrl
 *nvme_rdma_create_ctrl(struct device *dev,
 ?	spin_lock_init(&ctrl->lock);

?
? ctrl->queue_count = opts->nr_io_queues + 1; /* +1 for
admin queue */

drivers/nvme/host/rdma.c +1 -1
-	ctrl->ctrl.sqsize = opts->queue_size;
+	ctrl->ctrl.sqsize = opts->queue_size-1;

We should keep our own sqsize in normal notation in core, and just convert to strange notations on the wire.??Especially as we use it to driver all the resource allocation as Sagi said.

So what I'm understanding is to keep ctrl->ctrl.sqsize as ones-based in the code but when we actually do something like a connect() send sqsize-1 in the connect() command.

That almost works, but is still a tad ugly because there are code lines such as:

ctrl->ctrl.sqsize =
????????????????min_t(int, NVME_CAP_MQES(ctrl->cap) + 1, ctrl-

Quoted 1 line

ctrl.sqsize);

in which also need to be adjusted because what is currently going on here is NVME_CAP_MQES() is a zero-based value but ctrl->ctrl.sqsize is currently implemented as a one's-based value.

#11 lore raw

RFC PATCH 3/4 nvmet-rdma: +1 to *queue_size from hsqsize/hrqsize

In reply to Jay Freyensee

Now that the host will be sending hsqsize 0-based
values with hrqsize equaling hsqsize+1,
the target needs to be adjusted as well.

Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>
---
 drivers/nvme/target/rdma.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/nvme/target/rdma.c +4 -4
index e06d504..4a08946 100644
@@ -1004,11 +1004,11 @@ nvmet_rdma_parse_cm_connect_req(struct rdma_conn_param *conn,
 	queue->host_qid = le16_to_cpu(req->qid);
 
 	/*
-	 * req->hsqsize corresponds to our recv queue size
-	 * req->hrqsize corresponds to our send queue size
+	 * req->hsqsize corresponds to our recv queue size, plus 1
+	 * req->hrqsize corresponds to our send queue size, plus 1
 	 */
-	queue->recv_queue_size = le16_to_cpu(req->hsqsize);
-	queue->send_queue_size = le16_to_cpu(req->hrqsize);
+	queue->recv_queue_size = le16_to_cpu(req->hsqsize+1);
+	queue->send_queue_size = le16_to_cpu(req->hrqsize+1);
 
 	if (!queue->host_qid && queue->recv_queue_size > NVMF_AQ_DEPTH)
 		return NVME_RDMA_CM_INVALID_HSQSIZE;
Signature · 1 line
2.7.4
#12 lore raw

RFC PATCH 4/4 nvme-loop: set sqsize to 0-based value, per spec

In reply to Jay Freyensee
Signed-off-by Jay Freyensee <james_p_freyensee at linux.intel.com>
---
 drivers/nvme/target/loop.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/loop.c +1 -1
index 94e7829..21ce0e8 100644
@@ -622,7 +622,7 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev,
 
 	ret = -ENOMEM;
 
-	ctrl->ctrl.sqsize = opts->queue_size;
+	ctrl->ctrl.sqsize = opts->queue_size-1;
 	ctrl->ctrl.kato = opts->kato;
 
 	ctrl->queues = kcalloc(opts->nr_io_queues + 1, sizeof(*ctrl->queues),
Signature · 1 line
2.7.4