nvmet-tcp: fix connect error when inline data size to 0
Subsystems (2)
cover nvmet-tcp: fix connect error when inline data size to 0
Hi,
For nvme over tcp, connect error happens when setting inline data size to 0. Patch 2 tries to fix this.
Patch 1 is a small fixes, it could be merged with patch 2 if you want.
Hou Pu (2):
nvmet-tcp: fix inline data size comparison in nvmet_tcp_queue_response
nvmet-tcp: fix connect error when setting param_inline_data_size to
zero.
drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
Signature · 7 lines
2.28.0 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 1/2 nvmet-tcp: fix inline data size comparison in nvmet_tcp_queue_response
Using "<=" instead "<" to compare inline data size.
--- drivers/nvme/target/tcp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/target/tcp.c +1 -1
@@ -550,7 +550,7 @@ static void nvmet_tcp_queue_response(struct nvmet_req *req)* nvmet_req_init is completed. */ if (queue->rcv_state ==NVMET_TCP_RECV_PDU && - len && len < cmd->req.port->inline_data_size && + len && len <= cmd->req.port->inline_data_size &&nvme_is_write (cmd->req.cmd))return ; }
Signature · 7 lines
2.28.0 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/2] nvmet-tcp: fix inline data size comparison in nvmet_tcp_queue_response
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/2] nvmet-tcp: fix inline data size comparison in nvmet_tcp_queue_response
Quoted 2 lines · Sagi Grimberg
Reviewed-by Sagi Grimberg <[email protected]>
Thanks,
Hou
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 1/2] nvmet-tcp: fix inline data size comparison in nvmet_tcp_queue_response
In reply to Hou PuThanks,
applied to nvme-5.13.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH 2/2 nvmet-tcp: fix connect error when setting param_inline_data_size to zero.
In reply to Hou PuWhen setting inline_data_size to zero, connect failed. This could be reproduced with following steps.
Controller side:
mkdir /sys/kernel/config/nvmet/ports/1
cd /sys/kernel/config/nvmet/ports/1
echo 0.0.0.0 > addr_traddr
echo 4421 > addr_trsvcid
echo ipv4 > addr_adrfam
echo tcp > addr_trtype
echo 0 > param_inline_data_size
ln -s /sys/kernel/config/nvmet/subsystems/mysub /sys/kernel/config/nvmet/ports/1/subsystems/mysub
Host side:
[ 325.145323][ T203] nvme nvme1: Connect command failed, error wo/DNR bit: 22 [ 325.159481][ T203] nvme nvme1: failed to connect queue: 0 ret=16406
Failed to write to /dev/nvme-fabrics: Input/output error
Kernel log from controller side is:
[ 114.567411][ T56] nvmet_tcp: queue 0: failed to map data [ 114.568093][ T56] nvmet_tcp: unexpected pdu type 201
When admin-connect command comes with 1024 inline data size, in nvmet_tcp_map_data(), this size is compared with cmd->req.port->inline_data_size (which is 0), thus the command is responded with an error code. But admin-connect command is always allowed to use no more than 8192 bytes according to the nvme over fabrics specification.
The host side decides the inline data size when allocating the queue according the queue number, the size of queue 0 is 8k and others is ioccsz*16. The target side should do like this.
--- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +21 -3
@@ -167,6 +167,24 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops;static void nvmet_tcp_free_cmd (struct nvmet_tcp_cmd *c);static void nvmet_tcp_finish_cmd (struct nvmet_tcp_cmd *cmd); +static inline int nvmet_tcp_inline_data_size (struct nvmet_tcp_cmd *cmd) +{ +struct nvmet_tcp_queue *queue = cmd->queue; +struct nvme_command *nvme_cmd = cmd->req.cmd; +int inline_data_size =NVME_TCP_ADMIN_CCSZ ; +u16 qid =0 ; + +if (likely (queue->nvme_sq.ctrl)) { +/* The connect admin/io queue has been executed. */ + qid = queue->nvme_sq.qid; +if (qid) + inline_data_size = cmd->req.port->inline_data_size; + }else if (nvme_cmd->connect.qid) + inline_data_size = cmd->req.port->inline_data_size; + +return inline_data_size; +} +static inline u16 nvmet_tcp_cmd_tag (struct nvmet_tcp_queue *queue,struct nvmet_tcp_cmd *cmd) {
@@ -367,7 +385,7 @@ static int nvmet_tcp_map_data(struct nvmet_tcp_cmd *cmd)if (!nvme_is_write (cmd->req.cmd))return NVME_SC_INVALID_FIELD |NVME_SC_DNR ; -if (len > cmd->req.port->inline_data_size) +if (len >nvmet_tcp_inline_data_size (cmd))return NVME_SC_SGL_INVALID_OFFSET |NVME_SC_DNR ; cmd->pdu_len = len; }
@@ -550,7 +568,7 @@ static void nvmet_tcp_queue_response(struct nvmet_req *req)* nvmet_req_init is completed. */ if (queue->rcv_state ==NVMET_TCP_RECV_PDU && - len && len <= cmd->req.port->inline_data_size && + len && len <=nvmet_tcp_inline_data_size (cmd) &&nvme_is_write (cmd->req.cmd))return ; }
@@ -907,7 +925,7 @@ static void nvmet_tcp_handle_req_failure(struct nvmet_tcp_queue *queue,int ret;if (!nvme_is_write (cmd->req.cmd) || - data_len > cmd->req.port->inline_data_size) { + data_len >nvmet_tcp_inline_data_size (cmd)) {nvmet_prepare_receive_pdu (queue);return ; }
Signature · 7 lines
2.28.0 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 2/2] nvmet-tcp: fix connect error when setting param_inline_data_size to zero.
Quoted 51 lines
When setting
inline_data_sizeto zero, connect failed. This could be reproduced with following steps.Controller side:
mkdir/sys/kernel/config/nvmet/ports/1
cd/sys/kernel/config/nvmet/ports/1
echo 0.0.0.0 >addr_traddr
echo 4421 >addr_trsvcid
echo ipv4 >addr_adrfam
echo tcp >addr_trtype
echo 0 >param_inline_data_size
ln -s/sys/kernel/config/nvmet/subsystems/mysub/sys/kernel/config/nvmet/ports/1/subsystems/mysubHost side:
[ 325.145323][ T203] nvme nvme1: Connect command failed, error wo/DNR bit: 22 [ 325.159481][ T203] nvme nvme1: failed to connect queue: 0 ret=16406Failed to write to
/dev/nvme-fabrics: Input/output errorKernel log from controller side is:
[ 114.567411][ T56] nvmet_tcp: queue 0: failed to map data [ 114.568093][ T56] nvmet_tcp: unexpected pdu type 201When admin-connect command comes with 1024 inline data size, in
nvmet_tcp_map_data(), this size is compared withcmd->req.port->inline_data_size(which is 0), thus the command is responded with an error code. But admin-connect command is always allowed to use no more than 8192 bytes according to the nvme over fabrics specification.The host side decides the inline data size when allocating the queue according the queue number, the size of queue 0 is 8k and others is ioccsz*16. The target side should do like this.
Fixes 0d5ee2b2ab4f ("nvmet-rdma: support max(16KB, PAGE_SIZE) inline data")Signed-off-by Hou Pu <[email protected]>--- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)drivers/nvme/target/tcp.c +4
-0@@ -167,6 +167,24 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops;static void nvmet_tcp_free_cmd (struct nvmet_tcp_cmd *c);static void nvmet_tcp_finish_cmd (struct nvmet_tcp_cmd *cmd); +static inline int nvmet_tcp_inline_data_size (struct nvmet_tcp_cmd *cmd) +{ +struct nvmet_tcp_queue *queue = cmd->queue; +struct nvme_command *nvme_cmd = cmd->req.cmd;
drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops; · line 183
Quoted 10 lines
drivers/nvme/target/tcp.c +10
-0+int inline_data_size =NVME_TCP_ADMIN_CCSZ ; +u16 qid =0 ; + +if (likely (queue->nvme_sq.ctrl)) { +/* The connect admin/io queue has been executed. */ + qid = queue->nvme_sq.qid; +if (qid) + inline_data_size = cmd->req.port->inline_data_size; + }else if (nvme_cmd->connect.qid) + inline_data_size = cmd->req.port->inline_data_size;
How can a connection to an I/O queue arrive without having the ctrl reference installed? Is this for the failure case?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 2/2] nvmet-tcp: fix connect error when setting param_inline_data_size to zero.
Quoted 57 lines · Sagi Grimberg
Quoted 57 lines
When setting
inline_data_sizeto zero, connect failed. This could be reproduced with following steps.Controller side:
mkdir/sys/kernel/config/nvmet/ports/1
cd/sys/kernel/config/nvmet/ports/1
echo 0.0.0.0 >addr_traddr
echo 4421 >addr_trsvcid
echo ipv4 >addr_adrfam
echo tcp >addr_trtype
echo 0 >param_inline_data_size
ln -s/sys/kernel/config/nvmet/subsystems/mysub/sys/kernel/config/nvmet/ports/1/subsystems/mysubHost side:
[ 325.145323][ T203] nvme nvme1: Connect command failed, error wo/DNR bit: 22 [ 325.159481][ T203] nvme nvme1: failed to connect queue: 0 ret=16406Failed to write to
/dev/nvme-fabrics: Input/output errorKernel log from controller side is:
[ 114.567411][ T56] nvmet_tcp: queue 0: failed to map data [ 114.568093][ T56] nvmet_tcp: unexpected pdu type 201When admin-connect command comes with 1024 inline data size, in
nvmet_tcp_map_data(), this size is compared withcmd->req.port->inline_data_size(which is 0), thus the command is responded with an error code. But admin-connect command is always allowed to use no more than 8192 bytes according to the nvme over fabrics specification.The host side decides the inline data size when allocating the queue according the queue number, the size of queue 0 is 8k and others is ioccsz*16. The target side should do like this.
Fixes 0d5ee2b2ab4f ("nvmet-rdma: support max(16KB, PAGE_SIZE) inline data")Signed-off-by Hou Pu <[email protected]>--- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)drivers/nvme/target/tcp.c +9
-0@@ -167,6 +167,24 @@ static const struct nvmet_fabrics_ops nvmet_tcp_ops;static void nvmet_tcp_free_cmd (struct nvmet_tcp_cmd *c);static void nvmet_tcp_finish_cmd (struct nvmet_tcp_cmd *cmd); +static inline int nvmet_tcp_inline_data_size (struct nvmet_tcp_cmd *cmd) +{ +struct nvmet_tcp_queue *queue = cmd->queue; +struct nvme_command *nvme_cmd = cmd->req.cmd; +int inline_data_size =NVME_TCP_ADMIN_CCSZ ; +u16 qid =0 ; + +if (likely (queue->nvme_sq.ctrl)) { +/* The connect admin/io queue has been executed. */
Quoted 8 lines
drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops; · line 183
Quoted 5 lines
drivers/nvme/target/tcp.c +5
-0+ qid = queue->nvme_sq.qid; +if (qid) + inline_data_size = cmd->req.port->inline_data_size; + }else if (nvme_cmd->connect.qid) + inline_data_size = cmd->req.port->inline_data_size;How can a connection to an I/O queue arrive without having the ctrl reference installed? Is this for the failure case?
Hi Sagi,
AFAIK after the host finishes setting up the admin queue,
it connects to the io queue and sends the io-connect command. At this point the nvmet_tcp_queue is firstly allocated and does not have a valid queue->nvme_sq.ctrl. It is assigned after io-connect in nvmet_install_queue(). So this function tries to find the correct queue number before or after a fabrics connect command.
Thank,
Hou
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerRe: [PATCH 2/2] nvmet-tcp: fix connect error when setting param_inline_data_size to zero.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
drivers/nvme/target/tcp.c +9
-0+static inline int nvmet_tcp_inline_data_size (struct nvmet_tcp_cmd *cmd) +{ +struct nvmet_tcp_queue *queue = cmd->queue; +struct nvme_command *nvme_cmd = cmd->req.cmd; +int inline_data_size =NVME_TCP_ADMIN_CCSZ ; +u16 qid =0 ; + +if (likely (queue->nvme_sq.ctrl)) { +/* The connect admin/io queue has been executed. */ Quoted 8 lines
drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops; · line 183
Quoted 5 lines
drivers/nvme/target/tcp.c +5
-0+ qid = queue->nvme_sq.qid; +if (qid) + inline_data_size = cmd->req.port->inline_data_size; + }else if (nvme_cmd->connect.qid) + inline_data_size = cmd->req.port->inline_data_size;How can a connection to an I/O queue arrive without having the ctrl reference installed? Is this for the failure case?
Quoted 7 lines
Hi Sagi,
AFAIK after the host finishes setting up the admin queue,
it connects to the io queue and sends the io-connect command. At this point thenvmet_tcp_queueis firstly allocated and does not have a validqueue->nvme_sq.ctrl. It is assigned after io-connect innvmet_install_queue(). So this function tries to find the correct queue number before or after a fabrics connect command.
Why do you need the inline_data_size before the connect? its only relevant for nvme I/O..
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Re: [PATCH 2/2] nvmet-tcp: fix connect error when setting param_inline_data_size to zero.
Quoted 28 lines · Sagi Grimberg
Quoted 20 lines
Quoted 11 lines
Quoted 11 lines
drivers/nvme/target/tcp.c +9
-0+static inline int nvmet_tcp_inline_data_size (struct nvmet_tcp_cmd *cmd) +{ +struct nvmet_tcp_queue *queue = cmd->queue; +struct nvme_command *nvme_cmd = cmd->req.cmd; +int inline_data_size =NVME_TCP_ADMIN_CCSZ ; +u16 qid =0 ; + +if (likely (queue->nvme_sq.ctrl)) { +/* The connect admin/io queue has been executed. */ Quoted 8 lines
drivers/nvme/target/tcp.c static const struct nvmet_fabrics_ops nvmet_tcp_ops; · line 183
Quoted 5 lines
drivers/nvme/target/tcp.c +5
-0+ qid = queue->nvme_sq.qid; +if (qid) + inline_data_size = cmd->req.port->inline_data_size; + }else if (nvme_cmd->connect.qid) + inline_data_size = cmd->req.port->inline_data_size;How can a connection to an I/O queue arrive without having the ctrl reference installed? Is this for the failure case?
Quoted 7 lines
Hi Sagi,
AFAIK after the host finishes setting up the admin queue,
it connects to the io queue and sends the io-connect command. At this point thenvmet_tcp_queueis firstly allocated and does not have a validqueue->nvme_sq.ctrl. It is assigned after io-connect innvmet_install_queue(). So this function tries to find the correct queue number before or after a fabrics connect command.
Quoted 2 lines
Why do you need the
inline_data_sizebefore the connect? its only relevant for nvme I/O..
From my observation, the host always sends an admin-connect command with nvmf_connect_data as inline data (by setting sg type to NVME_SGL_FMT_DATA_DESC). This is fixed from the host side.
For io-connect command, the host sends nvmf_connect_data according to ioccsz. If the size of nvmf_connect_data is smaller than ioccsz*16 it is sent as inline data. ioccsz is obtained by identifying the controller and is controlled by param_inline_data_size from the target side.
If param_inline_data_size is set to zero, admin-connect will not be received by target correctly. As target side thought there is no inline data (by looking param_inline_data_size), but in fact admin-connect has inline data.
This patch is trying to make param_inline_data_size only affect io-queue. and admin-queue always allow 8k inline data. This is what the host side do currently and also what the specification says AFAIK.
Thanks,
Hou
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme