support nvme-tcp SUCCESS flag optimization
Subsystems (2)
Sagi Grimberg
maintainercover support nvme-tcp SUCCESS flag optimization
The SUCCESS flag is a bit that exists in the C2HData PDU header and allows a controller to indicate that the command completed successfully from the controller perspective, and by setting it, the controller omits a subsequent completion response capsule PDU.
Its optional usage in the spec, but the host definitly needs to support it given that the controller is allowed to set it anytime.
Thanks to Steve Blightman and Oliver Smith-Denny for raising my attention on this.
Sagi Grimberg (2):
nvme-tcp: support C2HData with SUCCESS flag
nvmet-tcp: implement C2HData SUCCESS optimization
drivers/nvme/host/tcp.c | 32 ++++++++++++++++++++++++++++---- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 2 files changed, 49 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.1
Sagi Grimberg
maintainerPATCH 1/2 nvme-tcp: support C2HData with SUCCESS flag
A C2HData PDU with the SUCCESS flag set indicates that the I/O was completed by the controller successfully and means that a subsequent completion response capsule PDU will be ommitted.
If we see this flag, fisrt we check that LAST_PDU flag is set as well, and then we complete the request when the data transfer (and data digest verification if its on) is done.
While we're at it, reuse a bit of code with nvme_fail_request.
--- drivers/nvme/host/tcp.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-)
drivers/nvme/host/tcp.c +28 -4
@@ -463,6 +463,15 @@ static int nvme_tcp_handle_c2h_data(struct nvme_tcp_queue *queue, queue->data_remaining =le32_to_cpu (pdu->data_length); +if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS && +unlikely (!(pdu->hdr.flags &NVME_TCP_F_DATA_LAST ))) { +dev_err (queue->ctrl->ctrl.device, +"queue %d tag %#x SUCCESS set but not last PDU\n" , +nvme_tcp_queue_id (queue), rq->tag); +nvme_tcp_error_recovery (&queue->ctrl->ctrl); +return -EPROTO ; + } +return 0 ; }
@@ -618,6 +627,14 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,return ret; } +static inline void nvme_tcp_end_request (struct request *rq,__le16 status) +{ +union nvme_result res = {}; + +nvme_end_request (rq,cpu_to_le16 (status <<1 ), res); +} + +static int nvme_tcp_recv_data (struct nvme_tcp_queue *queue,struct sk_buff *skb,unsigned int *offset,size_t *len) {
@@ -685,6 +702,8 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,nvme_tcp_ddgst_final (queue->rcv_hash, &queue->exp_ddgst); queue->ddgst_remaining =NVME_TCP_DIGEST_LENGTH ; }else { +if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) +nvme_tcp_end_request (rq,NVME_SC_SUCCESS );nvme_tcp_init_recv_ctx (queue); } }
@@ -695,6 +714,7 @@ static int nvme_tcp_recv_data(struct nvme_tcp_queue *queue, struct sk_buff *skb,static int nvme_tcp_recv_ddgst (struct nvme_tcp_queue *queue,struct sk_buff *skb,unsigned int *offset,size_t *len) { +struct nvme_tcp_data_pdu *pdu = (void *)queue->pdu;char *ddgst = (char *)&queue->recv_ddgst;size_t recv_len =min_t (size_t, *len, queue->ddgst_remaining);off_t off =NVME_TCP_DIGEST_LENGTH - queue->ddgst_remaining;
@@ -718,6 +738,13 @@ static int nvme_tcp_recv_ddgst(struct nvme_tcp_queue *queue,return -EIO ; } +if (pdu->hdr.flags &NVME_TCP_F_DATA_SUCCESS ) { +struct request *rq =blk_mq_tag_to_rq (nvme_tcp_tagset (queue), + pdu->command_id); + +nvme_tcp_end_request (rq,NVME_SC_SUCCESS ); + } +nvme_tcp_init_recv_ctx (queue);return 0 ; }
@@ -815,10 +842,7 @@ static inline void nvme_tcp_done_send_req(struct nvme_tcp_queue *queue)static void nvme_tcp_fail_request (struct nvme_tcp_request *req) { -union nvme_result res = {}; - -nvme_end_request (blk_mq_rq_from_pdu (req), -cpu_to_le16 (NVME_SC_DATA_XFER_ERROR ), res); +nvme_tcp_end_request (blk_mq_rq_from_pdu (req),NVME_SC_DATA_XFER_ERROR ); }static int nvme_tcp_try_send_data (struct nvme_tcp_request *req)
Signature · 1 line
2.17.1
Sagi Grimberg
maintainerPATCH 1/2 nvme-tcp: support C2HData with SUCCESS flag
Quoted 11 lines
A C2HData PDU with the SUCCESS flag set indicates that the I/O was completed by the controller successfully and means that a subsequent completion response capsule PDU will be ommitted.
If we see this flag, fisrt we check that
LAST_PDUflag is set as well, and then we complete the request when the data transfer (and data digest verification if its on) is done.While we're at it, reuse a bit of code with
nvme_fail_request.
rrr...
This should also have:
Quoted 3 lines
Suggested-by Oliver Smith-Denny <osmithde at cisco.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>---
PATCH 1/2 nvme-tcp: support C2HData with SUCCESS flag
In reply to Sagi GrimbergQuoted 8 lines · Sagi Grimberg
A C2HData PDU with the SUCCESS flag set indicates that the I/O was completed by the controller successfully and means that a subsequent completion response capsule PDU will be ommitted.
If we see this flag, fisrt we check that
LAST_PDUflag is set as well, and then we complete the request when the data transfer (and data digest verification if its on) is done.
Quoted 7 lines
While we're at it, reuse a bit of code with
nvme_fail_request.Suggested-by Oliver Smith-Denny <osmithde at cisco.com>Signed-off-by Sagi Grimberg <sagi at grimberg.me>--- drivers/nvme/host/tcp.c | 32 ++++++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-)
Looks good! I don't see any issues.
Tested-and-Reviewed-by: Oliver Smith-Denny
Thanks Sagi!
Oliver
PATCH 1/2 nvme-tcp: support C2HData with SUCCESS flag
In reply to Sagi GrimbergThanks,
applied to nvme-5.1.
Sagi Grimberg
maintainerPATCH 2/2 nvmet-tcp: implement C2HData SUCCESS optimization
In reply to Sagi GrimbergTP 8000 says that the use of the SUCCESS flag depends on weather the controller support disabling sq_head pointer updates. Given that we support it by default, makes sense that we go the extra mile to actually use the SUCCESS flag.
When we create the C2HData PDU header, we check if sqhd_disabled is set on our queue, if so, we set the SUCCESS flag in the PDU header and skip sending a completion response capsule.
--- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
drivers/nvme/target/tcp.c +21 -3
@@ -371,7 +371,8 @@ static void nvmet_setup_c2h_data_pdu(struct nvmet_tcp_cmd *cmd) cmd->state =NVMET_TCP_SEND_DATA_PDU ; pdu->hdr.type = nvme_tcp_c2h_data; - pdu->hdr.flags =NVME_TCP_F_DATA_LAST ; + pdu->hdr.flags =NVME_TCP_F_DATA_LAST | (queue->nvme_sq.sqhd_disabled ? +NVME_TCP_F_DATA_SUCCESS :0 ); pdu->hdr.hlen =sizeof (*pdu); pdu->hdr.pdo = pdu->hdr.hlen + hdgst; pdu->hdr.plen =
@@ -542,8 +543,19 @@ static int nvmet_try_send_data(struct nvmet_tcp_cmd *cmd) cmd->state =NVMET_TCP_SEND_DDGST ; cmd->offset =0 ; }else { -nvmet_setup_response_pdu (cmd); +if (queue->nvme_sq.sqhd_disabled) { + cmd->queue->snd_cmd =NULL ; +nvmet_tcp_put_cmd (cmd); + }else { +nvmet_setup_response_pdu (cmd); + } } + +if (queue->nvme_sq.sqhd_disabled) { +kfree (cmd->iov); +sgl_free (cmd->req.sg); + } +return 1 ; }
@@ -619,7 +631,13 @@ static int nvmet_try_send_ddgst(struct nvmet_tcp_cmd *cmd)return ret; cmd->offset += ret; -nvmet_setup_response_pdu (cmd); + +if (queue->nvme_sq.sqhd_disabled) { + cmd->queue->snd_cmd =NULL ; +nvmet_tcp_put_cmd (cmd); + }else { +nvmet_setup_response_pdu (cmd); + }return 1 ; }
Signature · 1 line
2.17.1
PATCH 2/2 nvmet-tcp: implement C2HData SUCCESS optimization
Quoted 10 lines · Sagi Grimberg
TP 8000 says that the use of the SUCCESS flag depends on weather the controller support disabling
sq_headpointer updates. Given that we support it by default, makes sense that we go the extra mile to actually use the SUCCESS flag.When we create the C2HData PDU header, we check if
sqhd_disabledis set on our queue, if so, we set the SUCCESS flag in the PDU header and skip sending a completion response capsule.Signed-off-by Sagi Grimberg <sagi at grimberg.me>
Target looks good, too.
Tested-and-Reviewed-by: Oliver Smith-Denny
PATCH 2/2 nvmet-tcp: implement C2HData SUCCESS optimization
In reply to Sagi GrimbergThanks,
applied to nvme-5.2.
Sagi Grimberg
maintainercover support nvme-tcp SUCCESS flag optimization
In reply to Sagi GrimbergSteve, Oliver, Did you get a chance to test this out?
Quoted 19 lines
The SUCCESS flag is a bit that exists in the C2HData PDU header and allows a controller to indicate that the command completed successfully from the controller perspective, and by setting it, the controller omits a subsequent completion response capsule PDU.
Its optional usage in the spec, but the host definitly needs to support it given that the controller is allowed to set it anytime.
Thanks to Steve Blightman and Oliver Smith-Denny for raising my attention on this.
Sagi Grimberg (2):
nvme-tcp: support C2HData with SUCCESS flag
nvmet-tcp: implement C2HData SUCCESS optimizationdrivers/nvme/host/tcp.c | 32 ++++++++++++++++++++++++++++---- drivers/nvme/target/tcp.c | 24 +++++++++++++++++++++--- 2 files changed, 49 insertions(+), 7 deletions(-)
cover support nvme-tcp SUCCESS flag optimization
Quoted 1 line · Sagi Grimberg
Steve, Oliver, Did you get a chance to test this out?
Hi Sagi, sorry I was on vacation for a few days. I am planning on testing this patch out right now and I will let you know how it goes!
Thanks,
Oliver