Skip to contentSkip to conversation

nvme-fc: set max_segments to lldd max value

PATCH Applied James Smart 5 participants 5 messages Jul 14, 2020 – Jul 26, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme-fc: set max_segments to lldd max value

Currently the FC transport is set max_hw_sectors based on the lldds max sgl segment count. However, the block queue max segments is set based on the controller's max_segments count, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.

Set the controller max segment count and derive max_hw_sectors from the max segment count.

---
Looks like the setting of max_segments has been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.

---
 drivers/nvme/host/fc.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fc.c +3 -2
index 6aa30bb5a762..e57e536546f7 100644
@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
 	if (ret)
 		goto out_disconnect_admin_queue;
 
-	ctrl->ctrl.max_hw_sectors =
-		(ctrl->lport->ops->max_sgl_segments - 1) << (PAGE_SHIFT - 9);
+	ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments;
+	ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments <<
+						(ilog2(SZ_4K) - 9);
 
 	blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
 
Signature · 7 lines
2.26.2


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by on v1 1/1
#1 lore raw
Quoted 13 lines · James Smart

Currently the FC transport is set max_hw_sectors based on the lldds max sgl segment count. However, the block queue max segments is set based on the controller's max_segments count, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.

Set the controller max segment count and derive max_hw_sectors from the max segment count.

Quoted 5 lines

---
Looks like the setting of max_segments has been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.

The TCP stack is not limiting the IO size, I guess because of the fact it's a streaming protocol.

So I don't see a bug in the tcp driver but we can consider limiting the IO size if it will improve performance in some cases.

BTW, AFAIK the iscsi TCP acts the same.

Sagi,

do you see some benefits we can get by limiting the mdts for tcp ?

Quoted 10 lines
---
  drivers/nvme/host/fc.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fc.c +0 -0
index 6aa30bb5a762..e57e536546f7 100644
@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
  	if (ret)

drivers/nvme/host/fc.c nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) · line 3009

Quoted 10 lines
drivers/nvme/host/fc.c +3 -2
  		goto out_disconnect_admin_queue;
 
-	ctrl->ctrl.max_hw_sectors =
-		(ctrl->lport->ops->max_sgl_segments - 1) << (PAGE_SHIFT - 9);
+	ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments;
+	ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments <<
+						(ilog2(SZ_4K) - 9);
 
  	blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
 

Looks good,

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#2 lore raw
Quoted 22 lines
Quoted 14 lines

Currently the FC transport is set max_hw_sectors based on the lldds max sgl segment count. However, the block queue max segments is set based on the controller's max_segments count, which the transport does not set.  As such, the lldd is receiving sgl lists that are exceeding its max segment count.

Set the controller max segment count and derive max_hw_sectors from the max segment count.

Quoted 5 lines

---
Looks like the setting of max_segments has been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.

The TCP stack is not limiting the IO size, I guess because of the fact it's a streaming protocol.

Quoted 7 lines

So I don't see a bug in the tcp driver but we can consider limiting the IO size if it will improve performance in some cases.

BTW, AFAIK the iscsi TCP acts the same.

Sagi,

do you see some benefits we can get by limiting the mdts for tcp ?

Not really...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

Reviewed-by on v1 1/1
#3 lore raw
In reply to James Smart
Quoted 29 lines · James Smart

Currently the FC transport is set max_hw_sectors based on the lldds max sgl segment count. However, the block queue max segments is set based on the controller's max_segments count, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.

Set the controller max segment count and derive max_hw_sectors from the max segment count.

---
Looks like the setting of max_segments has been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.

---
  drivers/nvme/host/fc.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fc.c +0 -0
index 6aa30bb5a762..e57e536546f7 100644
@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)
  	if (ret)

drivers/nvme/host/fc.c nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) · line 3009

Quoted 10 lines
drivers/nvme/host/fc.c +3 -2
  		goto out_disconnect_admin_queue;
 
-	ctrl->ctrl.max_hw_sectors =
-		(ctrl->lport->ops->max_sgl_segments - 1) << (PAGE_SHIFT - 9);
+	ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments;
+	ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments <<
+						(ilog2(SZ_4K) - 9);
 
  	blk_mq_unquiesce_queue(ctrl->ctrl.admin_q);
 
Signature · 6 lines
Himanshu Madhani                     Oracle Linux Engineering

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme