nvme-fc: set max_segments to lldd max value
Subsystems (2)
PATCH 1/1 nvme-fc: set max_segments to lldd max value
Currently the FC transport is set max_hw_sectors based on the lldds max sgl segment count. However, the block queue max segments is set based on the controller's max_segments count, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.
Set the controller max segment count and derive max_hw_sectors from the max segment count.
---
Looks like the setting of max_segments has been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.
--- drivers/nvme/host/fc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/nvme/host/fc.c +3 -2
@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)if (ret) goto out_disconnect_admin_queue; - ctrl->ctrl.max_hw_sectors = - (ctrl->lport->ops->max_sgl_segments -1 ) << (PAGE_SHIFT -9 ); + ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments; + ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments << + (ilog2 (SZ_4K ) -9 );blk_mq_unquiesce_queue (ctrl->ctrl.admin_q);
Signature · 7 lines
2.26.2 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 13 lines · James Smart
Currently the FC transport is set
max_hw_sectorsbased on the lldds max sgl segment count. However, the block queue max segments is set based on the controller'smax_segmentscount, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.Set the controller max segment count and derive
max_hw_sectorsfrom the max segment count.Signed-off-by James Smart <[email protected]>Reviewed-by Ewan D. Milne <[email protected]>CC Max Gurtovoy <[email protected]>
Quoted 5 lines
---
Looks like the setting ofmax_segmentshas been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.
The TCP stack is not limiting the IO size, I guess because of the fact it's a streaming protocol.
So I don't see a bug in the tcp driver but we can consider limiting the IO size if it will improve performance in some cases.
BTW, AFAIK the iscsi TCP acts the same.
Sagi,
do you see some benefits we can get by limiting the mdts for tcp ?
Quoted 10 lines
--- drivers/nvme/host/fc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/nvme/host/fc.c +0
-0@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)if (ret)
drivers/nvme/host/fc.c nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) · line 3009
Quoted 10 lines
drivers/nvme/host/fc.c +3
-2goto out_disconnect_admin_queue; - ctrl->ctrl.max_hw_sectors = - (ctrl->lport->ops->max_sgl_segments -1 ) << (PAGE_SHIFT -9 ); + ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments; + ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments << + (ilog2 (SZ_4K ) -9 );blk_mq_unquiesce_queue (ctrl->ctrl.admin_q);
Looks good,
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 22 lines
Quoted 14 lines
Currently the FC transport is set
max_hw_sectorsbased on the lldds max sgl segment count. However, the block queue max segments is set based on the controller'smax_segmentscount, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.Set the controller max segment count and derive
max_hw_sectorsfrom the max segment count.Signed-off-by James Smart <[email protected]>Reviewed-by Ewan D. Milne <[email protected]>CC Max Gurtovoy <[email protected]>Quoted 5 lines
---
Looks like the setting ofmax_segmentshas been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.The TCP stack is not limiting the IO size, I guess because of the fact it's a streaming protocol.
Quoted 7 lines
So I don't see a bug in the tcp driver but we can consider limiting the IO size if it will improve performance in some cases.
BTW, AFAIK the iscsi TCP acts the same.
Sagi,
do you see some benefits we can get by limiting the mdts for tcp ?
Not really...
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 29 lines · James Smart
Currently the FC transport is set
max_hw_sectorsbased on the lldds max sgl segment count. However, the block queue max segments is set based on the controller'smax_segmentscount, which the transport does not set. As such, the lldd is receiving sgl lists that are exceeding its max segment count.Set the controller max segment count and derive
max_hw_sectorsfrom the max segment count.Signed-off-by James Smart <[email protected]>Reviewed-by Ewan D. Milne <[email protected]>CC Max Gurtovoy <[email protected]>---
Looks like the setting ofmax_segmentshas been missing from all fabric transports for a while. Rdma had a fixup (ff13c1b87c97) from Max last fall that corrected this. But tcp and fc were still lacking. Copying Max to look at tcp.--- drivers/nvme/host/fc.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/nvme/host/fc.c +0
-0@@ -3001,8 +3001,9 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl)if (ret)
drivers/nvme/host/fc.c nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) · line 3009
Quoted 10 lines
drivers/nvme/host/fc.c +3
-2goto out_disconnect_admin_queue; - ctrl->ctrl.max_hw_sectors = - (ctrl->lport->ops->max_sgl_segments -1 ) << (PAGE_SHIFT -9 ); + ctrl->ctrl.max_segments = ctrl->lport->ops->max_sgl_segments; + ctrl->ctrl.max_hw_sectors = ctrl->ctrl.max_segments << + (ilog2 (SZ_4K ) -9 );blk_mq_unquiesce_queue (ctrl->ctrl.admin_q);
Signature · 6 lines
Himanshu Madhani Oracle Linux Engineering _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
AppliedApplied to nvme-5.9, thanks.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme