reduce iSERT Max IO size
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Thanks,
Krishnam Raju.
Quoted 9 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Max, remind me what was the point to support 16M? Did this resolve an issue?
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).
Thanks,
Krishnam Raju.
Quoted 12 lines · Sagi Grimberg
Quoted 9 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Sagi Grimberg
Tested-by Krishnamraju EraparajuQuoted 5 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).
I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO size
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
--
Quoted 17 lines
Thanks,
Krishnam Raju.Quoted 12 lines · Sagi Grimberg
Quoted 9 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 10 lines · Sagi Grimberg
Quoted 5 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?
Quoted 6 lines
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO size
I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.
Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.
So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.
Thanks,
Krishnam Raju.
Quoted 30 lines
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 17 lines
Thanks,
Krishnam Raju.Quoted 12 lines · Sagi Grimberg
Quoted 9 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 11 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
But, the max pages per MR supported by most adapters is around 256 pages only.
Not sure you're right.
Quoted 2 lines
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
Quoted 46 lines
Quoted 6 lines
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn't cause the reported issue.Thanks,
Krishnam Raju.Quoted 26 lines
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 14 lines
Thanks,
Krishnam Raju.Quoted 10 lines · Sagi Grimberg
Quoted 8 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 5 lines · Krishnamraju Eraparaju
Quoted 4 lines · Sagi Grimberg
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).
Quoted 8 lines
Quoted 4 lines
I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
Quoted 46 lines
Quoted 6 lines
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn't cause the reported issue.Thanks,
Krishnam Raju.Quoted 26 lines
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 14 lines
Thanks,
Krishnam Raju.Quoted 10 lines · Sagi Grimberg
Quoted 8 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 11 lines · Max Gurtovoy
Quoted 9 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
Quoted 7 lines
Quoted 4 lines
But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.
For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
Quoted 47 lines
Quoted 47 lines
Quoted 7 lines
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn't cause the reported issue.Thanks,
Krishnam Raju.Quoted 26 lines
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 14 lines
Thanks,
Krishnam Raju.Quoted 10 lines · Sagi Grimberg
Quoted 8 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 22 lines · Krishnamraju Eraparaju
Quoted 10 lines · Max Gurtovoy
Quoted 9 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.
Quoted 6 lines
Quoted 4 lines
But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
Quoted 8 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.
Sagi,
what about adding a module parameter for this as we did in iSER initiator ?
Quoted 44 lines
Quoted 44 lines
Quoted 44 lines
Quoted 5 lines
If we revert that it would restore the issue that you reported in the first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn't cause the reported issue.Thanks,
Krishnam Raju.Quoted 25 lines
Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 13 lines
Thanks,
Krishnam Raju.Quoted 10 lines · Sagi Grimberg
Quoted 8 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.Max, remind me what was the point to support 16M? Did this resolve an issue?
Quoted 15 lines
Quoted 6 lines
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:Quoted 8 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
Quoted 5 lines
For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.
Sagi,
what about adding a module parameter for this as we did in iSER initiator ?
I don't think we have any other choice...
Quoted 15 lines · Sagi Grimberg
Quoted 15 lines
Quoted 14 lines
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
Quoted 8 lines
Quoted 6 lines
For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.
Sagi,
what about adding a module parameter for this as we did in iSER initiator ?
I don't think we have any other choice...
Sagi,
I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?
Thanks
-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----
Quoted 39 lines
To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO sizeQuoted 3 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju
On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst
case
Quoted 1 line
Quoted 1 line
Quoted 1 line
was in case there are different capabilities on the initiator and
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
target with respect to number of pages per MR. There is no
handshake
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
that aligns expectations.
But, the max pages per MR supported by most adapters is around 256
pages
Quoted 2 lines
Quoted 2 lines
only.
And I think only those iSER initiators, whose max pages per MR is4096,
Quoted 4 lines
Quoted 1 line
could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
Quoted 5 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.
For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?
Quoted 67 lines
For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
If we revert that it would restore the issue that you reported in
the
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly
with
Quoted 4 lines
Quoted 4 lines
"
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single
target
Quoted 3 lines
Quoted 3 lines
and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn'tcause the
Quoted 5 lines
Quoted 5 lines
reported issue.
Thanks,
Krishnam Raju.Quoted 1 line
Current iSER target code allocates MR pool budget based on queue
size.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Since there is no handshake between iSER initiator and target on
max IO
Quoted 1 line
Quoted 1 line
Quoted 1 line
size, we'll set the iSER target to support upto 16MiB IO
operations and
Quoted 1 line
Quoted 1 line
Quoted 1 line
allocate the correct number of RDMA ctxs according to the factor
of MR's
Quoted 1 line
Quoted 1 line
Quoted 1 line
per IO operation. This would guaranty sufficient size of the MR
pool for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 4 lines
Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, SagiQuoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER
Target.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The PBL memory consumption has increased significantly after
increasing
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On
one variant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Max, remind me what was the point to support 16M? Did this
resolve
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an issue?
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 6 lines
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:Quoted 8 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.
Sagi,
what about adding a module parameter for this as we did in iSER initiator ?
Quoted 7 lines
Quoted 1 line
I don't think we have any other choice...
Sagi,
I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?
Its already done this way. The problem is that there is no handshake procedure in iSER between the host and the target for what the maximum transfer size would be, so currently isert sets up to a worse case of 16MB. This has implications on memory requirements.
In the absence of a handshake we should have the user choose with a param... Ideally that would be a configfs parameter, but a modparam can work as well here.
Quoted 47 lines · Bernard Metzler
-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----
Quoted 39 lines
To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO sizeQuoted 3 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju
On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst
case
Quoted 1 line
Quoted 1 line
Quoted 1 line
was in case there are different capabilities on the initiator and
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
target with respect to number of pages per MR. There is no
handshake
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
that aligns expectations.
But, the max pages per MR supported by most adapters is around 256
pages
Quoted 2 lines
Quoted 2 lines
only.
And I think only those iSER initiators, whose max pages per MR is4096,
Quoted 4 lines
Quoted 1 line
could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:Quoted 5 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.
Quoted 3 lines
For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?
If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.
Quoted 70 lines
Quoted 69 lines
For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
If we revert that it would restore the issue that you reported in
the
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly
with
Quoted 4 lines
Quoted 4 lines
"
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single
target
Quoted 3 lines
Quoted 3 lines
and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn'tcause the
Quoted 5 lines
Quoted 5 lines
reported issue.
Thanks,
Krishnam Raju.Quoted 1 line
Current iSER target code allocates MR pool budget based on queue
size.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Since there is no handshake between iSER initiator and target on
max IO
Quoted 1 line
Quoted 1 line
Quoted 1 line
size, we'll set the iSER target to support upto 16MiB IO
operations and
Quoted 1 line
Quoted 1 line
Quoted 1 line
allocate the correct number of RDMA ctxs according to the factor
of MR's
Quoted 1 line
Quoted 1 line
Quoted 1 line
per IO operation. This would guaranty sufficient size of the MR
pool for
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 4 lines
Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, SagiQuoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER
Target.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The PBL memory consumption has increased significantly after
increasing
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On
one variant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Max, remind me what was the point to support 16M? Did this
resolve
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an issue?
Quoted 45 lines · Krishnamraju Eraparaju
Quoted 44 lines · Bernard Metzler
-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----
Quoted 42 lines
To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO sizeQuoted 2 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju
On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst
case
Quoted 1 line
Quoted 1 line
Quoted 1 line
was in case there are different capabilities on the initiator and
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
target with respect to number of pages per MR. There is no
handshake
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
that aligns expectations.
But, the max pages per MR supported by most adapters is around 256
pages
Quoted 2 lines
Quoted 2 lines
only.
And I think only those iSER initiators, whose max pages per MR is4096,
Quoted 3 lines
Quoted 1 line
could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.
Quoted 6 lines
Quoted 3 lines
For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?
If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.
We can set a default to 1MiB, and add module param that can increase this size to a max IO size of 16MiB.
I'll sent a patch early next week.
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?Quoted 1 line
Quoted 1 line
Quoted 1 line
If we revert that it would restore the issue that you reported in
the
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly
with
Quoted 4 lines
Quoted 4 lines
"
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single
target
Quoted 3 lines
Quoted 3 lines
and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn'tcause the
Quoted 5 lines
Quoted 5 lines
reported issue.
Thanks,
Krishnam Raju.Quoted 1 line
Current iSER target code allocates MR pool budget based on queue
size.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Since there is no handshake between iSER initiator and target on
max IO
Quoted 1 line
Quoted 1 line
Quoted 1 line
size, we'll set the iSER target to support upto 16MiB IO
operations and
Quoted 1 line
Quoted 1 line
Quoted 1 line
allocate the correct number of RDMA ctxs according to the factor
of MR's
Quoted 1 line
Quoted 1 line
Quoted 1 line
per IO operation. This would guaranty sufficient size of the MR
pool for
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 3 lines
Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, SagiQuoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER
Target.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The PBL memory consumption has increased significantly after
increasing
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On
one variant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Max, remind me what was the point to support 16M? Did this
resolve
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an issue?
Quoted 50 lines · Max Gurtovoy
Quoted 48 lines · Krishnamraju Eraparaju
Quoted 47 lines · Bernard Metzler
-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----
Quoted 37 lines
To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO sizeQuoted 2 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju
On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg
Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Hi Sagi & Max,
Any update on this?
Please change the max IO size to 1MiB(256 pages).I think that the reason why this was changed to handle the worst
case
Quoted 1 line
Quoted 1 line
Quoted 1 line
was in case there are different capabilities on the initiator and
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
target with respect to number of pages per MR. There is no
handshake
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
that aligns expectations.
But, the max pages per MR supported by most adapters is around 256
pages
Quoted 2 lines
Quoted 2 lines
only.
And I think only those iSER initiators, whose max pages per MR is4096,
Quoted 3 lines
Quoted 1 line
could send 16MiB sized IOs, am I correct?
If the initiator can send 16MiB, we must make sure the target is capable to receive it.
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:Quoted 5 lines
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?
Quoted 8 lines
Quoted 3 lines
If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.
We can set a default to 1MiB, and add module param that can increase this size to a max IO size of 16MiB.
I'll sent a patch early next week.
Thanks Max!
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?Quoted 1 line
Quoted 1 line
Quoted 1 line
If we revert that it would restore the issue that you reported in
the
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
first place:
--
IB/isert: allocate RW ctxs according to max IO sizeI don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly
with
Quoted 4 lines
Quoted 4 lines
"
rdma_rw_mr_factor()" related changes, I think.Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single
target
Quoted 3 lines
Quoted 3 lines
and muli-luns case.
So reverting only
ISCSI_ISER_MAX_SG_TABLESIZEmacro to 256 doesn'tcause the
Quoted 5 lines
Quoted 5 lines
reported issue.
Thanks,
Krishnam Raju.Quoted 1 line
Current iSER target code allocates MR pool budget based on queue
size.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Since there is no handshake between iSER initiator and target on
max IO
Quoted 1 line
Quoted 1 line
Quoted 1 line
size, we'll set the iSER target to support upto 16MiB IO
operations and
Quoted 1 line
Quoted 1 line
Quoted 1 line
allocate the correct number of RDMA ctxs according to the factor
of MR's
Quoted 1 line
Quoted 1 line
Quoted 1 line
per IO operation. This would guaranty sufficient size of the MR
pool for
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
the required IO queue depth and IO size.
Reported-by Krishnamraju Eraparaju <[email protected]>Signed-off-by Max Gurtovoy <[email protected]>--
Quoted 3 lines
Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, SagiQuoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Hi,
Please reduce the Max IO size to 1MiB(256 pages), at iSER
Target.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
The PBL memory consumption has increased significantly after
increasing
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On
one variant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.
Max, remind me what was the point to support 16M? Did this
resolve
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
an issue?
Quoted 26 lines · Sagi Grimberg
Quoted 26 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
I think max IO size, at iSER initiator, depends on "
max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:
iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?
Quoted 8 lines
Quoted 6 lines
For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.
Sagi,
what about adding a module parameter for this as we did in iSER initiator ?
I don't think we have any other choice...
Sagi,
Quoted 8 lines
Quoted 3 lines
I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?
Its already done this way. The problem is that there is no handshake procedure in iSER between the host and the target for what the maximum transfer size would be, so currently isert sets up to a worse case of 16MB. This has implications on memory requirements.
Thanks for the explanation.