Skip to contentSkip to conversation

reduce iSERT Max IO size

DISCUSSION Krishnamraju Eraparaju 5 participants 17 messages Sep 22, 2020 – Oct 9, 2020 InfiniBand

lore
Share & tools
lore
#0 lore raw

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Thanks,
Krishnam Raju.

#1 lore raw
Quoted 9 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#2 lore raw

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

Thanks,
Krishnam Raju.

Quoted 12 lines · Sagi Grimberg
Quoted 9 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

Tested-by Krishnamraju Eraparaju
#3 lore raw
Quoted 5 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 17 lines

Thanks,
Krishnam Raju.

Quoted 12 lines · Sagi Grimberg
Quoted 9 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#4 lore raw
Quoted 10 lines · Sagi Grimberg
Quoted 5 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?

Quoted 6 lines

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.

Thanks,
Krishnam Raju.

Quoted 30 lines

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 17 lines

Thanks,
Krishnam Raju.

Quoted 12 lines · Sagi Grimberg
Quoted 9 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#5 lore raw
Quoted 11 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

But, the max pages per MR supported by most adapters is around 256 pages only.

Not sure you're right.

Quoted 2 lines

And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

Quoted 46 lines
Quoted 6 lines

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.

Thanks,
Krishnam Raju.

Quoted 26 lines

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 14 lines

Thanks,
Krishnam Raju.

Quoted 10 lines · Sagi Grimberg
Quoted 8 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#6 lore raw
In reply to Krishnamraju Eraparaju
Quoted 5 lines · Krishnamraju Eraparaju
Quoted 4 lines · Sagi Grimberg
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

Quoted 8 lines
Quoted 4 lines

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

Quoted 46 lines
Quoted 6 lines

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.

Thanks,
Krishnam Raju.

Quoted 26 lines

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 14 lines

Thanks,
Krishnam Raju.

Quoted 10 lines · Sagi Grimberg
Quoted 8 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#7 lore raw
Quoted 11 lines · Max Gurtovoy
Quoted 9 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

Quoted 7 lines
Quoted 4 lines

But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

Quoted 47 lines
Quoted 47 lines
Quoted 7 lines

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.

Thanks,
Krishnam Raju.

Quoted 26 lines

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 14 lines

Thanks,
Krishnam Raju.

Quoted 10 lines · Sagi Grimberg
Quoted 8 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#8 lore raw
Quoted 22 lines · Krishnamraju Eraparaju
Quoted 10 lines · Max Gurtovoy
Quoted 9 lines · Krishnamraju Eraparaju
Quoted 8 lines · Sagi Grimberg
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst case was in case there are different capabilities on the initiator and the target with respect to number of pages per MR. There is no handshake that aligns expectations.

Quoted 6 lines
Quoted 4 lines

But, the max pages per MR supported by most adapters is around 256 pages only.
And I think only those iSER initiators, whose max pages per MR is 4096, could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 8 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.

Sagi,

what about adding a module parameter for this as we did in iSER initiator ?

Quoted 44 lines
Quoted 44 lines
Quoted 44 lines
Quoted 5 lines

If we revert that it would restore the issue that you reported in the first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly with "rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single target and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't cause the reported issue.

Thanks,
Krishnam Raju.

Quoted 25 lines

Current iSER target code allocates MR pool budget based on queue size. Since there is no handshake between iSER initiator and target on max IO size, we'll set the iSER target to support upto 16MiB IO operations and allocate the correct number of RDMA ctxs according to the factor of MR's per IO operation. This would guaranty sufficient size of the MR pool for the required IO queue depth and IO size.

--

Quoted 13 lines

Thanks,
Krishnam Raju.

Quoted 10 lines · Sagi Grimberg
Quoted 8 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER Target. The PBL memory consumption has increased significantly after increasing the Max IO size to 16MiB(with commit:317000b926b07c).
Due to the large MR pool, the max no.of iSER connections(On one variant of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this resolve an issue?

#9 lore raw
Quoted 15 lines
Quoted 6 lines

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 8 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

Quoted 5 lines

For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.

Sagi,

what about adding a module parameter for this as we did in iSER initiator ?

I don't think we have any other choice...

#10 lore raw
Quoted 15 lines · Sagi Grimberg
Quoted 15 lines
Quoted 14 lines

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

Quoted 8 lines
Quoted 6 lines

For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.

Sagi,

what about adding a module parameter for this as we did in iSER initiator ?

I don't think we have any other choice...

Sagi,

I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?

Thanks

#11 lore raw
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 6 lines

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 8 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.

Sagi,

what about adding a module parameter for this as we did in iSER initiator ?

Quoted 7 lines
Quoted 1 line

I don't think we have any other choice...

Sagi,

I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?

Its already done this way. The problem is that there is no handshake procedure in iSER between the host and the target for what the maximum transfer size would be, so currently isert sets up to a worse case of 16MB. This has implications on memory requirements.

In the absence of a handshake we should have the user choose with a param... Ideally that would be a configfs parameter, but a modparam can work as well here.

#12 lore raw
Quoted 26 lines · Sagi Grimberg
Quoted 26 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For iwarp case, if 512 is the max pages supported by all iwarp drivers, then provisioning a gigantic MR pool at target(to accommodate never used 16MiB IO) wouldn't be a overkill?

Quoted 8 lines
Quoted 6 lines

For RoCE/IB Mellanox HCAs we support 16MiB IO size and even more. We limited to 16MiB in iSER/iSERT.

Sagi,

what about adding a module parameter for this as we did in iSER initiator ?

I don't think we have any other choice...

Sagi,

Quoted 8 lines
Quoted 3 lines

I didn't read whole thread and know little about ULPs, but wonder if isn't it possible to check device type (iWARP/RoCE) during iSERT initialization and create MR pool only after device is recognized?

Its already done this way. The problem is that there is no handshake procedure in iSER between the host and the target for what the maximum transfer size would be, so currently isert sets up to a worse case of 16MB. This has implications on memory requirements.

Thanks for the explanation.

#13 lore raw
In reply to Krishnamraju Eraparaju

-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----

Quoted 39 lines

To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO size

Quoted 3 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju

On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst

case

Quoted 1 line
Quoted 1 line
Quoted 1 line

was in case there are different capabilities on the initiator and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

target with respect to number of pages per MR. There is no

handshake

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

that aligns expectations.

But, the max pages per MR supported by most adapters is around 256

pages

Quoted 2 lines
Quoted 2 lines

only.
And I think only those iSER initiators, whose max pages per MR is

4096,

Quoted 4 lines
Quoted 1 line

could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 5 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?

Quoted 67 lines

For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If we revert that it would restore the issue that you reported in

the

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly

with

Quoted 4 lines
Quoted 4 lines

"rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single

target

Quoted 3 lines
Quoted 3 lines

and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't

cause the

Quoted 5 lines
Quoted 5 lines

reported issue.

Thanks,
Krishnam Raju.

Quoted 1 line

Current iSER target code allocates MR pool budget based on queue

size.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Since there is no handshake between iSER initiator and target on

max IO

Quoted 1 line
Quoted 1 line
Quoted 1 line

size, we'll set the iSER target to support upto 16MiB IO

operations and

Quoted 1 line
Quoted 1 line
Quoted 1 line

allocate the correct number of RDMA ctxs according to the factor

of MR's

Quoted 1 line
Quoted 1 line
Quoted 1 line

per IO operation. This would guaranty sufficient size of the MR

pool for

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

the required IO queue depth and IO size.

--

Quoted 4 lines

Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, Sagi

Quoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER

Target.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The PBL memory consumption has increased significantly after

increasing

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On

one variant

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this

resolve

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

an issue?

#14 lore raw
Quoted 47 lines · Bernard Metzler

-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----

Quoted 39 lines

To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO size

Quoted 3 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju

On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst

case

Quoted 1 line
Quoted 1 line
Quoted 1 line

was in case there are different capabilities on the initiator and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

target with respect to number of pages per MR. There is no

handshake

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

that aligns expectations.

But, the max pages per MR supported by most adapters is around 256

pages

Quoted 2 lines
Quoted 2 lines

only.
And I think only those iSER initiators, whose max pages per MR is

4096,

Quoted 4 lines
Quoted 1 line

could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 5 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

Quoted 3 lines

For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?

If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.

Quoted 70 lines
Quoted 69 lines

For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

If we revert that it would restore the issue that you reported in

the

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly

with

Quoted 4 lines
Quoted 4 lines

"rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single

target

Quoted 3 lines
Quoted 3 lines

and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't

cause the

Quoted 5 lines
Quoted 5 lines

reported issue.

Thanks,
Krishnam Raju.

Quoted 1 line

Current iSER target code allocates MR pool budget based on queue

size.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Since there is no handshake between iSER initiator and target on

max IO

Quoted 1 line
Quoted 1 line
Quoted 1 line

size, we'll set the iSER target to support upto 16MiB IO

operations and

Quoted 1 line
Quoted 1 line
Quoted 1 line

allocate the correct number of RDMA ctxs according to the factor

of MR's

Quoted 1 line
Quoted 1 line
Quoted 1 line

per IO operation. This would guaranty sufficient size of the MR

pool for

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

the required IO queue depth and IO size.

--

Quoted 4 lines

Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, Sagi

Quoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER

Target.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The PBL memory consumption has increased significantly after

increasing

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On

one variant

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this

resolve

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

an issue?

#15 lore raw
Quoted 45 lines · Krishnamraju Eraparaju
Quoted 44 lines · Bernard Metzler

-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----

Quoted 42 lines

To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO size

Quoted 2 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju

On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst

case

Quoted 1 line
Quoted 1 line
Quoted 1 line

was in case there are different capabilities on the initiator and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

target with respect to number of pages per MR. There is no

handshake

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

that aligns expectations.

But, the max pages per MR supported by most adapters is around 256

pages

Quoted 2 lines
Quoted 2 lines

only.
And I think only those iSER initiators, whose max pages per MR is

4096,

Quoted 3 lines
Quoted 1 line

could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

Quoted 6 lines
Quoted 3 lines

For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?

If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.

We can set a default to 1MiB, and add module param that can increase this size to a max IO size of 16MiB.

I'll sent a patch early next week.

Quoted 64 lines
Quoted 64 lines
Quoted 64 lines

For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?

Quoted 1 line
Quoted 1 line
Quoted 1 line

If we revert that it would restore the issue that you reported in

the

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly

with

Quoted 4 lines
Quoted 4 lines

"rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single

target

Quoted 3 lines
Quoted 3 lines

and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't

cause the

Quoted 5 lines
Quoted 5 lines

reported issue.

Thanks,
Krishnam Raju.

Quoted 1 line

Current iSER target code allocates MR pool budget based on queue

size.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Since there is no handshake between iSER initiator and target on

max IO

Quoted 1 line
Quoted 1 line
Quoted 1 line

size, we'll set the iSER target to support upto 16MiB IO

operations and

Quoted 1 line
Quoted 1 line
Quoted 1 line

allocate the correct number of RDMA ctxs according to the factor

of MR's

Quoted 1 line
Quoted 1 line
Quoted 1 line

per IO operation. This would guaranty sufficient size of the MR

pool for

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

the required IO queue depth and IO size.

--

Quoted 3 lines

Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, Sagi

Quoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER

Target.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The PBL memory consumption has increased significantly after

increasing

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On

one variant

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this

resolve

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

an issue?

#16 lore raw
Quoted 50 lines · Max Gurtovoy
Quoted 48 lines · Krishnamraju Eraparaju
Quoted 47 lines · Bernard Metzler

-----"Krishnamraju Eraparaju" <[email protected]> wrote: -----

Quoted 37 lines

To: "Max Gurtovoy" <[email protected]>
From: "Krishnamraju Eraparaju" <[email protected]>
Date: 10/07/2020 05:36AM
Cc: "Sagi Grimberg" <[email protected]>, [email protected], "Potnuri Bharat Teja" <[email protected]>, "Max Gurtovoy" <[email protected]>
Subject: [EXTERNAL] Re: reduce iSERT Max IO size

Quoted 2 lines · Max Gurtovoy
Quoted 1 line · Krishnamraju Eraparaju

On Friday, October 10/02/20, 2020 at 13:29:30 -0700, Sagi Grimberg

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Hi Sagi & Max,

Any update on this?
Please change the max IO size to 1MiB(256 pages).

I think that the reason why this was changed to handle the worst

case

Quoted 1 line
Quoted 1 line
Quoted 1 line

was in case there are different capabilities on the initiator and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

target with respect to number of pages per MR. There is no

handshake

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

that aligns expectations.

But, the max pages per MR supported by most adapters is around 256

pages

Quoted 2 lines
Quoted 2 lines

only.
And I think only those iSER initiators, whose max pages per MR is

4096,

Quoted 3 lines
Quoted 1 line

could send 16MiB sized IOs, am I correct?

If the initiator can send 16MiB, we must make sure the target is capable to receive it.

I think max IO size, at iSER initiator, depends on "max_fast_reg_page_list_len".
currently, below are the supported "max_fast_reg_page_list_len" of various iwarp drivers:

Quoted 5 lines

iw_cxgb4: 128 pages
Softiwarp: 256 pages
i40iw: 512 pages
qedr: couldn't find.

For siw, this limit is not determined by resource constraints. We could bump it up to 512, or higher. What is a reasonable maximum, from iSER view?

Quoted 8 lines
Quoted 3 lines

If the most common IO sizes are 4K & 8K, then the reasonable max IO size of 256 pages(1 MiB) would be appropriate, by default. currently, NVMet-rdma also limits max IO size to 1MiB.

We can set a default to 1MiB, and add module param that can increase this size to a max IO size of 16MiB.

I'll sent a patch early next week.

Thanks Max!

Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines

For iwarp case, if 512 is the max pages supported by all iwarp drivers,
then provisioning a gigantic MR pool at target(to accommodate never used
16MiB IO) wouldn't be a overkill?

Quoted 1 line
Quoted 1 line
Quoted 1 line

If we revert that it would restore the issue that you reported in

the

Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

first place:

--
IB/isert: allocate RW ctxs according to max IO size

I don't see the reported issue after reducing the IO size to 256 pages(keeping all other changes of this patch intact). That is, "attr.cap.max_rdma_ctxs" is now getting filled properly

with

Quoted 4 lines
Quoted 4 lines

"rdma_rw_mr_factor()" related changes, I think.

Before this change "attr.cap.max_rdma_ctxs" was hardcoded with 128(ISCSI_DEF_XMIT_CMDS_MAX) pages, which is very low for single

target

Quoted 3 lines
Quoted 3 lines

and muli-luns case.

So reverting only ISCSI_ISER_MAX_SG_TABLESIZE macro to 256 doesn't

cause the

Quoted 5 lines
Quoted 5 lines

reported issue.

Thanks,
Krishnam Raju.

Quoted 1 line

Current iSER target code allocates MR pool budget based on queue

size.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Since there is no handshake between iSER initiator and target on

max IO

Quoted 1 line
Quoted 1 line
Quoted 1 line

size, we'll set the iSER target to support upto 16MiB IO

operations and

Quoted 1 line
Quoted 1 line
Quoted 1 line

allocate the correct number of RDMA ctxs according to the factor

of MR's

Quoted 1 line
Quoted 1 line
Quoted 1 line

per IO operation. This would guaranty sufficient size of the MR

pool for

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

the required IO queue depth and IO size.

--

Quoted 3 lines

Thanks,
Krishnam Raju.
On Wednesday, September 09/23/20, 2020 at 01:57:47 -0700, Sagi

Quoted 3 lines · Grimberg
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Hi,

Please reduce the Max IO size to 1MiB(256 pages), at iSER

Target.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

The PBL memory consumption has increased significantly after

increasing

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the Max IO size to 16MiB(with commit:317000b926b07c). Due to the large MR pool, the max no.of iSER connections(On

one variant

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines

of Chelsio cards) came down to 9, before it was 250. NVMe-RDMA target also uses 1MiB max IO size.

Max, remind me what was the point to support 16M? Did this

resolve

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

an issue?