RDMA/rxe: Increase value of RXE_MAX_UCONTEXT
Subsystems (2)
PATCH 1/3 RDMA/rxe: Increase value of RXE_MAX_UCONTEXT
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_param.h +1 -1
@@ -76,7 +76,7 @@ enum rxe_device_param {RXE_MAX_PKEYS =1 ,RXE_LOCAL_CA_ACK_DELAY =15 , -RXE_MAX_UCONTEXT =512 , +RXE_MAX_UCONTEXT =17408 , RXE_NUM_PORT = 1,
Signature · 1 line
2.31.1
PATCH 2/3 RDMA/rxe: Increase value of RXE_MAX_SRQ
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_param.h +1 -1
@@ -67,7 +67,7 @@ enum rxe_device_param {RXE_MAX_MCAST_QP_ATTACH =56 ,RXE_MAX_TOT_MCAST_QP_ATTACH =0x70000 ,RXE_MAX_AH =100 , -RXE_MAX_SRQ =960 , +RXE_MAX_SRQ =17408 ,RXE_MAX_SRQ_WR =0x4000 ,RXE_MIN_SRQ_WR =1 , RXE_MAX_SRQ_SGE = 27,
Signature · 1 line
2.31.1
Jason Gunthorpe
maintainerRe: [PATCH v1 2/3] RDMA/rxe: Increase value of RXE_MAX_SRQ
Quoted 11 lines · Rao Shoaib
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
Signed-off-by Rao Shoaib <[email protected]>--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Well, this doesn't apply anymore so please resend the series. I'd prefer you try to get rid of any hard limits that are not exposed in uverbs someplace - like ucontext.
Jason
Re: [PATCH v1 2/3] RDMA/rxe: Increase value of RXE_MAX_SRQ
Quoted 12 lines · Jason Gunthorpe
Quoted 11 lines · Rao Shoaib
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
Signed-off-by Rao Shoaib <[email protected]>--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 5 lines
Well, this doesn't apply anymore so please resend the series. I'd prefer you try to get rid of any hard limits that are not exposed in uverbs someplace - like ucontext.
Jason
Jason,
Most of the limits are exposed via uverbs. For example, following limits are most likely to be hit by a user application.
RXE_MAX_QPRXE_MAX_QP_WRRXE_MAX_CQRXE_MAX_MRRXE_MAX_PDRXE_MAX_AHRXE_MAX_FMRRXE_MAX_SRQRXE_MAX_SRQ_WRRXE_MAX_PKEYSRXE_MAX_UCONTEXT
However, they are reported in ib_device_attr/ib_uverbs_query_device and these are the limits that I am interested in. So unless I misunderstand you the current patch still applies? Kindly elaborate.
Regards,
Shoaib
PATCH 3/3 RDMA/rxe: Increase value of RXE_MAX_AH
In reply to Rao ShoaibFrom: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/sw/rxe/rxe_param.h +1 -1
@@ -66,7 +66,7 @@ enum rxe_device_param {RXE_MAX_MCAST_GRP =8192 ,RXE_MAX_MCAST_QP_ATTACH =56 ,RXE_MAX_TOT_MCAST_QP_ATTACH =0x70000 , -RXE_MAX_AH =100 , +RXE_MAX_AH =64000 ,RXE_MAX_SRQ =17408 ,RXE_MAX_SRQ_WR =0x4000 , RXE_MIN_SRQ_WR = 1,
Signature · 1 line
2.31.1
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 18 lines · Rao Shoaib
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
Signed-off-by Rao Shoaib <[email protected]>--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/infiniband/sw/rxe/rxe_param.h +0
-0
drivers/infiniband/sw/rxe/rxe_param.h enum rxe_device_param { · line 72
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_param.h +1
-1@@ -66,7 +66,7 @@ enum rxe_device_param {RXE_MAX_MCAST_GRP =8192 ,RXE_MAX_MCAST_QP_ATTACH =56 ,RXE_MAX_TOT_MCAST_QP_ATTACH =0x70000 , -RXE_MAX_AH =100 , +RXE_MAX_AH =64000 ,RXE_MAX_SRQ =17408 ,RXE_MAX_SRQ_WR =0x4000 , RXE_MIN_SRQ_WR = 1,
Interesting. There is no real reason to pick most of these values since it's just memory and does not reflect underlying hardware resources. It is advantageous to also be able to set them smaller to verify whether test cases correctly limit resources. It seems that there should be a way (module parameter or other) to adjust these values without having to recompile the driver. Thoughts?
Regards,
Bob Pearson
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 13 lines · Bob Pearson
Quoted 12 lines · Rao Shoaib
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
Signed-off-by Rao Shoaib <[email protected]>--- drivers/infiniband/sw/rxe/rxe_param.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_param.h +0
-0drivers/infiniband/sw/rxe/rxe_param.h enum rxe_device_param { · line 72
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_param.h +1
-1@@ -66,7 +66,7 @@ enum rxe_device_param {RXE_MAX_MCAST_GRP =8192 ,RXE_MAX_MCAST_QP_ATTACH =56 ,RXE_MAX_TOT_MCAST_QP_ATTACH =0x70000 , -RXE_MAX_AH =100 , +RXE_MAX_AH =64000 ,RXE_MAX_SRQ =17408 ,RXE_MAX_SRQ_WR =0x4000 , RXE_MIN_SRQ_WR = 1,Interesting. There is no real reason to pick most of these values since it's just memory and does not reflect underlying hardware resources. It is advantageous to also be able to set them smaller to verify whether test cases correctly limit resources. It seems that there should be a way (module parameter or other) to adjust these values without having to recompile the driver. Thoughts?
I agree with you 100% but it seems like the original design did not intent to make them configurable at run time. I see that recently some other values were bumped up by others so I went with hard coded values. In the inhouse kernel version we used to decide on the values had them configurable because we did not know what values we wanted, and recompiling was time consuming.
While some tests may require small values we want to test with lots of users/connections.
If there is consensus I can submit a different patch to make these values configurable. In the meantime I hope we can bump up these values.
Kindly let me know.
Shoaib
Quoted 4 lines
Regards,
Bob Pearson
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 33 lines · Shoaib Rao
Quoted 13 lines · Bob Pearson
Quoted 12 lines · Rao Shoaib
From: Rao Shoaib <[email protected]>
In our internal testing we have found that the current limit is too small, this patch bumps it up to a higher value required for our tests, which are indicative of our customer usage.
Signed-off-by Rao Shoaib <[email protected]>---
drivers/infiniband/sw/rxe/rxe_param.h| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)Quoted 15 lines
Quoted 6 lines
drivers/infiniband/sw/rxe/rxe_param.h +0
-0drivers/infiniband/sw/rxe/rxe_param.h enum rxe_device_param { · line 72
Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_param.h +1
-1@@ -66,7 +66,7 @@ enum rxe_device_param { RXE_MAX_MCAST_GRP = 8192, RXE_MAX_MCAST_QP_ATTACH = 56, RXE_MAX_TOT_MCAST_QP_ATTACH = 0x70000, - RXE_MAX_AH = 100, + RXE_MAX_AH = 64000, RXE_MAX_SRQ = 17408, RXE_MAX_SRQ_WR = 0x4000, RXE_MIN_SRQ_WR = 1,Interesting. There is no real reason to pick most of these values since it's just memory and does not reflect underlying hardware resources. It is advantageous to also be able to set them smaller to verify whether test cases correctly limit resources. It seems that there should be a way (module parameter or other) to adjust these values without having to recompile the driver. Thoughts?
I agree with you 100% but it seems like the original design did not intent to make them configurable at run time. I see that recently some other values were bumped up by others so I went with hard coded values. In the inhouse kernel version we used to decide on the values had them configurable because we did not know what values we wanted, and recompiling was time consuming.
Quoted 12 lines
While some tests may require small values we want to test with lots of users/connections.
If there is consensus I can submit a different patch to make these values configurable. In the meantime I hope we can bump up these values.
Kindly let me know.
Shoaib
Quoted 4 lines
Regards,
Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Jason
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 8 lines · Jason Gunthorpe
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Jason
Not sure where you are headed here. Do you mean just lift the limits all together? That would not be in the spirit of the IBA even if that is not very meaningful. One of the useful things rxe does is to provide a cheap 'reference' implementation of the spec. If OTOH you want to put it in production then the limits are not very useful.
Bob
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 11 lines · Bob Pearson
Quoted 8 lines · Jason Gunthorpe
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Jason
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
Jason
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 7 lines · Jason Gunthorpe
Quoted 6 lines · Bob Pearson
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Jason
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
Jason
So do you want me to submit a patch that removes all limits or just these limits or something else?
Shoaib
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
In reply to Jason GunthorpeQuoted 5 lines · Jason Gunthorpe
Quoted 4 lines · Bob Pearson
Quoted 3 lines · Jason Gunthorpe
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Can we just delete the concept completely?
Jason
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
These limits are reported via uverbs, so what do we report without current applications. Creating pool also requires limits but I guess we can use something like -1 to indicate there is no limit. I would have to look at all the values to see if we can implement this.
Shoaib
Quoted 2 lines
Jason
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 11 lines · Shoaib Rao
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Bob Pearson
Quoted 7 lines · Jason Gunthorpe
On Thu, Jun 17, 2021 at 10:56:58PM -0500, Bob Pearson wrote:
Quoted 1 line
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Jason
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
These limits are reported via uverbs, so what do we report without current applications. Creating pool also requires limits but I guess we can use something like -1 to indicate there is no limit. I would have to look at all the values to see if we can implement this.
Shoaib
Quoted 2 lines
Jason
The object create in pools (rxe_alloc_locked) just calls kzalloc for objects allocated by rxe and checks the limits. For objects allocated by rdma-core (__rxe_add_to_pool) it just checks the limits. The only place where the limit really matters is when a pool is indexed (RXE_POOL_INDEXED). Then there is a bitmask used to allocate the indices for the objects which consumes one byte for each 8 objects.
From Jason's comment I think just setting the limits fairly large but not excessive is the right approach.
Bob
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 25 lines · Bob Pearson
Quoted 11 lines · Shoaib Rao
Quoted 9 lines · Jason Gunthorpe
Quoted 8 lines · Bob Pearson
Quoted 7 lines · Jason Gunthorpe
On Thu, Jun 17, 2021 at 10:56:58PM -0500, Bob Pearson wrote:
Quoted 1 line
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Can we just delete the concept completely?
Jason
Quoted 12 lines
Quoted 4 lines
Quoted 2 lines
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
These limits are reported via uverbs, so what do we report without current applications. Creating pool also requires limits but I guess we can use something like -1 to indicate there is no limit. I would have to look at all the values to see if we can implement this.
Shoaib
Quoted 2 lines
Jason
Quoted 7 lines
The object create in pools (
rxe_alloc_locked) just calls kzalloc for objects allocated by rxe and checks the limits. For objects allocated by rdma-core (__rxe_add_to_pool) it just checks the limits. The only place where the limit really matters is when a pool is indexed (RXE_POOL_INDEXED). Then there is a bitmask used to allocate the indices for the objects which consumes one byte for each 8 objects.
Use an ida or xarray instead?
Jason
Re: [PATCH v1 3/3] RDMA/rxe: Increase value of RXE_MAX_AH
Quoted 29 lines · Jason Gunthorpe
Quoted 20 lines · Bob Pearson
Quoted 19 lines · Shoaib Rao
Quoted 5 lines · Jason Gunthorpe
Quoted 4 lines · Bob Pearson
Quoted 3 lines · Jason Gunthorpe
Quoted 1 line · Bob Pearson
It isn't my call. But I am in favor of tunable parameters. -- Bob Pearson
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Can we just delete the concept completely?
Jason
Not sure where you are headed here. Do you mean just lift the limits all together?
Yes.. The spec doesn't have like a UCONTEXT limit for instance, and real HW like mlx5 has huge reported limits anyhow.
These limits are reported via uverbs, so what do we report without current applications. Creating pool also requires limits but I guess we can use something like -1 to indicate there is no limit. I would have to look at all the values to see if we can implement this.
Shoaib
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Jason
The object create in pools (
rxe_alloc_locked) just calls kzalloc for objects allocated by rxe and checks the limits. For objects allocated by rdma-core (__rxe_add_to_pool) it just checks the limits. The only place where the limit really matters is when a pool is indexed (RXE_POOL_INDEXED). Then there is a bitmask used to allocate the indices for the objects which consumes one byte for each 8 objects.
Quoted 3 lines
Use an ida or xarray instead?
Jason
We do not have to change any data structures, just decide on the max limit.
RXE_POOL_INDEX flag as shown below is used to indicate what type of index to use and is not an issue
if (rxe_type_info[type].flags & RXE_POOL_INDEX) { err = rxe_pool_init_index(pool, rxe_type_info[type].max_index, rxe_type_info[type].min_index);
if (err)
goto out;
}
And
static int rxe_pool_init_index(struct rxe_pool *pool, u32 max, u32 min) {
So the maximum index can be U32_MAX or for now we can limit is to (1 << 20) i.e 1048576
I am assuming that changing the limits for the objects I listed earlier (list below) is fine.
Shoaib
RXE_MAX_QPRXE_MAX_QP_WRRXE_MAX_CQRXE_MAX_MRRXE_MAX_PDRXE_MAX_AHRXE_MAX_FMRRXE_MAX_SRQRXE_MAX_SRQ_WRRXE_MAX_PKEYSRXE_MAX_UCONTEXT