Small set of mlx5 related fixes
Subsystems (5)
cover Small set of mlx5 related fixes
From: Leon Romanovsky <[email protected]>
Hi,
Despite one change in mlx5_core, all those changes are for mlx5_ib. The change in mlx5_core is related to MR cache initialization that is not used in mlx5_core at all.
Thanks
Artemy Kovalyov (3):
IB/mlx5: WQE dump jumps over first 16 bytes net/mlx5: Decrease default mr cache size IB/mlx5: Compare only index part of a memory window rkey
Moni Shoua (1):
IB/mlx5: Reset access mask when looping inside page fault handler
drivers/infiniband/hw/mlx5/odp.c | 14 +++++++++++-- drivers/infiniband/hw/mlx5/qp.c | 5 ++--- .../net/ethernet/mellanox/mlx5/core/main.c | 20 ------------------- 3 files changed, 14 insertions(+), 25 deletions(-)
PATCH 1/4 IB/mlx5: Reset access mask when looping inside page fault handler
From: Moni Shoua <[email protected]>
If page-fault handler spans multiple MRs then the access mask needs to be reset before each MR handling or otherwise write access will be granted to mapped pages instead of read-only.
--- drivers/infiniband/hw/mlx5/odp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/odp.c +2 -1
@@ -585,7 +585,7 @@ static int pagefault_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr,struct ib_umem_odp *odp_mr =to_ib_umem_odp (mr->umem);bool downgrade = flags &MLX5_PF_FLAGS_DOWNGRADE ;bool prefetch = flags &MLX5_PF_FLAGS_PREFETCH ; -u64 access_mask =ODP_READ_ALLOWED_BIT ; +u64 access_mask;u64 start_idx, page_mask;struct ib_umem_odp *odp;size_t size;
@@ -607,6 +607,7 @@ static int pagefault_mr(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr, page_shift = mr->umem->page_shift; page_mask = ~(BIT (page_shift) -1 ); start_idx = (io_virt - (mr->mmkey.iova & page_mask)) >> page_shift; + access_mask =ODP_READ_ALLOWED_BIT ;if (prefetch && !downgrade && !mr->umem->writable) {/* prefetch with write-access must
Signature · 1 line
2.20.1
PATCH 2/4 IB/mlx5: WQE dump jumps over first 16 bytes
In reply to Leon RomanovskyFrom: Artemy Kovalyov <[email protected]>
Move index increment after its is used or otherwise it will start the dump of the WQE from second WQE BB.
--- drivers/infiniband/hw/mlx5/qp.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/qp.c +2 -3
@@ -4724,16 +4724,15 @@ static void set_linv_wr(struct mlx5_ib_qp *qp, void **seg, int *size,static void dump_wqe (struct mlx5_ib_qp *qp,u32 idx,int size_16) {__be32 *p =NULL ; -u32 tidx = idx;int i, j;pr_debug ("dump WQE index %u:\n" , idx);for (i =0 , j =0 ; i < size_16 *4 ; i +=4 , j +=4 ) {if ((i &0xf ) ==0 ) { - tidx = (tidx +1 ) & (qp->sq.wqe_cnt -1 ); - p =mlx5_frag_buf_get_wqe (&qp->sq.fbc, tidx); + p =mlx5_frag_buf_get_wqe (&qp->sq.fbc, idx);pr_debug ("WQBB at %p:\n" , (void *)p); j =0 ; + idx = (idx +1 ) & (qp->sq.wqe_cnt -1 ); } pr_debug("%08x %08x %08x %08x\n", be32_to_cpu(p[j]), be32_to_cpu(p[j + 1]), be32_to_cpu(p[j + 2]),
Signature · 1 line
2.20.1
PATCH 3/4 net/mlx5: Decrease default mr cache size
In reply to Leon RomanovskyFrom: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 20 ------------------- 1 file changed, 20 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +0 -20
@@ -164,26 +164,6 @@ static struct mlx5_profile profile[] = { .size = 8, .limit =4 }, - .mr_cache[16] = { - .size = 8, - .limit =4 - }, - .mr_cache[17] = { - .size = 8, - .limit =4 - }, - .mr_cache[18] = { - .size = 8, - .limit =4 - }, - .mr_cache[19] = { - .size = 4, - .limit =2 - }, - .mr_cache[20] = { - .size = 4, - .limit =2 - }, }, };
Signature · 1 line
2.20.1
Re: [PATCH mlx5-next 3/4] net/mlx5: Decrease default mr cache size
Quoted 5 lines · Leon Romanovsky
From: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
Please add here:
This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB
Quoted 3 lines
Signed-off-by Artemy Kovalyov <[email protected]>Signed-off-by Moni Shoua <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>
Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that.
Quoted 38 lines
--- .../net/ethernet/mellanox/mlx5/core/main.c | 20 ------------------- 1 file changed, 20 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/main.c +0
-21@@ -164,26 +164,6 @@ static struct mlx5_profile profile[] = { .size = 8, .limit =4 }, - .mr_cache[16] = { - .size = 8, - .limit =4 - }, - .mr_cache[17] = { - .size = 8, - .limit =4 - }, - .mr_cache[18] = { - .size = 8, - .limit =4 - }, - .mr_cache[19] = { - .size = 4, - .limit =2 - }, - .mr_cache[20] = { - .size = 4, - .limit =2 - }, }, }; --2.20.1
Re: [PATCH mlx5-next 3/4] net/mlx5: Decrease default mr cache size
Quoted 17 lines · Or Gerlitz
Quoted 5 lines · Leon Romanovsky
From: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
Please add here:
This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB
Quoted 3 lines
Signed-off-by Artemy Kovalyov <[email protected]>Signed-off-by Moni Shoua <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>
Quoted 5 lines
Reported-by Shalom Toledo <[email protected]>Acked-by Or Gerlitz <[email protected]>Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that.
Thanks,
I'll add it at the "apply" stage.
Re: [PATCH mlx5-next 3/4] net/mlx5: Decrease default mr cache size
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Or Gerlitz
Quoted 5 lines · Leon Romanovsky
From: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
Please add here:
This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB
Quoted 3 lines
Signed-off-by Artemy Kovalyov <[email protected]>Signed-off-by Moni Shoua <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>
Quoted 7 lines
Quoted 5 lines
Reported-by Shalom Toledo <[email protected]>Acked-by Or Gerlitz <[email protected]>Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that.
I'll add it at the "apply" stage.
Just to make sure, by "add it" you mean the signatures, the text and the Fixes that?
Also, is there any reason not to eliminate the mr cache pre-population all together?
Currently we consume 100-200MB per function after the patch which is also problematic for some
environments.
Re: [PATCH mlx5-next 3/4] net/mlx5: Decrease default mr cache size
Quoted 22 lines · Or Gerlitz
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Or Gerlitz
Quoted 5 lines · Leon Romanovsky
From: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
Please add here:
This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB
Quoted 3 lines
Signed-off-by Artemy Kovalyov <[email protected]>Signed-off-by Moni Shoua <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>Reported-by Shalom Toledo <[email protected]>Acked-by Or Gerlitz <[email protected]>
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that.
I'll add it at the "apply" stage.
Just to make sure, by "add it" you mean the signatures, the text and the Fixes that?
Yes, of course.
Quoted 3 lines
Also, is there any reason not to eliminate the mr cache pre-population all together?
AFAIK, pre-populated cache helps MPI application to start faster and with out-of-box experience. Nobody seems to care enough to challenge this internal assumption.
Quoted 4 lines
Currently we consume 100-200MB per function after the patch which is also problematic for some
environments.
Re: [PATCH mlx5-next 3/4] net/mlx5: Decrease default mr cache size
Quoted 10 lines · Leon Romanovsky
Quoted 9 lines · Or Gerlitz
Quoted 8 lines · Leon Romanovsky
Quoted 7 lines · Or Gerlitz
Quoted 5 lines · Leon Romanovsky
From: Artemy Kovalyov <[email protected]>
Delete initialization of high order entries in mr cache to decrease initial memory footprint. When required, the administrator can populate the entries with memory keys via the /sys interface.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Please add here:
This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB
here it needs to be "per physical function" and not "per function"
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that.
I'll add it at the "apply" stage.
Just to make sure, by "add it" you mean the signatures, the text and the Fixes that?
Yes, of course.
thanks
Quoted 6 lines
Quoted 2 lines
Also, is there any reason not to eliminate the mr cache pre-population all together?
AFAIK, pre-populated cache helps MPI application to start faster and with out-of-box experience. Nobody seems to care enough to challenge this internal assumption.
I see, we are cutting here ~0.8GB so lets just do it and wonder on the remaining 0.1GB later..
It would be good to push it into 5.1-rc so the fallout into stables will start right away and
not only by 5.2 time - can we do that?
PATCH 4/4 IB/mlx5: Compare only index part of a memory window rkey
In reply to Leon RomanovskyFrom: Artemy Kovalyov <[email protected]>
The InfiniBand Architecture Specification section 10.6.7.2.4 TYPE 2 MEMORY WINDOWS says that if the CI supports the Base Memory Management Extensions defined in this specification, the R_Key format for a Type 2 Memory Window must consist of:
* 24 bit index in the most significant bits of the R_Key, which is owned by the CI, and
* 8 bit key in the least significant bits of the R_Key, which is owned by the Consumer.
This means that the kernel should compare only the index part of a R_Key to determine equality with another R_Key.
--- drivers/infiniband/hw/mlx5/odp.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/odp.c +10 -1
@@ -711,6 +711,15 @@ struct pf_frame {int depth; }; +static bool mkey_is_eq (struct mlx5_core_mkey *mmkey,u32 key) +{ +if (!mmkey) +return false; +if (mmkey->type ==MLX5_MKEY_MW ) +return mlx5_base_mkey (mmkey->key) ==mlx5_base_mkey (key); +return mmkey->key == key; +} +static int get_indirect_num_descs (struct mlx5_core_mkey *mmkey) {struct mlx5_ib_mw *mw;
@@ -760,7 +769,7 @@ static int pagefault_single_data_segment(struct mlx5_ib_dev *dev, next_mr: mmkey =__mlx5_mr_lookup (dev->mdev,mlx5_base_mkey (key)); -if (!mmkey || mmkey->key != key) { +if (!mkey_is_eq (mmkey, key)) {mlx5_ib_dbg (dev,"failed to find mkey %x\n" , key); ret = -EFAULT ; goto srcu_unlock;
Signature · 1 line
2.20.1
Jason Gunthorpe
AppliedQuoted 8 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
Despite one change in
mlx5_core, all those changes are formlx5_ib. The change inmlx5_coreis related to MR cache initialization that is not used inmlx5_coreat all.
Quoted 5 lines
Thanks
Artemy Kovalyov (3):
IB/mlx5: WQE dump jumps over first 16 bytes IB/mlx5: Compare only index part of a memory window rkey
Applied to for-next
Quoted 1 line
net/mlx5: Decrease default mr cache size
For you and Saeed
Quoted 2 lines
Moni Shoua (1):
IB/mlx5: Reset access mask when looping inside page fault handler
I sent this one to for-rc since it is @stable
Thanks,
Jason