First round of mlx5 fixes for 4.9
Subsystems (2)
cover First round of mlx5 fixes for 4.9
Hi Doug,
Please find below first round of fixes and improvements to ConnectX-4/Connect-IB device driver.
This patchset was generated against v4.9-rc1.
Available in the "topic/mlx5-fixes-4.9" topic branch of this git repo:git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git
Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/mlx5-fixes-4.9
Thanks
Daniel Jurgens (1):
IB/mlx5: Use cache line size to select CQE stride
Eli Cohen (4):
IB/mlx5: Wait for all async command completions to complete IB/mlx5: Fix fatal error dispatching
IB/mlx5: Fix reported max SGE calculation IB/mlx5: Fix NULL pointer dereference on debug print
Majd Dibbiny (2):
IB/mlx5: Fix memory leak in query device
IB/mlx5: Limit mkey page size to 2GB
Maor Gottlieb (3):
IB/mlx5: Fix atomic cap in indirect UMR
IB/mlx5: Put non zero value in max_ah
IB/mlx5: Validate requested RQT size
Max Gurtovoy (1):
IB/mlx5: Replace numerical constant with predefined MACRO
Moshe Lazer (1):
IB/mlx5: Resolve soft lock on massive reg MRs
drivers/infiniband/hw/mlx5/cq.c | 7 ++--- drivers/infiniband/hw/mlx5/main.c | 20 +++++++----- drivers/infiniband/hw/mlx5/mem.c | 7 ++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 8 ++++- drivers/infiniband/hw/mlx5/mr.c | 37 ++++++++++++++++++++-- drivers/infiniband/hw/mlx5/qp.c | 59 +++++++++++++++++++++++++++++------- drivers/infiniband/hw/mlx5/srq.c | 2 +- 7 files changed, 113 insertions(+), 27 deletions(-)
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1477575407-20562-1-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH 1/12 IB/mlx5: Replace numerical constant with predefined MACRO
From: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Replace the pre-defined macro signifying inline umr instead of the numerical constant.
--- drivers/infiniband/hw/mlx5/qp.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/qp.c +1 -1
@@ -3068,7 +3068,7 @@ static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr) {memset (umr,0 ,sizeof (*umr)); umr->mkey_mask =cpu_to_be64 (MLX5_MKEY_MASK_FREE ); - umr->flags =1 <<7 ; + umr->flags =MLX5_UMR_INLINE ; }static __be64 get_umr_reg_mr_mask (void )
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1477575407-20562-2-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH rdma-rc 01/12] IB/mlx5: Replace numerical constant with predefined MACRO
Quoted 4 lines · Leon Romanovsky
From: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Replace the pre-defined macro signifying inline umr instead of the numerical constant.
Leon,
By all means (or no means, choose), this is a fix. If you are telling vmware and Broadcom people here for months how to write their code, let's stop for a minute and look in the mirror, drop this patch.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJ3xEMiNynmpbBt0M=k5qywrvnfAq+ogZLD2PUbHBvYVJ4pLyQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH rdma-rc 01/12] IB/mlx5: Replace numerical constant with predefined MACRO
Quoted 11 lines · Or Gerlitz
Quoted 4 lines · Leon Romanovsky
From: Max Gurtovoy <maxg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Replace the pre-defined macro signifying inline umr instead of the numerical constant.
Leon,
By all means (or no means, choose), this is a fix. If you are telling vmware and Broadcom people here for months how to write their code, let's stop for a minute and look in the mirror, drop this patch.
Thanks Or,
I appreciate your time invested in the review and providing so much valuable feedback.
Since change of numeric value to the same value defined by macro is not a fix but improvement and provided here to simplify maintainer's work (Doug), I have no plans to change the patch or/and drop it.
Quoted 6 lines
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20161028145534.GM3617-2ukJVAZIZ/[email protected]
Re: [PATCH rdma-rc 01/12] IB/mlx5: Replace numerical constant with predefined MACRO
Quoted 6 lines · Leon Romanovsky
Quoted 1 line · Or Gerlitz
By all means (or no means, choose), this is a fix. If you are telling
Since change of numeric value to the same value defined by macro is not a fix but improvement [..] I have no plans to change the patch or/and drop it.
RU listening? it's a -next patch, not rc
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 2/12 IB/mlx5: Fix atomic cap in indirect UMR
In reply to a message not in the archiveFrom: Maor Gottlieb <maorg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Remove from the driver the limitation imposed by firmware check to not allow change of atomic permissions for indirect UMRs. In order to avoid failures on old firmware, we only ask for change of atomic permissions if atomic operations are supported.
--- drivers/infiniband/hw/mlx5/qp.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx5/qp.c +7 -5
@@ -3071,7 +3071,7 @@ static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr) umr->flags =MLX5_UMR_INLINE ; } -static __be64 get_umr_reg_mr_mask (void ) +static __be64 get_umr_reg_mr_mask (int atomic) {u64 result;
@@ -3084,9 +3084,11 @@ static __be64 get_umr_reg_mr_mask(void)MLX5_MKEY_MASK_KEY |MLX5_MKEY_MASK_RR |MLX5_MKEY_MASK_RW | -MLX5_MKEY_MASK_A |MLX5_MKEY_MASK_FREE ; +if (atomic) + result |=MLX5_MKEY_MASK_A ; +return cpu_to_be64 (result); }
@@ -3147,7 +3149,7 @@ static __be64 get_umr_update_pd_mask(void) }static void set_reg_umr_segment (struct mlx5_wqe_umr_ctrl_seg *umr, -struct ib_send_wr *wr) +struct ib_send_wr *wr,int atomic) {struct mlx5_umr_wr *umrwr =umr_wr (wr);
@@ -3172,7 +3174,7 @@ static void set_reg_umr_segment(struct mlx5_wqe_umr_ctrl_seg *umr,if (wr->send_flags &MLX5_IB_SEND_UMR_UPDATE_PD ) umr->mkey_mask |=get_umr_update_pd_mask ();if (!umr->mkey_mask) - umr->mkey_mask =get_umr_reg_mr_mask (); + umr->mkey_mask =get_umr_reg_mr_mask (atomic); }else { umr->mkey_mask =get_umr_unreg_mr_mask (); }
@@ -4025,7 +4027,7 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, } qp->sq.wr_data[idx] =MLX5_IB_WR_UMR ; ctrl->imm =cpu_to_be32 (umr_wr (wr)->mkey); -set_reg_umr_segment (seg, wr); +set_reg_umr_segment (seg, wr, !!(MLX5_CAP_GEN (mdev, atomic))); seg +=sizeof (struct mlx5_wqe_umr_ctrl_seg ); size +=sizeof (struct mlx5_wqe_umr_ctrl_seg ) /16 ;if (unlikely ((seg == qend)))
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/12 IB/mlx5: Put non zero value in max_ah
In reply to a message not in the archiveFrom: Maor Gottlieb <maorg-VPRAkNaXOzVWk0Htik3J/[email protected]>
We put INT_MAX since this is the max value that can be held. Though there is no hardware limitation, this is practically a large enough number so we can use it.
--- drivers/infiniband/hw/mlx5/main.c | 1 + 1 file changed, 1 insertion(+)
drivers/infiniband/hw/mlx5/main.c +1 -0
@@ -643,6 +643,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, props->max_total_mcast_qp_attach = props->max_mcast_qp_attach * props->max_mcast_grp; props->max_map_per_fmr =INT_MAX ;/* no limit in ConnectIB */ + props->max_ah =INT_MAX ; props->hca_core_clock =MLX5_CAP_GEN (mdev, device_frequency_khz); props->timestamp_mask =0x7FFFFFFFFFFFFFFFULL ;
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 4/12 IB/mlx5: Fix memory leak in query device
In reply to a message not in the archiveFrom: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/[email protected]>
We need to free dev->port when we fail to enable RoCE or initialize node data.
--- drivers/infiniband/hw/mlx5/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/main.c +2 -2
@@ -3116,7 +3116,7 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) } err =init_node_data (dev);if (err) - goto err_dealloc; + goto err_free_port;mutex_init (&dev->flow_db.lock);mutex_init (&dev->cap_mask_mutex);
@@ -3126,7 +3126,7 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev)if (ll ==IB_LINK_LAYER_ETHERNET ) { err =mlx5_enable_roce (dev);if (err) - goto err_dealloc; + goto err_free_port; } err =create_dev_resources (&dev->devr);
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 5/12 IB/mlx5: Validate requested RQT size
In reply to a message not in the archiveFrom: Maor Gottlieb <maorg-VPRAkNaXOzVWk0Htik3J/[email protected]>
Validate that the requested size of RQT is supported by firmware.
--- drivers/infiniband/hw/mlx5/qp.c | 8 ++++++++ 1 file changed, 8 insertions(+)
drivers/infiniband/hw/mlx5/qp.c +8 -0
@@ -4817,6 +4817,14 @@ struct ib_rwq_ind_table *mlx5_ib_create_rwq_ind_table(struct ib_device *device, udata->inlen))return ERR_PTR (-EOPNOTSUPP ); +if (init_attr->log_ind_tbl_size > +MLX5_CAP_GEN (dev->mdev, log_max_rqt_size)) { +mlx5_ib_dbg (dev,"log_ind_tbl_size = %d is bigger than supported = %d\n" , + init_attr->log_ind_tbl_size, +MLX5_CAP_GEN (dev->mdev, log_max_rqt_size)); +return ERR_PTR (-EINVAL ); + } + min_resp_len = offsetof(typeof (resp ), reserved) +sizeof (resp.reserved);if (udata->outlen && udata->outlen < min_resp_len)return ERR_PTR (-EINVAL );
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 6/12 IB/mlx5: Use cache line size to select CQE stride
In reply to a message not in the archiveFrom: Daniel Jurgens <danielj-VPRAkNaXOzVWk0Htik3J/[email protected]>
When creating kernel CQs use 128B CQE stride if the cache line size is 128B, 64B otherwise. This prevents multiple CQEs from residing in a 128B cache line, which can cause retries when there are concurrent read and writes in one cache line.
Tested with IPoIB on PPC64, saw ~5% throughput
improvement.
--- drivers/infiniband/hw/mlx5/cq.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +1 -2
@@ -932,8 +932,7 @@ struct ib_cq *mlx5_ib_create_cq(struct ib_device *ibdev,if (err) goto err_create; }else { -/* for now choose 64 bytes till we have a proper interface */ - cqe_size =64 ; + cqe_size =cache_line_size () ==128 ?128 :64 ; err =create_cq_kernel (dev, cq, entries, cqe_size, &cqb, &index, &inlen);if (err)
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 7/12 IB/mlx5: Resolve soft lock on massive reg MRs
In reply to a message not in the archiveFrom: Moshe Lazer <moshel-VPRAkNaXOzVWk0Htik3J/[email protected]>
When calling reg_mr of large MRs (e.g. 4GB) from multiple processes and MR caches can't supply the required amount of MRs the slow-path of MR allocation may be used. In this case we need to serialize the slow-path between the processes to avoid soft lock.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 ++ drivers/infiniband/hw/mlx5/mr.c | 6 +++++- 2 files changed, 7 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
@@ -626,6 +626,8 @@ struct mlx5_ib_dev {struct mlx5_ib_resources devr;struct mlx5_mr_cache cache;struct timer_list delay_timer; +/* Prevents soft lock on massive reg MRs */ +struct mutex slow_path_mutex;int fill_delay;#ifdef CONFIG_INFINIBAND_ON_DEMAND_PAGING struct ib_odp_caps odp_caps;
drivers/infiniband/hw/mlx5/mr.c +5 -1
@@ -610,6 +610,7 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev)int err;int i; +mutex_init (&dev->slow_path_mutex); cache->wq =alloc_ordered_workqueue ("mkey_cache" ,WQ_MEM_RECLAIM );if (!cache->wq) {mlx5_ib_warn (dev,"failed to create work queue\n" );
@@ -1182,9 +1183,12 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, goto error; } -if (!mr) +if (!mr) { +mutex_lock (&dev->slow_path_mutex); mr =reg_create (NULL , pd, virt_addr, length, umem, ncont, page_shift, access_flags); +mutex_unlock (&dev->slow_path_mutex); + }if (IS_ERR (mr)) { err =PTR_ERR (mr);
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 8/12 IB/mlx5: Wait for all async command completions to complete
In reply to a message not in the archiveFrom: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]>
Wait before continuing unload till all pending mkey async creation requests are done.
--- drivers/infiniband/hw/mlx5/mr.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
drivers/infiniband/hw/mlx5/mr.c +28 -0
@@ -646,6 +646,33 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev)return 0 ; } +static void wait_for_async_commands (struct mlx5_ib_dev *dev) +{ +struct mlx5_mr_cache *cache = &dev->cache; +struct mlx5_cache_ent *ent; +int total =0 ; +int i; +int j; + +for (i =0 ; i <MAX_MR_CACHE_ENTRIES ; i++) { + ent = &cache->ent[i]; +for (j =0 ; j <1000 ; j++) { +if (!ent->pending) +break ; +msleep (50 ); + } + } +for (i =0 ; i <MAX_MR_CACHE_ENTRIES ; i++) { + ent = &cache->ent[i]; + total += ent->pending; + } + +if (total) +mlx5_ib_warn (dev,"aborted while there are %d pending mr requests\n" , total); +else +mlx5_ib_warn (dev,"done with all pending requests\n" ); +} +int mlx5_mr_cache_cleanup (struct mlx5_ib_dev *dev) {int i;
@@ -659,6 +686,7 @@ int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev)clean_keys (dev, i);destroy_workqueue (dev->cache.wq); +wait_for_async_commands (dev);del_timer_sync (&dev->delay_timer);return 0 ;
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1477575407-20562-9-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH rdma-rc 08/12] IB/mlx5: Wait for all async command completions to complete
Quoted 5 lines · Leon Romanovsky
drivers/infiniband/hw/mlx5/mr.c +5
-0+static void wait_for_async_commands (struct mlx5_ib_dev *dev) +{ +struct mlx5_mr_cache *cache = &dev->cache; +struct mlx5_cache_ent *ent; +int total =0 ;
drivers/infiniband/hw/mlx5/mr.c int mlx5_mr_cache_init(struct mlx5_ib_dev *dev) · line 663
Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10
-0+int i; +int j; + +for (i =0 ; i <MAX_MR_CACHE_ENTRIES ; i++) { + ent = &cache->ent[i]; +for (j =0 ; j <1000 ; j++) { +if (!ent->pending) +break ; +msleep (50 ); + }
you had another patch on this series which change a hard coded constant into a define, why this patch add two new hard coded constants, so all to all, we're not making progress on that no-hard-coded-constants front... better decide where you want to go --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 9/12 IB/mlx5: Fix fatal error dispatching
In reply to a message not in the archiveFrom: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]>
When an internal error condition is detected, make sure to set the device inactive after dispatching the event so ULPs can get a notification of this event.
--- drivers/infiniband/hw/mlx5/main.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/main.c +5 -2
@@ -2312,14 +2312,14 @@ static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context, {struct mlx5_ib_dev *ibdev = (struct mlx5_ib_dev *)context;struct ib_event ibev; - +bool fatal = false;u8 port =0 ;switch (event) {case MLX5_DEV_EVENT_SYS_ERROR : - ibdev->ib_active = false; ibev.event =IB_EVENT_DEVICE_FATAL ;mlx5_ib_handle_internal_error (ibdev); + fatal = true;break ;case MLX5_DEV_EVENT_PORT_UP :
@@ -2371,6 +2371,9 @@ static void mlx5_ib_event(struct mlx5_core_dev *dev, void *context,if (ibdev->ib_active)ib_dispatch_event (&ibev); + +if (fatal) + ibdev->ib_active = false; }static void get_ext_port_caps (struct mlx5_ib_dev *dev)
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 10/12 IB/mlx5: Fix reported max SGE calculation
In reply to a message not in the archiveFrom: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add the 512 bytes limit of RDMA READ and the size of remote address to the max SGE calculation.
--- drivers/infiniband/hw/mlx5/main.c | 8 +++++--- drivers/infiniband/hw/mlx5/qp.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 33 insertions(+), 4 deletions(-)
drivers/infiniband/hw/mlx5/main.c +5 -3
@@ -496,6 +496,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,struct mlx5_ib_dev *dev =to_mdev (ibdev);struct mlx5_core_dev *mdev = dev->mdev;int err = -ENOMEM ; +int max_sq_desc;int max_rq_sg;int max_sq_sg;u64 min_page_size =1ull <<MLX5_CAP_GEN (mdev, log_pg_sz);
@@ -618,9 +619,10 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, props->max_qp_wr =1 <<MLX5_CAP_GEN (mdev, log_max_qp_sz); max_rq_sg =MLX5_CAP_GEN (mdev, max_wqe_sz_rq) /sizeof (struct mlx5_wqe_data_seg ); - max_sq_sg = (MLX5_CAP_GEN (mdev, max_wqe_sz_sq) - -sizeof (struct mlx5_wqe_ctrl_seg )) / -sizeof (struct mlx5_wqe_data_seg ); + max_sq_desc =min_t (int,MLX5_CAP_GEN (mdev, max_wqe_sz_sq),512 ); + max_sq_sg = (max_sq_desc -sizeof (struct mlx5_wqe_ctrl_seg ) - +sizeof (struct mlx5_wqe_raddr_seg )) / +sizeof (struct mlx5_wqe_data_seg ); props->max_sge =min (max_rq_sg, max_sq_sg); props->max_sge_rd =MLX5_MAX_SGE_RD ; props->max_cq =1 <<MLX5_CAP_GEN (mdev, log_max_cq);
drivers/infiniband/hw/mlx5/qp.c +28 -1
@@ -352,6 +352,29 @@ static int calc_send_wqe(struct ib_qp_init_attr *attr)return ALIGN (max_t (int, inl_size, size),MLX5_SEND_WQE_BB ); } +static int get_send_sge (struct ib_qp_init_attr *attr,int wqe_size) +{ +int max_sge; + +if (attr->qp_type ==IB_QPT_RC ) + max_sge = (min_t (int, wqe_size,512 ) - +sizeof (struct mlx5_wqe_ctrl_seg ) - +sizeof (struct mlx5_wqe_raddr_seg )) / +sizeof (struct mlx5_wqe_data_seg ); +else if (attr->qp_type ==IB_QPT_XRC_INI ) + max_sge = (min_t (int, wqe_size,512 ) - +sizeof (struct mlx5_wqe_ctrl_seg ) - +sizeof (struct mlx5_wqe_xrc_seg ) - +sizeof (struct mlx5_wqe_raddr_seg )) / +sizeof (struct mlx5_wqe_data_seg ); +else + max_sge = (wqe_size -sq_overhead (attr)) / +sizeof (struct mlx5_wqe_data_seg ); + +return min_t (int, max_sge, wqe_size -sq_overhead (attr) / +sizeof (struct mlx5_wqe_data_seg )); +} +static int calc_sq_size (struct mlx5_ib_dev *dev,struct ib_qp_init_attr *attr,struct mlx5_ib_qp *qp) {
@@ -388,7 +411,11 @@ static int calc_sq_size(struct mlx5_ib_dev *dev, struct ib_qp_init_attr *attr,return -ENOMEM ; } qp->sq.wqe_shift =ilog2 (MLX5_SEND_WQE_BB ); - qp->sq.max_gs = attr->cap.max_send_sge; + qp->sq.max_gs =get_send_sge (attr, wqe_size); +if (qp->sq.max_gs < attr->cap.max_send_sge) +return -ENOMEM ; + + attr->cap.max_send_sge = qp->sq.max_gs; qp->sq.max_post = wq_size / wqe_size; attr->cap.max_send_wr = qp->sq.max_post;
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1477575407-20562-11-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH rdma-rc 10/12] IB/mlx5: Fix reported max SGE calculation
Quoted 6 lines · Leon Romanovsky
From: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]>
Add the 512 bytes limit of RDMA READ and the size of remote address to the max SGE calculation.
Fixes e126ba97dba9 ('mlx5: Add driver for Mellanox Connect-IB adapters')
Doug, for some reason the above line is different in the commit present @ your git hub tree, bug somewhere?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 11/12 IB/mlx5: Fix NULL pointer dereference on debug print
In reply to a message not in the archiveFrom: Eli Cohen <eli-VPRAkNaXOzVWk0Htik3J/[email protected]>
For XRC QP CQs may not exist. Check before attempting dereference.
--- drivers/infiniband/hw/mlx5/qp.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/qp.c +2 -2
@@ -2079,8 +2079,8 @@ struct ib_qp *mlx5_ib_create_qp(struct ib_pd *pd,mlx5_ib_dbg (dev,"ib qpnum 0x%x, mlx qpn 0x%x, rcqn 0x%x, scqn 0x%x\n" , qp->ibqp.qp_num, qp->trans_qp.base.mqp.qpn, -to_mcq (init_attr->recv_cq)->mcq.cqn, -to_mcq (init_attr->send_cq)->mcq.cqn); + init_attr->recv_cq ?to_mcq (init_attr->recv_cq)->mcq.cqn :-1 , + init_attr->send_cq ?to_mcq (init_attr->send_cq)->mcq.cqn :-1 ); qp->trans_qp.xrcdn = xrcdn;
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 12/12 IB/mlx5: Limit mkey page size to 2GB
In reply to a message not in the archiveFrom: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/[email protected]>
The maximum page size in the mkey context is 2GB.
Until today, we didn't enforce this requirement in the code, and therefore, if we got a page size larger than 2GB, we have passed zeros in the log_page_shift instead of the actual value and the registration failed.
This patch limits the driver to use compound pages of 2GB for mkeys.
--- drivers/infiniband/hw/mlx5/cq.c | 4 ++-- drivers/infiniband/hw/mlx5/mem.c | 7 ++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 6 +++++- drivers/infiniband/hw/mlx5/mr.c | 3 ++- drivers/infiniband/hw/mlx5/qp.c | 4 ++-- drivers/infiniband/hw/mlx5/srq.c | 2 +- 6 files changed, 18 insertions(+), 8 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +2 -2
@@ -770,7 +770,7 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata,if (err) goto err_umem; -mlx5_ib_cont_pages (cq->buf.umem, ucmd.buf_addr, &npages, &page_shift, +mlx5_ib_cont_pages (cq->buf.umem, ucmd.buf_addr,0 , &npages, &page_shift, &ncont,NULL );mlx5_ib_dbg (dev,"addr 0x%llx, size %u, npages %d, page_shift %d, ncont %d\n" , ucmd.buf_addr, entries * ucmd.cqe_size, npages, page_shift, ncont);
@@ -1124,7 +1124,7 @@ static int resize_user(struct mlx5_ib_dev *dev, struct mlx5_ib_cq *cq,return err; } -mlx5_ib_cont_pages (umem, ucmd.buf_addr, &npages, page_shift, +mlx5_ib_cont_pages (umem, ucmd.buf_addr,0 , &npages, page_shift, npas,NULL ); cq->resize_umem = umem;
drivers/infiniband/hw/mlx5/mem.c +6 -1
@@ -37,12 +37,15 @@/* @umem: umem object to scan * @addr: ib virtual address requested by the user +* @max_page_shift: high limit for page_shift - 0 means no limit * @count: number of PAGE_SIZE pages covered by umem * @shift: page shift for the compound pages found in the region * @ncont: number of compund pages * @order: log2 of the number of compound pages */ -void mlx5_ib_cont_pages (struct ib_umem *umem,u64 addr,int *count,int *shift, +void mlx5_ib_cont_pages (struct ib_umem *umem,u64 addr, +unsigned long max_page_shift, +int *count,int *shift,int *ncont,int *order) {unsigned long tmp;
@@ -72,6 +75,8 @@ void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr, int *count, int *shift, addr = addr >> page_shift; tmp = (unsigned long )addr; m =find_first_bit (&tmp,BITS_PER_LONG ); +if (max_page_shift) + m = min_t(unsigned long, max_page_shift - page_shift, m); skip =1 << m; mask = skip -1 ; i =0 ;
drivers/infiniband/hw/mlx5/mlx5_ib.h +5 -1
@@ -63,6 +63,8 @@ pr_warn("%s:%s:%d:(pid %d): " format, (dev)->ib_dev.name, __func__, \#define MLX5_IB_DEFAULT_UIDX 0xffffff#define MLX5_USER_ASSIGNED_UIDX_MASK __mlx5_mask(qpc, user_index) +#define MLX5_MKEY_PAGE_SHIFT_MASK __mlx5_mask(mkc, log_page_size) +enum {MLX5_IB_MMAP_CMD_SHIFT =8 ,MLX5_IB_MMAP_CMD_MASK =0xff ,
@@ -825,7 +827,9 @@ int mlx5_ib_query_port(struct ib_device *ibdev, u8 port, struct ib_port_attr *props);int mlx5_ib_init_fmr (struct mlx5_ib_dev *dev);void mlx5_ib_cleanup_fmr (struct mlx5_ib_dev *dev); -void mlx5_ib_cont_pages (struct ib_umem *umem,u64 addr,int *count,int *shift, +void mlx5_ib_cont_pages (struct ib_umem *umem,u64 addr, +unsigned long max_page_shift, +int *count,int *shift,int *ncont,int *order); void __mlx5_ib_populate_pas(struct mlx5_ib_dev *dev, struct ib_umem *umem, int page_shift, size_t offset, size_t num_pages,
drivers/infiniband/hw/mlx5/mr.c +2 -1
@@ -856,7 +856,8 @@ static struct ib_umem *mr_umem_get(struct ib_pd *pd, u64 start, u64 length,return (void *)umem; } -mlx5_ib_cont_pages (umem, start, npages, page_shift, ncont, order); +mlx5_ib_cont_pages (umem, start,MLX5_MKEY_PAGE_SHIFT_MASK , npages, + page_shift, ncont, order);if (!*npages) {mlx5_ib_warn (dev,"avoid zero region\n" );ib_umem_release (umem);
drivers/infiniband/hw/mlx5/qp.c +2 -2
@@ -675,7 +675,7 @@ static int mlx5_ib_umem_get(struct mlx5_ib_dev *dev,return PTR_ERR (*umem); } -mlx5_ib_cont_pages (*umem, addr, npages, page_shift, ncont,NULL ); +mlx5_ib_cont_pages (*umem, addr,0 , npages, page_shift, ncont,NULL ); err =mlx5_ib_get_buf_offset (addr, *page_shift, offset);if (err) {
@@ -728,7 +728,7 @@ static int create_user_rq(struct mlx5_ib_dev *dev, struct ib_pd *pd,return err; } -mlx5_ib_cont_pages (rwq->umem, ucmd->buf_addr, &npages, &page_shift, +mlx5_ib_cont_pages (rwq->umem, ucmd->buf_addr,0 , &npages, &page_shift, &ncont,NULL ); err =mlx5_ib_get_buf_offset (ucmd->buf_addr, page_shift, &rwq->rq_page_offset);
drivers/infiniband/hw/mlx5/srq.c +1 -1
@@ -118,7 +118,7 @@ static int create_srq_user(struct ib_pd *pd, struct mlx5_ib_srq *srq,return err; } -mlx5_ib_cont_pages (srq->umem, ucmd.buf_addr, &npages, +mlx5_ib_cont_pages (srq->umem, ucmd.buf_addr,0 , &npages, &page_shift, &ncont,NULL ); err =mlx5_ib_get_buf_offset (ucmd.buf_addr, page_shift, &offset);
Signature · 6 lines
2.7.4 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1477575407-20562-13-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH rdma-rc 12/12] IB/mlx5: Limit mkey page size to 2GB
Quoted 6 lines · Leon Romanovsky
From: Majd Dibbiny <majd-VPRAkNaXOzVWk0Htik3J/[email protected]>
The maximum page size in the mkey context is 2GB.
Until today, we didn't enforce this requirement in the code, and therefore, if we got a page size larger than 2GB, we
got a page size larger than 2GB? who are those pages?
Quoted 2 lines
have passed zeros in the
log_page_shiftinstead of the actual value and the registration failed.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 31841A29-5CE1-45A9-999A-1FAAA3CDFD4F-VPRAkNaXOzVWk0Htik3J/[email protected]
Re: [PATCH rdma-rc 12/12] IB/mlx5: Limit mkey page size to 2GB
Quoted 4 lines · Majd Dibbiny
On Oct 28, 2016, at 4:02 PM, Or Gerlitz <[email protected]> wrote:
In case we are working with 1GB hugepage, it is possible to find more than two continuous pages, and end up with more than 2GB..
got it
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH rdma-rc 12/12] IB/mlx5: Limit mkey page size to 2GB
In reply to a message not in the archiveQuoted 1 line
Fixes e126ba97dba9 ('mlx5: Add driver for Mellanox Connect-IB adapters')
the above line is different @ your git hub copy, bug as well? --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html