Introduce dynamic UAR allocation mode
Subsystems (5)
cover Introduce dynamic UAR allocation mode
From: Leon Romanovsky <[email protected]>
Changelog:
v1: * Added patch that moved mlx5_bfreg_info from global header to the mlx5_ib.h
* No other changes.
v0: * https://lore.kernel.org/linux-rdma/[email protected]
----------------------------------------------------------------------------------
From Yishai,
This series exposes API to enable a dynamic allocation and management of a UAR which now becomes to be a regular uobject.
Moving to that mode enables allocating a UAR only upon demand and drop the redundant static allocation of UARs upon context creation.
In addition, it allows master and secondary processes that own the same command FD to allocate and manage UARs according to their needs, this can’t be achieved today.
As part of this option, QP & CQ creation flows were adapted to support this dynamic UAR mode once asked by user space.
Once this mode is asked by mlx5 user space driver on a given context, it will be mutual exclusive, means both the static and legacy dynamic modes for using UARs will be blocked.
The legacy modes are supported for backward compatible reasons, looking forward we expect this new mode to be the default.
Thanks
Leon Romanovsky (1):
IB/mlx5: Limit the scope of struct mlx5_bfreg_info to mlx5_ib
Yishai Hadas (4):
IB/mlx5: Expose UAR object and its alloc/destroy commands IB/mlx5: Extend CQ creation to get uar page index from user space IB/mlx5: Extend QP creation to get uar page index from user space IB/mlx5: Move to fully dynamic UAR mode once user space supports it
drivers/infiniband/hw/mlx5/cq.c | 21 ++- drivers/infiniband/hw/mlx5/main.c | 185 ++++++++++++++++++++-- drivers/infiniband/hw/mlx5/mlx5_ib.h | 20 +++ drivers/infiniband/hw/mlx5/qp.c | 33 ++-- include/linux/mlx5/driver.h | 17 -- include/rdma/uverbs_ioctl.h | 2 +- include/uapi/rdma/mlx5-abi.h | 6 + include/uapi/rdma/mlx5_user_ioctl_cmds.h | 18 +++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 5 + 9 files changed, 263 insertions(+), 44 deletions(-)
--
2.24.1
PATCH 1/5 IB/mlx5: Expose UAR object and its alloc/destroy commands
From: Yishai Hadas <[email protected]>
Expose UAR object and its alloc/destroy commands to be used over the ioctl interface by user space applications.
This API supports both BF & NC modes and enables a dynamic allocation of UARs once really needed.
As the number of driver objects were limited by the core ones when the merged tree is prepared, had to decrease the number of core objects to enable the new UAR object usage.
--- drivers/infiniband/hw/mlx5/main.c | 172 ++++++++++++++++++++-- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + include/rdma/uverbs_ioctl.h | 2 +- include/uapi/rdma/mlx5_user_ioctl_cmds.h | 18 +++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 5 + 5 files changed, 189 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/main.c +163 -9
@@ -2022,6 +2022,17 @@ static phys_addr_t uar_index2pfn(struct mlx5_ib_dev *dev,return (dev->mdev->bar_addr >>PAGE_SHIFT ) + uar_idx / fw_uars_per_page; } +static u64 uar_index2paddress (struct mlx5_ib_dev *dev, +int uar_idx) +{ +unsigned int fw_uars_per_page; + + fw_uars_per_page =MLX5_CAP_GEN (dev->mdev, uar_4k) ? +MLX5_UARS_IN_PAGE :1 ; + +return (dev->mdev->bar_addr + (uar_idx / fw_uars_per_page) *PAGE_SIZE ); +} +static int get_command (unsigned long offset) {return (offset >>MLX5_IB_MMAP_CMD_SHIFT ) &MLX5_IB_MMAP_CMD_MASK ;
@@ -2106,6 +2117,11 @@ static void mlx5_ib_mmap_free(struct rdma_user_mmap_entry *entry)mutex_unlock (&var_table->bitmap_lock);kfree (mentry);break ; +case MLX5_IB_MMAP_TYPE_UAR_WC : +case MLX5_IB_MMAP_TYPE_UAR_NC : +mlx5_cmd_free_uar (dev->mdev, mentry->page_idx); +kfree (mentry); +break ;default :WARN_ON (true); }
@@ -2257,7 +2273,8 @@ static int mlx5_ib_mmap_offset(struct mlx5_ib_dev *dev, mentry =to_mmmap (entry); pfn = (mentry->address >>PAGE_SHIFT ); -if (mentry->mmap_flag ==MLX5_IB_MMAP_TYPE_VAR ) +if (mentry->mmap_flag ==MLX5_IB_MMAP_TYPE_VAR || + mentry->mmap_flag ==MLX5_IB_MMAP_TYPE_UAR_NC ) prot =pgprot_noncached (vma->vm_page_prot);else prot =pgprot_writecombine (vma->vm_page_prot);
@@ -6079,9 +6096,9 @@ static void mlx5_ib_cleanup_multiport_master(struct mlx5_ib_dev *dev)mlx5_nic_vport_disable_roce (dev->mdev); } -static int var_obj_cleanup (struct ib_uobject *uobject, -enum rdma_remove_reason why, -struct uverbs_attr_bundle *attrs) +static int mmap_obj_cleanup (struct ib_uobject *uobject, +enum rdma_remove_reason why, +struct uverbs_attr_bundle *attrs) {struct mlx5_user_mmap_entry *obj = uobject->object;
@@ -6089,6 +6106,16 @@ static int var_obj_cleanup(struct ib_uobject *uobject,return 0 ; } +static int mlx5_rdma_user_mmap_entry_insert (struct mlx5_ib_ucontext *c, +struct mlx5_user_mmap_entry *entry, +size_t length) +{ +return rdma_user_mmap_entry_insert_range ( + &c->ibucontext, &entry->rdma_entry, length, + (MLX5_IB_MMAP_OFFSET_START <<16 ), + ((MLX5_IB_MMAP_OFFSET_END <<16 ) + (1UL <<16 ) -1 )); +} +static struct mlx5_user_mmap_entry *alloc_var_entry (struct mlx5_ib_ucontext *c) {
@@ -6119,10 +6146,8 @@ alloc_var_entry(struct mlx5_ib_ucontext *c) entry->page_idx = page_idx; entry->mmap_flag =MLX5_IB_MMAP_TYPE_VAR ; - err =rdma_user_mmap_entry_insert_range ( - &c->ibucontext, &entry->rdma_entry, var_table->stride_size, -MLX5_IB_MMAP_OFFSET_START <<16 , - (MLX5_IB_MMAP_OFFSET_END <<16 ) + (1UL <<16 ) -1 ); + err =mlx5_rdma_user_mmap_entry_insert (c, entry, + var_table->stride_size);if (err) goto err_insert;
@@ -6206,7 +6231,7 @@ DECLARE_UVERBS_NAMED_METHOD_DESTROY( UA_MANDATORY));DECLARE_UVERBS_NAMED_OBJECT (MLX5_IB_OBJECT_VAR , -UVERBS_TYPE_ALLOC_IDR (var_obj_cleanup), +UVERBS_TYPE_ALLOC_IDR (mmap_obj_cleanup), &UVERBS_METHOD (MLX5_IB_METHOD_VAR_OBJ_ALLOC ), &UVERBS_METHOD (MLX5_IB_METHOD_VAR_OBJ_DESTROY ));
@@ -6218,6 +6243,134 @@ static bool var_is_supported(struct ib_device *device) MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_NET_Q); } +static struct mlx5_user_mmap_entry * +alloc_uar_entry (struct mlx5_ib_ucontext *c, +enum mlx5_ib_uapi_uar_alloc_type alloc_type) +{ +struct mlx5_user_mmap_entry *entry; +struct mlx5_ib_dev *dev; +u32 uar_index; +int err; + + entry =kzalloc (sizeof (*entry),GFP_KERNEL ); +if (!entry) +return ERR_PTR (-ENOMEM ); + + dev =to_mdev (c->ibucontext.device); + err =mlx5_cmd_alloc_uar (dev->mdev, &uar_index); +if (err) + goto end; + + entry->page_idx = uar_index; + entry->address =uar_index2paddress (dev, uar_index); +if (alloc_type ==MLX5_IB_UAPI_UAR_ALLOC_TYPE_BF ) + entry->mmap_flag =MLX5_IB_MMAP_TYPE_UAR_WC ; +else + entry->mmap_flag =MLX5_IB_MMAP_TYPE_UAR_NC ; + + err =mlx5_rdma_user_mmap_entry_insert (c, entry,PAGE_SIZE ); +if (err) + goto err_insert; + +return entry; + +err_insert: +mlx5_cmd_free_uar (dev->mdev, uar_index); +end: +kfree (entry); +return ERR_PTR (err); +} + +static int UVERBS_HANDLER (MLX5_IB_METHOD_UAR_OBJ_ALLOC )( +struct uverbs_attr_bundle *attrs) +{ +struct ib_uobject *uobj =uverbs_attr_get_uobject ( + attrs,MLX5_IB_ATTR_UAR_OBJ_ALLOC_HANDLE ); +enum mlx5_ib_uapi_uar_alloc_type alloc_type; +struct mlx5_ib_ucontext *c; +struct mlx5_user_mmap_entry *entry; +u64 mmap_offset; +u32 length; +int err; + + c =to_mucontext (ib_uverbs_get_ucontext (attrs)); +if (IS_ERR (c)) +return PTR_ERR (c); + + err =uverbs_get_const (&alloc_type, attrs, +MLX5_IB_ATTR_UAR_OBJ_ALLOC_TYPE ); +if (err) +return err; + +if (alloc_type !=MLX5_IB_UAPI_UAR_ALLOC_TYPE_BF && + alloc_type !=MLX5_IB_UAPI_UAR_ALLOC_TYPE_NC ) +return -EOPNOTSUPP ; + +if (!to_mdev (c->ibucontext.device)->wc_support && + alloc_type ==MLX5_IB_UAPI_UAR_ALLOC_TYPE_BF ) +return -EOPNOTSUPP ; + + entry =alloc_uar_entry (c, alloc_type); +if (IS_ERR (entry)) +return PTR_ERR (entry); + + mmap_offset =mlx5_entry_to_mmap_offset (entry); + length = entry->rdma_entry.npages *PAGE_SIZE ; + uobj->object = entry; + + err =uverbs_copy_to (attrs,MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_OFFSET , + &mmap_offset,sizeof (mmap_offset)); +if (err) + goto err; + + err =uverbs_copy_to (attrs,MLX5_IB_ATTR_UAR_OBJ_ALLOC_PAGE_ID , + &entry->page_idx,sizeof (entry->page_idx)); +if (err) + goto err; + + err =uverbs_copy_to (attrs,MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_LENGTH , + &length,sizeof (length)); +if (err) + goto err; + +return 0 ; + +err: +rdma_user_mmap_entry_remove (&entry->rdma_entry); +return err; +} + +DECLARE_UVERBS_NAMED_METHOD ( +MLX5_IB_METHOD_UAR_OBJ_ALLOC , +UVERBS_ATTR_IDR (MLX5_IB_ATTR_UAR_OBJ_ALLOC_HANDLE , +MLX5_IB_OBJECT_UAR , +UVERBS_ACCESS_NEW , +UA_MANDATORY ), +UVERBS_ATTR_CONST_IN (MLX5_IB_ATTR_UAR_OBJ_ALLOC_TYPE , + enum mlx5_ib_uapi_uar_alloc_type, +UA_MANDATORY ), +UVERBS_ATTR_PTR_OUT (MLX5_IB_ATTR_UAR_OBJ_ALLOC_PAGE_ID , +UVERBS_ATTR_TYPE (u32), +UA_MANDATORY ), +UVERBS_ATTR_PTR_OUT (MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_LENGTH , +UVERBS_ATTR_TYPE (u32), +UA_MANDATORY ), +UVERBS_ATTR_PTR_OUT (MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_OFFSET , +UVERBS_ATTR_TYPE (u64), +UA_MANDATORY )); + +DECLARE_UVERBS_NAMED_METHOD_DESTROY ( +MLX5_IB_METHOD_UAR_OBJ_DESTROY , +UVERBS_ATTR_IDR (MLX5_IB_ATTR_UAR_OBJ_DESTROY_HANDLE , +MLX5_IB_OBJECT_UAR , +UVERBS_ACCESS_DESTROY , +UA_MANDATORY )); + +DECLARE_UVERBS_NAMED_OBJECT (MLX5_IB_OBJECT_UAR , +UVERBS_TYPE_ALLOC_IDR (mmap_obj_cleanup), + &UVERBS_METHOD (MLX5_IB_METHOD_UAR_OBJ_ALLOC ), + &UVERBS_METHOD (MLX5_IB_METHOD_UAR_OBJ_DESTROY )); + ADD_UVERBS_ATTRIBUTES_SIMPLE( mlx5_ib_dm, UVERBS_OBJECT_DM,
@@ -6249,6 +6402,7 @@ static const struct uapi_definition mlx5_ib_defs[] = {UAPI_DEF_CHAIN_OBJ_TREE (UVERBS_OBJECT_DM , &mlx5_ib_dm),UAPI_DEF_CHAIN_OBJ_TREE_NAMED (MLX5_IB_OBJECT_VAR ,UAPI_DEF_IS_OBJ_SUPPORTED (var_is_supported)), + UAPI_DEF_CHAIN_OBJ_TREE_NAMED(MLX5_IB_OBJECT_UAR), {} };
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
@@ -124,6 +124,8 @@ enum {enum mlx5_ib_mmap_type {MLX5_IB_MMAP_TYPE_MEMIC =1 ,MLX5_IB_MMAP_TYPE_VAR =2 , +MLX5_IB_MMAP_TYPE_UAR_WC =3 , +MLX5_IB_MMAP_TYPE_UAR_NC =4 , };struct mlx5_ib_ucontext {
include/rdma/uverbs_ioctl.h +1 -1
@@ -173,7 +173,7 @@ enum uapi_radix_data {UVERBS_API_OBJ_KEY_BITS =5 ,UVERBS_API_OBJ_KEY_SHIFT =UVERBS_API_METHOD_KEY_BITS +UVERBS_API_METHOD_KEY_SHIFT , -UVERBS_API_OBJ_KEY_NUM_CORE =24 , +UVERBS_API_OBJ_KEY_NUM_CORE =20 ,UVERBS_API_OBJ_KEY_NUM_DRIVER = (1 <<UVERBS_API_OBJ_KEY_BITS ) -UVERBS_API_OBJ_KEY_NUM_CORE , UVERBS_API_OBJ_KEY_MASK = GENMASK(31, UVERBS_API_OBJ_KEY_SHIFT),
include/uapi/rdma/mlx5_user_ioctl_cmds.h +18 -0
@@ -131,6 +131,23 @@ enum mlx5_ib_var_obj_methods { MLX5_IB_METHOD_VAR_OBJ_DESTROY, }; +enum mlx5_ib_uar_alloc_attrs { +MLX5_IB_ATTR_UAR_OBJ_ALLOC_HANDLE = (1U <<UVERBS_ID_NS_SHIFT ), +MLX5_IB_ATTR_UAR_OBJ_ALLOC_TYPE , +MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_OFFSET , +MLX5_IB_ATTR_UAR_OBJ_ALLOC_MMAP_LENGTH , +MLX5_IB_ATTR_UAR_OBJ_ALLOC_PAGE_ID , +}; + +enum mlx5_ib_uar_obj_destroy_attrs { +MLX5_IB_ATTR_UAR_OBJ_DESTROY_HANDLE = (1U <<UVERBS_ID_NS_SHIFT ), +}; + +enum mlx5_ib_uar_obj_methods { +MLX5_IB_METHOD_UAR_OBJ_ALLOC = (1U <<UVERBS_ID_NS_SHIFT ), +MLX5_IB_METHOD_UAR_OBJ_DESTROY , +}; +enum mlx5_ib_devx_umem_reg_attrs {MLX5_IB_ATTR_DEVX_UMEM_REG_HANDLE = (1U <<UVERBS_ID_NS_SHIFT ),MLX5_IB_ATTR_DEVX_UMEM_REG_ADDR ,
@@ -190,6 +207,7 @@ enum mlx5_ib_objects {MLX5_IB_OBJECT_DEVX_ASYNC_EVENT_FD ,MLX5_IB_OBJECT_VAR ,MLX5_IB_OBJECT_PP , + MLX5_IB_OBJECT_UAR, }; enum mlx5_ib_flow_matcher_create_attrs {
include/uapi/rdma/mlx5_user_ioctl_verbs.h +5 -0
@@ -77,5 +77,10 @@ enum mlx5_ib_uapi_pp_alloc_flags { MLX5_IB_UAPI_PP_ALLOC_FLAGS_DEDICATED_INDEX = 1 << 0, }; +enum mlx5_ib_uapi_uar_alloc_type { +MLX5_IB_UAPI_UAR_ALLOC_TYPE_BF =0x0 , +MLX5_IB_UAPI_UAR_ALLOC_TYPE_NC =0x1 , +}; +#endif
Signature · 1 line
2.24.1
PATCH 3/5 IB/mlx5: Extend QP creation to get uar page index from user space
In reply to Leon RomanovskyFrom: Yishai Hadas <[email protected]>
Extend QP creation to get uar page index from user space, this mode can be used with the UAR dynamic mode APIs to allocate/destroy a UAR object.
As part of enabling this option blocked the weird/un-supported cross channel option which uses index 0 hard-coded.
This QP flag wasn't exposed to user space as part of any formal upstream release, the dynamic option can allow having valid UAR page index instead.
--- drivers/infiniband/hw/mlx5/qp.c | 27 +++++++++++++++++---------- include/uapi/rdma/mlx5-abi.h | 1 + 2 files changed, 18 insertions(+), 10 deletions(-)
drivers/infiniband/hw/mlx5/qp.c +17 -10
@@ -919,6 +919,7 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd,void *qpc;int err;u16 uid; +u32 uar_flags; err =ib_copy_from_udata (&ucmd, udata,sizeof (ucmd));if (err) {
@@ -928,24 +929,29 @@ static int create_user_qp(struct mlx5_ib_dev *dev, struct ib_pd *pd, context = rdma_udata_to_drv_context(udata, struct mlx5_ib_ucontext, ibucontext); -if (ucmd.flags &MLX5_QP_FLAG_BFREG_INDEX ) { + uar_flags = ucmd.flags & (MLX5_QP_FLAG_UAR_PAGE_INDEX | +MLX5_QP_FLAG_BFREG_INDEX ); +switch (uar_flags) { +case MLX5_QP_FLAG_UAR_PAGE_INDEX : + uar_index = ucmd.bfreg_index; + bfregn =MLX5_IB_INVALID_BFREG ; +break ; +case MLX5_QP_FLAG_BFREG_INDEX : uar_index =bfregn_to_uar_index (dev, &context->bfregi, ucmd.bfreg_index, true);if (uar_index <0 )return uar_index; - bfregn =MLX5_IB_INVALID_BFREG ; - }else if (qp->flags &MLX5_IB_QP_CROSS_CHANNEL ) { -/* -* TBD: should come from the verbs when we have the API -*/ -/* In CROSS_CHANNEL CQ and QP must use the same UAR */ - bfregn =MLX5_CROSS_CHANNEL_BFREG ; - } -else { +break ; +case 0 : +if (qp->flags &MLX5_IB_QP_CROSS_CHANNEL ) +return -EINVAL ; bfregn =alloc_bfreg (dev, &context->bfregi);if (bfregn <0 )return bfregn; +break ; +default : +return -EINVAL ; }mlx5_ib_dbg (dev,"bfregn 0x%x, uar_index 0x%x\n" , bfregn, uar_index);
@@ -2100,6 +2106,7 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC |MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_UC |MLX5_QP_FLAG_TUNNEL_OFFLOADS | +MLX5_QP_FLAG_UAR_PAGE_INDEX |MLX5_QP_FLAG_TYPE_DCI | MLX5_QP_FLAG_TYPE_DCT)) return -EINVAL;
include/uapi/rdma/mlx5-abi.h +1 -0
@@ -49,6 +49,7 @@ enum {MLX5_QP_FLAG_TIR_ALLOW_SELF_LB_MC =1 <<7 ,MLX5_QP_FLAG_ALLOW_SCATTER_CQE =1 <<8 ,MLX5_QP_FLAG_PACKET_BASED_CREDIT_MODE =1 <<9 , + MLX5_QP_FLAG_UAR_PAGE_INDEX = 1 << 10, }; enum {
Signature · 1 line
2.24.1
PATCH 4/5 IB/mlx5: Limit the scope of struct mlx5_bfreg_info to mlx5_ib
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
struct mlx5_bfreg_info is used by mlx5_ib only but is exposed to both RDMA and netdev parts of mlx5 driver. Move that struct to mlx5_ib namespace, clean vertical space alignment and convert lib_uar_4k from bool to bitfield.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++++++++++++++++ include/linux/mlx5/driver.h | 17 ----------------- 2 files changed, 17 insertions(+), 17 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +17 -0
@@ -128,6 +128,23 @@ enum mlx5_ib_mmap_type { MLX5_IB_MMAP_TYPE_UAR_NC = 4, }; +struct mlx5_bfreg_info { +u32 *sys_pages; +int num_low_latency_bfregs; +unsigned int *count; + +/* +* protect bfreg allocation data structs +*/ +struct mutex lock; +u32 ver; +u8 lib_uar_4k :1 ; +u32 num_sys_pages; +u32 num_static_sys_pages; +u32 total_num_bfregs; +u32 num_dyn_bfregs; +}; +struct mlx5_ib_ucontext {struct ib_ucontext ibucontext;struct list_head db_page_list;
include/linux/mlx5/driver.h +0 -17
@@ -213,23 +213,6 @@ enum mlx5_port_status { MLX5_PORT_DOWN = 2, }; -struct mlx5_bfreg_info { -u32 *sys_pages; -int num_low_latency_bfregs; -unsigned int *count; - -/* -* protect bfreg allocation data structs -*/ -struct mutex lock; -u32 ver; -bool lib_uar_4k; -u32 num_sys_pages; -u32 num_static_sys_pages; -u32 total_num_bfregs; -u32 num_dyn_bfregs; -}; -struct mlx5_cmd_first {__be32 data[4 ]; };
Signature · 1 line
2.24.1
PATCH 5/5 IB/mlx5: Move to fully dynamic UAR mode once user space supports it
In reply to Leon RomanovskyFrom: Yishai Hadas <[email protected]>
Move to fully dynamic UAR mode once user space supports it. In this case we prevent any legacy mode of UARs on the allocated context and prevent redundant allocation of the static ones.
--- drivers/infiniband/hw/mlx5/cq.c | 8 ++++++-- drivers/infiniband/hw/mlx5/main.c | 13 ++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 1 + drivers/infiniband/hw/mlx5/qp.c | 6 ++++++ include/uapi/rdma/mlx5-abi.h | 1 + 5 files changed, 26 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +6 -2
@@ -764,10 +764,14 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata,MLX5_SET (cqc, cqc, log_page_size, page_shift -MLX5_ADAPTER_PAGE_SHIFT ); -if (ucmd.flags &MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX ) +if (ucmd.flags &MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX ) { *index = ucmd.uar_page_index; -else + }else if (context->bfregi.lib_uar_dyn) { + err = -EINVAL ; + goto err_cqb; + }else { *index = context->bfregi.sys_pages[0 ]; + }if (ucmd.cqe_comp_en ==1 ) {int mini_cqe_format;
drivers/infiniband/hw/mlx5/main.c +12 -1
@@ -1787,6 +1787,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, max_cqe_version);u32 dump_fill_mkey;bool lib_uar_4k; +bool lib_uar_dyn;if (!dev->ib_active)return -EAGAIN ;
@@ -1845,8 +1846,14 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx, } lib_uar_4k = req.lib_caps &MLX5_LIB_CAP_4K_UAR ; + lib_uar_dyn = req.lib_caps &MLX5_LIB_CAP_DYN_UAR ; bfregi = &context->bfregi; +if (lib_uar_dyn) { + bfregi->lib_uar_dyn = lib_uar_dyn; + goto uar_done; + } +/* updates req->total_num_bfregs */ err =calc_total_bfregs (dev, lib_uar_4k, &req, bfregi);if (err)
@@ -1873,6 +1880,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,if (err) goto out_sys_pages; +uar_done:if (req.flags &MLX5_IB_ALLOC_UCTX_DEVX ) { err =mlx5_ib_devx_create (dev, true);if (err <0 )
@@ -1894,7 +1902,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,INIT_LIST_HEAD (&context->db_page_list);mutex_init (&context->db_page_mutex); - resp.tot_bfregs = req.total_num_bfregs; + resp.tot_bfregs = lib_uar_dyn ?0 : req.total_num_bfregs; resp.num_ports = dev->num_ports;if (offsetofend (typeof (resp), cqe_version) <= udata->outlen)
@@ -2142,6 +2150,9 @@ static int uar_mmap(struct mlx5_ib_dev *dev, enum mlx5_ib_mmap_cmd cmd,int max_valid_idx = dyn_uar ? bfregi->num_sys_pages : bfregi->num_static_sys_pages; +if (bfregi->lib_uar_dyn) +return -EINVAL ; +if (vma->vm_end - vma->vm_start !=PAGE_SIZE )return -EINVAL ;
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
@@ -139,6 +139,7 @@ struct mlx5_bfreg_info {struct mutex lock;u32 ver; u8 lib_uar_4k : 1; + u8 lib_uar_dyn : 1;u32 num_sys_pages;u32 num_static_sys_pages;u32 total_num_bfregs;
drivers/infiniband/hw/mlx5/qp.c +6 -0
@@ -697,6 +697,9 @@ static int alloc_bfreg(struct mlx5_ib_dev *dev, {int bfregn = -ENOMEM ; +if (bfregi->lib_uar_dyn) +return -EINVAL ; +mutex_lock (&bfregi->lock);if (bfregi->ver >=2 ) { bfregn =alloc_high_class_bfreg (dev, bfregi);
@@ -768,6 +771,9 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev,u32 index_of_sys_page;u32 offset; +if (bfregi->lib_uar_dyn) +return -EINVAL ; + bfregs_per_sys_page =get_uars_per_sys_page (dev, bfregi->lib_uar_4k) *MLX5_NON_FP_BFREGS_PER_UAR ; index_of_sys_page = bfregn / bfregs_per_sys_page;
include/uapi/rdma/mlx5-abi.h +1 -0
@@ -79,6 +79,7 @@ struct mlx5_ib_alloc_ucontext_req {enum mlx5_lib_caps {MLX5_LIB_CAP_4K_UAR = (__u64 )1 <<0 , +MLX5_LIB_CAP_DYN_UAR = (__u64 )1 <<1 , };enum mlx5_ib_alloc_uctx_v2_flags {
Signature · 1 line
2.24.1
PATCH 2/5 IB/mlx5: Extend CQ creation to get uar page index from user space
In reply to Leon RomanovskyFrom: Yishai Hadas <[email protected]>
Extend CQ creation to get uar page index from user space, this mode can be used with the UAR dynamic mode APIs to allocate/destroy a UAR object.
--- drivers/infiniband/hw/mlx5/cq.c | 17 +++++++++++------ include/uapi/rdma/mlx5-abi.h | 4 ++++ 2 files changed, 15 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mlx5/cq.c +11 -6
@@ -715,17 +715,19 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata,struct mlx5_ib_ucontext *context =rdma_udata_to_drv_context ( udata, struct mlx5_ib_ucontext, ibucontext); - ucmdlen = udata->inlen <sizeof (ucmd) ? - (sizeof (ucmd) -sizeof (ucmd.flags)) :sizeof (ucmd); + ucmdlen =min (udata->inlen,sizeof (ucmd)); +if (ucmdlen < offsetof(struct mlx5_ib_create_cq , flags)) +return -EINVAL ;if (ib_copy_from_udata (&ucmd, udata, ucmdlen))return -EFAULT ; -if (ucmdlen ==sizeof (ucmd) && - (ucmd.flags & ~(MLX5_IB_CREATE_CQ_FLAGS_CQE_128B_PAD ))) +if ((ucmd.flags & ~(MLX5_IB_CREATE_CQ_FLAGS_CQE_128B_PAD | +MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX )))return -EINVAL ; -if (ucmd.cqe_size !=64 && ucmd.cqe_size !=128 ) +if ((ucmd.cqe_size !=64 && ucmd.cqe_size !=128 ) || + ucmd.reserved0 || ucmd.reserved1)return -EINVAL ; *cqe_size = ucmd.cqe_size;
@@ -762,7 +764,10 @@ static int create_cq_user(struct mlx5_ib_dev *dev, struct ib_udata *udata,MLX5_SET (cqc, cqc, log_page_size, page_shift -MLX5_ADAPTER_PAGE_SHIFT ); - *index = context->bfregi.sys_pages[0 ]; +if (ucmd.flags &MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX ) + *index = ucmd.uar_page_index; +else + *index = context->bfregi.sys_pages[0 ];if (ucmd.cqe_comp_en ==1 ) {int mini_cqe_format;
include/uapi/rdma/mlx5-abi.h +4 -0
@@ -266,6 +266,7 @@ struct mlx5_ib_query_device_resp {enum mlx5_ib_create_cq_flags {MLX5_IB_CREATE_CQ_FLAGS_CQE_128B_PAD =1 <<0 , +MLX5_IB_CREATE_CQ_FLAGS_UAR_PAGE_INDEX =1 <<1 , };struct mlx5_ib_create_cq {
@@ -275,6 +276,9 @@ struct mlx5_ib_create_cq {__u8 cqe_comp_en;__u8 cqe_comp_res_format;__u16 flags; +__u16 uar_page_index; +__u16 reserved0; +__u32 reserved1; };struct mlx5_ib_create_cq_resp {
Signature · 1 line
2.24.1
Jason Gunthorpe
AppliedQuoted 33 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog:
v1: * Added patch that movedmlx5_bfreg_infofrom global header to themlx5_ib.h
* No other changes.
v0: * https://lore.kernel.org/linux-rdma/[email protected]----------------------------------------------------------------------------------
Quoted 1 line
From Yishai,
This series exposes API to enable a dynamic allocation and management of a UAR which now becomes to be a regular uobject.
Moving to that mode enables allocating a UAR only upon demand and drop the redundant static allocation of UARs upon context creation.
In addition, it allows master and secondary processes that own the same command FD to allocate and manage UARs according to their needs, this can’t be achieved today.
As part of this option, QP & CQ creation flows were adapted to support this dynamic UAR mode once asked by user space.
Once this mode is asked by mlx5 user space driver on a given context, it will be mutual exclusive, means both the static and legacy dynamic modes for using UARs will be blocked.
The legacy modes are supported for backward compatible reasons, looking forward we expect this new mode to be the default.
Thanks
Quoted 8 lines
Leon Romanovsky (1):
IB/mlx5: Limit the scope of structmlx5_bfreg_infotomlx5_ibYishai Hadas (4):
IB/mlx5: Expose UAR object and its alloc/destroy commands IB/mlx5: Extend CQ creation to get uar page index from user space IB/mlx5: Extend QP creation to get uar page index from user space IB/mlx5: Move to fully dynamic UAR mode once user space supports it
Applied to for-next, thanks
Jason