User space steering
Subsystems (5)
cover User space steering
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Ariel,
This series of patches adds user space managed steering infrastructure to the mlx5_ib driver.
User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the mlx5_ib API to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.
The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Thanks
Ariel Levkovich (8):
net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX user net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 155 +++++++++- drivers/infiniband/hw/mlx5/cmd.h | 8 +- drivers/infiniband/hw/mlx5/devx.c | 4 + drivers/infiniband/hw/mlx5/main.c | 264 +++++++++++++++--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 40 ++- drivers/infiniband/hw/mlx5/mr.c | 40 ++- drivers/infiniband/hw/mlx5/qp.c | 46 ++- .../ethernet/mellanox/mlx5/core/transobj.c | 18 +- include/linux/mlx5/mlx5_ifc.h | 51 +++- include/linux/mlx5/transobj.h | 3 + include/uapi/rdma/mlx5-abi.h | 2 + include/uapi/rdma/mlx5_user_ioctl_cmds.h | 1 + include/uapi/rdma/mlx5_user_ioctl_verbs.h | 6 + 14 files changed, 541 insertions(+), 99 deletions(-)
PATCH 1/8 net/mlx5: Expose SW ICM related device memory capabilities
From: Ariel Levkovich <[email protected]>
Add SW ICM related fields to the device memory capabilities structure and sw ownership capability in flow table properties.
The currently supported SW ICM types are steering and header modify and the changes exposes the device memory capabilities for each of these two types.
SW ICM memory can be allocated by SW and then be accessed by RDMA operations for direct management of the HW packet handling tables.
--- include/linux/mlx5/mlx5_ifc.h | 45 +++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-)
include/linux/mlx5/mlx5_ifc.h +43 -2
@@ -80,6 +80,14 @@ enum { MLX5_SHARED_RESOURCE_UID = 0xffff, }; +enum { +MLX5_OBJ_TYPE_SW_ICM =0x0008 , +}; + +enum { +MLX5_GENERAL_OBJ_TYPES_CAP_SW_ICM = (1ULL <<MLX5_OBJ_TYPE_SW_ICM ), +}; +enum {MLX5_CMD_OP_QUERY_HCA_CAP =0x100 ,MLX5_CMD_OP_QUERY_ADAPTER =0x101 ,
@@ -357,7 +365,8 @@ struct mlx5_ifc_flow_table_prop_layout_bits {u8 pop_vlan_2[0x1 ];u8 push_vlan_2[0x1 ];u8 reformat_and_vlan_action[0x1 ]; -u8 reserved_at_10[0x2 ]; +u8 reserved_at_10[0x1 ]; +u8 sw_owner[0x1 ];u8 reformat_l3_tunnel_to_l2[0x1 ];u8 reformat_l2_to_l3_tunnel[0x1 ];u8 reformat_and_modify_action[0x1 ];
@@ -770,7 +779,19 @@ struct mlx5_ifc_device_mem_cap_bits {u8 max_memic_size[0x20 ]; -u8 reserved_at_c0[0x740 ]; +u8 steering_sw_icm_start_address[0x40 ]; + +u8 reserved_at_100[0x8 ]; +u8 log_header_modify_sw_icm_size[0x8 ]; +u8 reserved_at_110[0x2 ]; +u8 log_sw_icm_alloc_granularity[0x6 ]; +u8 log_steering_sw_icm_size[0x8 ]; + +u8 reserved_at_120[0x20 ]; + +u8 header_modify_sw_icm_start_address[0x40 ]; + +u8 reserved_at_180[0x680 ]; };enum {
@@ -919,6 +940,7 @@ enum {enum {MLX5_UCTX_CAP_RAW_TX =1UL <<0 , +MLX5_UCTX_CAP_INTERNAL_DEV_RES =1UL <<1 , };struct mlx5_ifc_cmd_hca_cap_bits {
@@ -2922,6 +2944,7 @@ enum {MLX5_MKC_ACCESS_MODE_MTT =0x1 ,MLX5_MKC_ACCESS_MODE_KLMS =0x2 ,MLX5_MKC_ACCESS_MODE_KSM =0x3 , +MLX5_MKC_ACCESS_MODE_SW_ICM =0x4 , MLX5_MKC_ACCESS_MODE_MEMIC = 0x5, };
@@ -9444,6 +9467,19 @@ struct mlx5_ifc_uctx_bits {u8 reserved_at_20[0x160 ]; }; +struct mlx5_ifc_sw_icm_bits { +u8 modify_field_select[0x40 ]; + +u8 reserved_at_40[0x18 ]; +u8 log_sw_icm_size[0x8 ]; + +u8 reserved_at_60[0x20 ]; + +u8 sw_icm_start_addr[0x40 ]; + +u8 reserved_at_c0[0x140 ]; +}; +struct mlx5_ifc_create_umem_in_bits {u8 opcode[0x10 ];u8 uid[0x10 ];
@@ -9481,6 +9517,11 @@ struct mlx5_ifc_destroy_uctx_in_bits {u8 reserved_at_60[0x20 ]; }; +struct mlx5_ifc_create_sw_icm_in_bits { +struct mlx5_ifc_general_obj_in_cmd_hdr_bits hdr; +struct mlx5_ifc_sw_icm_bits sw_icm; +}; +struct mlx5_ifc_mtrc_string_db_param_bits {u8 string_db_base_address[0x20 ];
Signature · 1 line
2.20.1
PATCH 2/8 IB/mlx5: Support device memory type attribute
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
This patch intoruduces a new mlx5_ib driver attribute to the DM allocation method - the DM type.
In order to allow addition of new types in downstream patches this patch also refactors the allocation, deallocation and registration handlers to consider the requested type and perform the necessary actions according to it.
Since not all future device memory types will be such that are mapped to user memory, the mandatory page index output attribute is modified to be optional.
--- drivers/infiniband/hw/mlx5/cmd.c | 30 ++--- drivers/infiniband/hw/mlx5/cmd.h | 4 +- drivers/infiniband/hw/mlx5/main.c | 130 ++++++++++++++-------- drivers/infiniband/hw/mlx5/mlx5_ib.h | 23 ++-- drivers/infiniband/hw/mlx5/mr.c | 33 +++--- include/uapi/rdma/mlx5_user_ioctl_cmds.h | 1 + include/uapi/rdma/mlx5_user_ioctl_verbs.h | 4 + 7 files changed, 143 insertions(+), 82 deletions(-)
drivers/infiniband/hw/mlx5/cmd.c +15 -15
@@ -82,10 +82,10 @@ int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *dev,return mlx5_cmd_exec (dev, in, in_size, out,sizeof (out)); } -int mlx5_cmd_alloc_memic (struct mlx5_memic *memic,phys_addr_t *addr, -u64 length,u32 alignment) +int mlx5_cmd_alloc_memic (struct mlx5_dm *dm,phys_addr_t *addr, +u64 length,u32 alignment) { -struct mlx5_core_dev *dev = memic->dev; +struct mlx5_core_dev *dev = dm->dev;u64 num_memic_hw_pages =MLX5_CAP_DEV_MEM (dev, memic_bar_size) >>PAGE_SHIFT ;u64 hw_start_addr =MLX5_CAP64_DEV_MEM (dev, memic_bar_start_addr);
@@ -115,17 +115,17 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr, mlx5_alignment);while (page_idx < num_memic_hw_pages) { -spin_lock (&memic->memic_lock); - page_idx =bitmap_find_next_zero_area (memic->memic_alloc_pages, +spin_lock (&dm->lock); + page_idx =bitmap_find_next_zero_area (dm->memic_alloc_pages, num_memic_hw_pages, page_idx, num_pages,0 );if (page_idx < num_memic_hw_pages) -bitmap_set (memic->memic_alloc_pages, +bitmap_set (dm->memic_alloc_pages, page_idx, num_pages); -spin_unlock (&memic->memic_lock); +spin_unlock (&dm->lock);if (page_idx >= num_memic_hw_pages)break ;
@@ -135,10 +135,10 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr, ret =mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out));if (ret) { -spin_lock (&memic->memic_lock); -bitmap_clear (memic->memic_alloc_pages, +spin_lock (&dm->lock); +bitmap_clear (dm->memic_alloc_pages, page_idx, num_pages); -spin_unlock (&memic->memic_lock); +spin_unlock (&dm->lock);if (ret == -EAGAIN ) { page_idx++;
@@ -157,9 +157,9 @@ int mlx5_cmd_alloc_memic(struct mlx5_memic *memic, phys_addr_t *addr,return -ENOMEM ; } -int mlx5_cmd_dealloc_memic (struct mlx5_memic *memic,u64 addr,u64 length) +int mlx5_cmd_dealloc_memic (struct mlx5_dm *dm,u64 addr,u64 length) { -struct mlx5_core_dev *dev = memic->dev; +struct mlx5_core_dev *dev = dm->dev;u64 hw_start_addr =MLX5_CAP64_DEV_MEM (dev, memic_bar_start_addr);u32 num_pages =DIV_ROUND_UP (length,PAGE_SIZE );u32 out[MLX5_ST_SZ_DW (dealloc_memic_out)] = {0 };
@@ -177,10 +177,10 @@ int mlx5_cmd_dealloc_memic(struct mlx5_memic *memic, u64 addr, u64 length) err =mlx5_cmd_exec (dev, in,sizeof (in), out,sizeof (out));if (!err) { -spin_lock (&memic->memic_lock); -bitmap_clear (memic->memic_alloc_pages, +spin_lock (&dm->lock); +bitmap_clear (dm->memic_alloc_pages, start_page_idx, num_pages); -spin_unlock (&memic->memic_lock); +spin_unlock (&dm->lock); }return err;
drivers/infiniband/hw/mlx5/cmd.h +2 -2
@@ -44,9 +44,9 @@ int mlx5_cmd_query_cong_params(struct mlx5_core_dev *dev, int cong_point,int mlx5_cmd_query_ext_ppcnt_counters (struct mlx5_core_dev *dev,void *out);int mlx5_cmd_modify_cong_params (struct mlx5_core_dev *mdev,void *in,int in_size); -int mlx5_cmd_alloc_memic (struct mlx5_memic *memic,phys_addr_t *addr, +int mlx5_cmd_alloc_memic (struct mlx5_dm *dm,phys_addr_t *addr,u64 length,u32 alignment); -int mlx5_cmd_dealloc_memic (struct mlx5_memic *memic,u64 addr,u64 length); +int mlx5_cmd_dealloc_memic (struct mlx5_dm *dm,u64 addr,u64 length);void mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid);void mlx5_cmd_destroy_tir (struct mlx5_core_dev *dev,u32 tirn,u16 uid);void mlx5_cmd_destroy_tis (struct mlx5_core_dev *dev,u32 tisn,u16 uid);
drivers/infiniband/hw/mlx5/main.c +86 -44
181 lines not on this page · raw message
drivers/infiniband/hw/mlx5/mlx5_ib.h +15 -8
@@ -48,6 +48,7 @@#include <rdma/mlx5-abi.h> #include <rdma/uverbs_ioctl.h> #include <rdma/mlx5_user_ioctl_cmds.h> +#include <rdma/mlx5_user_ioctl_verbs.h> #include "srq.h"
@@ -558,15 +559,17 @@ enum mlx5_ib_mtt_access_flags {struct mlx5_ib_dm {struct ib_dm ibdm;phys_addr_t dev_addr; +u32 type; +size_t size; };#define MLX5_IB_MTT_PRESENT (MLX5_IB_MTT_READ | MLX5_IB_MTT_WRITE) -#define MLX5_IB_DM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\ - IB_ACCESS_REMOTE_WRITE |\ - IB_ACCESS_REMOTE_READ |\ - IB_ACCESS_REMOTE_ATOMIC |\ - IB_ZERO_BASED) +#define MLX5_IB_DM_MEMIC_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\ + IB_ACCESS_REMOTE_WRITE |\ + IB_ACCESS_REMOTE_READ |\ + IB_ACCESS_REMOTE_ATOMIC |\ + IB_ZERO_BASED)struct mlx5_ib_mr {struct ib_mr ibmr;
@@ -847,9 +850,13 @@ struct mlx5_ib_flow_action { }; }; -struct mlx5_memic { +struct mlx5_dm {struct mlx5_core_dev *dev; -spinlock_t memic_lock; +/* This lock is used to protect the access to the shared +* allocation map when concurrent requests by different +* processes are handled. +*/ +spinlock_t lock; DECLARE_BITMAP(memic_alloc_pages, MLX5_MAX_MEMIC_PAGES); };
@@ -953,7 +960,7 @@ struct mlx5_ib_dev {u8 umr_fence;struct list_head ib_dev_list;u64 sys_image_guid; -struct mlx5_memic memic; +struct mlx5_dm dm;u16 devx_whitelist_uid;struct mlx5_srq_table srq_table;struct mlx5_async_ctx async_ctx;
drivers/infiniband/hw/mlx5/mr.c +20 -13
@@ -1159,8 +1159,8 @@ static void set_mr_fields(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr, mr->access_flags = access_flags; } -static struct ib_mr *mlx5_ib_get_memic_mr (struct ib_pd *pd,u64 memic_addr, -u64 length,int acc) +static struct ib_mr *mlx5_ib_get_dm_mr (struct ib_pd *pd,u64 start_addr, +u64 length,int acc,int mode) {struct mlx5_ib_dev *dev =to_mdev (pd->device);int inlen =MLX5_ST_SZ_BYTES (create_mkey_in);
@@ -1182,9 +1182,8 @@ static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr, mkc =MLX5_ADDR_OF (create_mkey_in, in, memory_key_mkey_entry); -MLX5_SET (mkc, mkc, access_mode_1_0,MLX5_MKC_ACCESS_MODE_MEMIC &0x3 ); -MLX5_SET (mkc, mkc, access_mode_4_2, - (MLX5_MKC_ACCESS_MODE_MEMIC >>2 ) &0x7 ); +MLX5_SET (mkc, mkc, access_mode_1_0, mode &0x3 ); +MLX5_SET (mkc, mkc, access_mode_4_2, (mode >>2 ) &0x7 );MLX5_SET (mkc, mkc, a, !!(acc &IB_ACCESS_REMOTE_ATOMIC ));MLX5_SET (mkc, mkc, rw, !!(acc &IB_ACCESS_REMOTE_WRITE ));MLX5_SET (mkc, mkc, rr, !!(acc &IB_ACCESS_REMOTE_READ ));
@@ -1194,8 +1193,7 @@ static struct ib_mr *mlx5_ib_get_memic_mr(struct ib_pd *pd, u64 memic_addr,MLX5_SET64 (mkc, mkc, len, length);MLX5_SET (mkc, mkc, pd,to_mpd (pd)->pdn);MLX5_SET (mkc, mkc, qpn,0xffffff ); -MLX5_SET64 (mkc, mkc, start_addr, - memic_addr -pci_resource_start (dev->mdev->pdev,0 )); +MLX5_SET64 (mkc, mkc, start_addr, start_addr); err =mlx5_core_create_mkey (mdev, &mr->mmkey, in, inlen);if (err)
@@ -1237,15 +1235,24 @@ struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm, struct uverbs_attr_bundle *attrs) {struct mlx5_ib_dm *mdm =to_mdm (dm); -u64 memic_addr; +struct mlx5_core_dev *dev =to_mdev (dm->device)->mdev; +u64 start_addr = mdm->dev_addr + attr->offset; +int mode; -if (attr->access_flags & ~MLX5_IB_DM_ALLOWED_ACCESS ) -return ERR_PTR (-EINVAL ); +switch (mdm->type) { +case MLX5_IB_UAPI_DM_TYPE_MEMIC : +if (attr->access_flags & ~MLX5_IB_DM_MEMIC_ALLOWED_ACCESS ) +return ERR_PTR (-EINVAL ); - memic_addr = mdm->dev_addr + attr->offset; + mode =MLX5_MKC_ACCESS_MODE_MEMIC ; + start_addr -=pci_resource_start (dev->pdev,0 ); +break ; +default : +return ERR_PTR (-EINVAL ); + } -return mlx5_ib_get_memic_mr (pd, memic_addr, attr->length, - attr->access_flags); +return mlx5_ib_get_dm_mr (pd, start_addr, attr->length, + attr->access_flags, mode); } struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
include/uapi/rdma/mlx5_user_ioctl_cmds.h +1 -0
@@ -44,6 +44,7 @@ enum mlx5_ib_create_flow_action_attrs {enum mlx5_ib_alloc_dm_attrs {MLX5_IB_ATTR_ALLOC_DM_RESP_START_OFFSET = (1U <<UVERBS_ID_NS_SHIFT ),MLX5_IB_ATTR_ALLOC_DM_RESP_PAGE_INDEX , +MLX5_IB_ATTR_ALLOC_DM_REQ_TYPE , };enum mlx5_ib_devx_methods {
include/uapi/rdma/mlx5_user_ioctl_verbs.h +4 -0
@@ -57,5 +57,9 @@ struct mlx5_ib_uapi_devx_async_cmd_hdr { __u8 out_data[]; }; +enum mlx5_ib_uapi_dm_type { +MLX5_IB_UAPI_DM_TYPE_MEMIC , +}; +#endif
Signature · 1 line
2.20.1
PATCH 3/8 IB/mlx5: Warn on allocated MEMIC buffers during cleanup
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
Adding a warning on allocated MEMIC buffers that weren't freed prior to driver tear down.
--- drivers/infiniband/hw/mlx5/main.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/infiniband/hw/mlx5/main.c +2 -0
@@ -6005,6 +6005,8 @@ static void mlx5_ib_stage_init_cleanup(struct mlx5_ib_dev *dev)srcu_barrier (&dev->mr_srcu);cleanup_srcu_struct (&dev->mr_srcu); } + +WARN_ON (!bitmap_empty (dev->dm.memic_alloc_pages,MLX5_MAX_MEMIC_PAGES )); }static int mlx5_ib_stage_init_init (struct mlx5_ib_dev *dev)
Signature · 1 line
2.20.1
PATCH 4/8 IB/mlx5: Add steering SW ICM device memory type
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type, STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.
The type is provided by the user via the dedicated attribute in the alloc_dm ioctl command.
--- drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)
drivers/infiniband/hw/mlx5/Kconfig +1 -1
@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT ---help--- This driver provides low-level InfiniBand support for Mellanox Connect-IB PCI Express host channel adapters (HCAs).
drivers/infiniband/hw/mlx5/cmd.c +126 -1
139 lines not on this page · raw message
drivers/infiniband/hw/mlx5/cmd.h +5 -1
@@ -46,7 +46,7 @@ int mlx5_cmd_modify_cong_params(struct mlx5_core_dev *mdev, void *in, int in_size);int mlx5_cmd_alloc_memic (struct mlx5_dm *dm,phys_addr_t *addr,u64 length,u32 alignment); -int mlx5_cmd_dealloc_memic (struct mlx5_dm *dm,u64 addr,u64 length); +int mlx5_cmd_dealloc_memic (struct mlx5_dm *dm,phys_addr_t addr,u64 length);void mlx5_cmd_dealloc_pd (struct mlx5_core_dev *dev,u32 pdn,u16 uid);void mlx5_cmd_destroy_tir (struct mlx5_core_dev *dev,u32 tirn,u16 uid);void mlx5_cmd_destroy_tis (struct mlx5_core_dev *dev,u32 tisn,u16 uid);
@@ -65,4 +65,8 @@ int mlx5_cmd_alloc_q_counter(struct mlx5_core_dev *dev, u16 *counter_id, u16 uid);int mlx5_cmd_mad_ifc (struct mlx5_core_dev *dev,const void *inb,void *outb,u16 opmod,u8 port); +int mlx5_cmd_alloc_sw_icm (struct mlx5_dm *dm,int type,u64 length, +u16 uid,phys_addr_t *addr,u32 *obj_id); +int mlx5_cmd_dealloc_sw_icm (struct mlx5_dm *dm,int type,u64 length, +u16 uid,phys_addr_t addr,u32 obj_id);#endif /* MLX5_IB_CMD_H */
drivers/infiniband/hw/mlx5/main.c +133 -3
213 lines not on this page · raw message
drivers/infiniband/hw/mlx5/mlx5_ib.h +17 -0
@@ -118,6 +118,10 @@ enum { MLX5_MEMIC_BASE_SIZE = 1 << MLX5_MEMIC_BASE_ALIGN, }; +#define MLX5_LOG_SW_ICM_BLOCK_SIZE (dev) \ + (MLX5_CAP_DEV_MEM(dev, log_sw_icm_alloc_granularity)) +#define MLX5_SW_ICM_BLOCK_SIZE (dev) (1 << MLX5_LOG_SW_ICM_BLOCK_SIZE(dev)) +struct mlx5_ib_ucontext {struct ib_ucontext ibucontext;struct list_head db_page_list;
@@ -561,6 +565,12 @@ struct mlx5_ib_dm {phys_addr_t dev_addr;u32 type;size_t size; +union { +struct { +u32 obj_id; + } icm_dm; +/* other dm types specific params should be added here */ + }; };#define MLX5_IB_MTT_PRESENT (MLX5_IB_MTT_READ | MLX5_IB_MTT_WRITE)
@@ -571,6 +581,11 @@ struct mlx5_ib_dm { IB_ACCESS_REMOTE_ATOMIC |\ IB_ZERO_BASED) +#define MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS (IB_ACCESS_LOCAL_WRITE |\ + IB_ACCESS_REMOTE_WRITE |\ + IB_ACCESS_REMOTE_READ |\ + IB_ZERO_BASED) +struct mlx5_ib_mr {struct ib_mr ibmr;void *descs;
@@ -858,6 +873,8 @@ struct mlx5_dm {*/ spinlock_t lock;DECLARE_BITMAP (memic_alloc_pages,MLX5_MAX_MEMIC_PAGES ); +unsigned long *steering_sw_icm_alloc_blocks; +unsigned long *header_modify_sw_icm_alloc_blocks; };struct mlx5_read_counters_attr {
drivers/infiniband/hw/mlx5/mr.c +7 -0
@@ -1247,6 +1247,13 @@ struct ib_mr *mlx5_ib_reg_dm_mr(struct ib_pd *pd, struct ib_dm *dm, mode =MLX5_MKC_ACCESS_MODE_MEMIC ; start_addr -=pci_resource_start (dev->pdev,0 );break ; +case MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM : +case MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM : +if (attr->access_flags & ~MLX5_IB_DM_SW_ICM_ALLOWED_ACCESS ) +return ERR_PTR (-EINVAL ); + + mode =MLX5_MKC_ACCESS_MODE_SW_ICM ; +break ;default :return ERR_PTR (-EINVAL ); }
include/uapi/rdma/mlx5_user_ioctl_verbs.h +2 -0
@@ -59,6 +59,8 @@ struct mlx5_ib_uapi_devx_async_cmd_hdr {enum mlx5_ib_uapi_dm_type {MLX5_IB_UAPI_DM_TYPE_MEMIC , +MLX5_IB_UAPI_DM_TYPE_STEERING_SW_ICM , +MLX5_IB_UAPI_DM_TYPE_HEADER_MODIFY_SW_ICM , };#endif
Signature · 1 line
2.20.1
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 27 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)drivers/infiniband/hw/mlx5/Kconfig +1
-0+++ b/drivers/infiniband/hw/mlx5/Kconfig
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 4 lines
drivers/infiniband/hw/mlx5/Kconfig +1
-1@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BIT
Why do we need this?
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)
Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
We are using phys_addr_t type and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. Such PHYS_ADDR_T_64BIT will ensure that phys_addr_t will be always 64 bits.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.
???
PHYS_ADDR_T should only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.
This is not a reason to force PHYS_ADDR_T_64BIT from a driver
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 41 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.
Quoted 7 lines
???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driver
So how will I mark that we are forwarding physical address to FW?
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 44 lines · Leon Romanovsky
Quoted 43 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
Quoted 7 lines
Quoted 5 lines
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
I don't know what this means.. phys_addr_t will transparently 0 extend to 64 bit which is the correct thing to do when passing a phys_addr_t to HW.
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 49 lines · Jason Gunthorpe
Quoted 48 lines · Leon Romanovsky
Quoted 47 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.
It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 52 lines · Leon Romanovsky
Quoted 51 lines · Jason Gunthorpe
Quoted 50 lines · Leon Romanovsky
Quoted 48 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driver
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
So how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
So there is no reason for the kconfig hunk then
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 55 lines · Jason Gunthorpe
Quoted 54 lines · Leon Romanovsky
Quoted 53 lines · Jason Gunthorpe
Quoted 51 lines · Leon Romanovsky
Quoted 41 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.Quoted 7 lines
???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
So there is no reason for the kconfig hunk then
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
Thanks
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 60 lines · Leon Romanovsky
Quoted 59 lines · Jason Gunthorpe
Quoted 58 lines · Leon Romanovsky
Quoted 56 lines · Jason Gunthorpe
Quoted 44 lines · Leon Romanovsky
Quoted 43 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
Quoted 7 lines
Quoted 5 lines
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
So there is no reason for the kconfig hunk then
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
What?
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 64 lines · Jason Gunthorpe
Quoted 63 lines · Leon Romanovsky
Quoted 62 lines · Jason Gunthorpe
Quoted 60 lines · Leon Romanovsky
Quoted 49 lines · Jason Gunthorpe
Quoted 48 lines · Leon Romanovsky
Quoted 47 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.Quoted 7 lines
Quoted 3 lines
Quoted 1 line
This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
So there is no reason for the kconfig hunk then
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
What?
I wanted to say "not truncated", sorry about that.
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 65 lines · Leon Romanovsky
Quoted 64 lines · Jason Gunthorpe
Quoted 63 lines · Leon Romanovsky
Quoted 62 lines · Jason Gunthorpe
Quoted 60 lines · Leon Romanovsky
Quoted 49 lines · Jason Gunthorpe
Quoted 48 lines · Leon Romanovsky
Quoted 47 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.Quoted 7 lines
Quoted 3 lines
Quoted 1 line
This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
So there is no reason for the kconfig hunk then
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
What?
I wanted to say "not truncated", sorry about that.
Are we reading back a u64 physical value from the HW? Where?
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 68 lines · Jason Gunthorpe
Quoted 67 lines · Leon Romanovsky
Quoted 66 lines · Jason Gunthorpe
Quoted 65 lines · Leon Romanovsky
Quoted 63 lines · Jason Gunthorpe
Quoted 52 lines · Leon Romanovsky
Quoted 51 lines · Jason Gunthorpe
Quoted 50 lines · Leon Romanovsky
Quoted 48 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering a new device memory type,
STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in the
alloc_dmioctl command.Signed-off-by Ariel Levkovich <[email protected]>Reviewed-by Eli Cohen <[email protected]>Reviewed-by Mark Bloch <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++- drivers/infiniband/hw/mlx5/cmd.h | 6 +- drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++- drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++ drivers/infiniband/hw/mlx5/mr.c | 7 ++ include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 + 7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
drivers/infiniband/hw/mlx5/Kconfig line 3
Quoted 8 lines
drivers/infiniband/hw/mlx5/Kconfig +2
-1+++ b/drivers/infiniband/hw/mlx5/Kconfig@@ -1,6 +1,6 @@ config MLX5_INFINIBAND tristate "Mellanox 5th generation network adapters (ConnectX series) support" - depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE + depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverQuoted 8 lines
Quoted 5 lines
Quoted 1 line
So how will I mark that we are forwarding physical address to FW?
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
So there is no reason for the kconfig hunk then
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
What?
I wanted to say "not truncated", sorry about that.
Are we reading back a u64 physical value from the HW? Where?
No, we are setting address only.
Quoted 2 lines
Jason
Re: [PATCH rdma-next 4/8] IB/mlx5: Add steering SW ICM device memory type
Quoted 72 lines · Leon Romanovsky
Quoted 71 lines · Jason Gunthorpe
Quoted 70 lines · Leon Romanovsky
Quoted 69 lines · Jason Gunthorpe
Quoted 67 lines · Leon Romanovsky
Quoted 55 lines · Jason Gunthorpe
Quoted 54 lines · Leon Romanovsky
Quoted 53 lines · Jason Gunthorpe
Quoted 51 lines · Leon Romanovsky
Quoted 41 lines · Jason Gunthorpe
Quoted 36 lines · Leon Romanovsky
Quoted 24 lines · Jason Gunthorpe
Quoted 23 lines · Leon Romanovsky
From: Ariel Levkovich <[email protected]>
This patch adds support for allocating, deallocating and registering
a new device memory type, STEERING_SW_ICM.
This memory can be allocated and used by a privileged user for direct
rule insertion and management of the device's steering tables.The type is provided by the user via the dedicated attribute in
the alloc_dm ioctl command.Signed-off-by: Ariel Levkovich <[email protected]>
Reviewed-by: Eli Cohen <[email protected]>
Reviewed-by: Mark Bloch <[email protected]>
Signed-off-by: Leon Romanovsky <[email protected]>
drivers/infiniband/hw/mlx5/Kconfig | 2 +-
drivers/infiniband/hw/mlx5/cmd.c | 127 +++++++++++++++++++-
drivers/infiniband/hw/mlx5/cmd.h | 6 +-
drivers/infiniband/hw/mlx5/main.c | 136 +++++++++++++++++++++-
drivers/infiniband/hw/mlx5/mlx5_ib.h | 17 +++
drivers/infiniband/hw/mlx5/mr.c | 7 ++
include/uapi/rdma/mlx5_user_ioctl_verbs.h | 2 +
7 files changed, 291 insertions(+), 6 deletions(-)Quoted 10 lines
Quoted 8 lines
diff --git a/drivers/infiniband/hw/mlx5/Kconfig b/drivers/infiniband/hw/mlx5/Kconfig
index 8d651c05de62..347d457fcb2f 100644
+++ b/drivers/infiniband/hw/mlx5/Kconfig
@@ -1,6 +1,6 @@
config MLX5_INFINIBAND
tristate "Mellanox 5th generation network adapters (ConnectX series) support"
- depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE
+ depends on NETDEVICES && ETHERNET && PCI && MLX5_CORE && PHYS_ADDR_T_64BITWhy do we need this?
Quoted 3 lines
We are using
phys_addr_ttype and not u64 as an input to FW to emphasize the caller interface, but it is different between 32 and 64 bits systems. SuchPHYS_ADDR_T_64BITwill ensure thatphys_addr_twill be always 64 bits.Quoted 7 lines
???
PHYS_ADDR_Tshould only ever be used to store a bus address - and if you are passing a bus address to/from FW then you 0 extend it on 32 bit, and range check on 64 bit.This is not a reason to force
PHYS_ADDR_T_64BITfrom a driverSo how will I mark that we are forwarding physical address to FW?
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I don't know what this means..
phys_addr_twill transparently 0 extend to 64 bit which is the correct thing to do when passing aphys_addr_tto HW.It looks to me very similar to what I wanted: mark input as an address and ensure that it is always u64 as our FW expects.
So there is no reason for the kconfig hunk then
The kconfig is needed to ensure that u64 value is truncated to be u32 on 32bits systems.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
What?
I wanted to say "not truncated", sorry about that.
Are we reading back a u64 physical value from the HW? Where?
No, we are setting address only.
There there is no need for the kconfig, the phys_addr_t transparently 0 extends to 64 bits which is the right thing to do.
Jason
PATCH 5/8 IB/mlx5: Device resource control for privileged DEVX user
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
For DEVX users who have SYS_RAWIO capability, we set the internal device resources capability when creating the UCTX. This will allow the device to restrict the allocation of internal device resources such as SW ICM memory to privileged DEVX users only.
--- drivers/infiniband/hw/mlx5/devx.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/infiniband/hw/mlx5/devx.c +4 -0
@@ -85,6 +85,10 @@ int mlx5_ib_devx_create(struct mlx5_ib_dev *dev, bool is_user)if (is_user &&capable (CAP_NET_RAW ) && (MLX5_CAP_GEN (dev->mdev, uctx_cap) &MLX5_UCTX_CAP_RAW_TX )) cap |=MLX5_UCTX_CAP_RAW_TX ; +if (is_user &&capable (CAP_SYS_RAWIO ) && + (MLX5_CAP_GEN (dev->mdev, uctx_cap) & +MLX5_UCTX_CAP_INTERNAL_DEV_RES )) + cap |=MLX5_UCTX_CAP_INTERNAL_DEV_RES ;MLX5_SET (create_uctx_in, in, opcode,MLX5_CMD_OP_CREATE_UCTX );MLX5_SET (uctx, uctx, cap, cap);
Signature · 1 line
2.20.1
PATCH 6/8 net/mlx5: Expose TIR ICM address in command outbox
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
Adding the TIR ICM address to the create_tir command outbox through which the device reports the ICM address of the newly created TIR.
The TIR address can be used for direct attachment to a steering rule in SW managed steering mode.
--- include/linux/mlx5/mlx5_ifc.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
include/linux/mlx5/mlx5_ifc.h +3 -3
@@ -6899,14 +6899,14 @@ struct mlx5_ifc_create_tis_in_bits {struct mlx5_ifc_create_tir_out_bits {u8 status[0x8 ]; -u8 reserved_at_8[0x18 ]; +u8 icm_address_63_40[0x18 ];u8 syndrome[0x20 ]; -u8 reserved_at_40[0x8 ]; +u8 icm_address_39_32[0x8 ];u8 tirn[0x18 ]; -u8 reserved_at_60[0x20 ]; +u8 icm_address_31_0[0x20 ]; };struct mlx5_ifc_create_tir_in_bits {
Signature · 1 line
2.20.1
PATCH 7/8 net/mlx5: Introduce new TIR creation core API
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
Introducing new TIR creation core API which allows caller to receive back from the call the full command outbox.
This comes as a preparation for the next patch that will retrieve the TIR ICM address from the command outbox.
--- .../net/ethernet/mellanox/mlx5/core/transobj.c | 18 +++++++++++++----- include/linux/mlx5/transobj.h | 3 +++ 2 files changed, 16 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/transobj.c +13 -5
@@ -182,16 +182,24 @@ int mlx5_core_query_sq_state(struct mlx5_core_dev *dev, u32 sqn, u8 *state) }EXPORT_SYMBOL_GPL (mlx5_core_query_sq_state); +int mlx5_core_create_tir_out (struct mlx5_core_dev *dev, +u32 *in,int inlen, +u32 *out,int outlen) +{ +MLX5_SET (create_tir_in, in, opcode,MLX5_CMD_OP_CREATE_TIR ); + +return mlx5_cmd_exec (dev, in, inlen, out, outlen); +} +EXPORT_SYMBOL (mlx5_core_create_tir_out); +int mlx5_core_create_tir (struct mlx5_core_dev *dev,u32 *in,int inlen,u32 *tirn) { -u32 out[MLX5_ST_SZ_DW (create_tir_out)] = {0 }; +u32 out[MLX5_ST_SZ_DW (create_tir_out)] = {};int err; -MLX5_SET (create_tir_in, in, opcode,MLX5_CMD_OP_CREATE_TIR ); - -memset (out,0 ,sizeof (out)); - err =mlx5_cmd_exec (dev, in, inlen, out,sizeof (out)); + err =mlx5_core_create_tir_out (dev, in, inlen, + out,sizeof (out));if (!err) *tirn =MLX5_GET (create_tir_out, out, tirn);
include/linux/mlx5/transobj.h +3 -0
@@ -50,6 +50,9 @@ int mlx5_core_query_sq(struct mlx5_core_dev *dev, u32 sqn, u32 *out);int mlx5_core_query_sq_state (struct mlx5_core_dev *dev,u32 sqn,u8 *state);int mlx5_core_create_tir (struct mlx5_core_dev *dev,u32 *in,int inlen,u32 *tirn); +int mlx5_core_create_tir_out (struct mlx5_core_dev *dev, +u32 *in,int inlen, +u32 *out,int outlen);int mlx5_core_modify_tir (struct mlx5_core_dev *dev,u32 tirn,u32 *in,int inlen);void mlx5_core_destroy_tir (struct mlx5_core_dev *dev,u32 tirn);
Signature · 1 line
2.20.1
PATCH 8/8 IB/mlx5: Expose TIR ICM address to user space
In reply to Leon RomanovskyFrom: Ariel Levkovich <[email protected]>
This patch exposes the TIR ICM address of raw packet and RSS QPs to user space.
In order to pass the new field, the patch extends the mlx5 specific QP creation response structure and fills it with the icm address returned by the FW command, if available.
--- drivers/infiniband/hw/mlx5/qp.c | 46 +++++++++++++++++++++++++++++---- include/uapi/rdma/mlx5-abi.h | 2 ++ 2 files changed, 43 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx5/qp.c +41 -5
@@ -1397,7 +1397,8 @@ static void destroy_raw_packet_qp_tir(struct mlx5_ib_dev *dev,static int create_raw_packet_qp_tir (struct mlx5_ib_dev *dev,struct mlx5_ib_rq *rq,u32 tdn,u32 *qp_flags_en, -struct ib_pd *pd) +struct ib_pd *pd, +u32 *out,int outlen) {u8 lb_flag =0 ;u32 *in;
@@ -1431,8 +1432,9 @@ static int create_raw_packet_qp_tir(struct mlx5_ib_dev *dev,MLX5_SET (tirc, tirc, self_lb_block, lb_flag); - err =mlx5_core_create_tir (dev->mdev, in, inlen, &rq->tirn); + err =mlx5_core_create_tir_out (dev->mdev, in, inlen, out, outlen); + rq->tirn =MLX5_GET (create_tir_out, out, tirn);if (!err &&MLX5_GET (tirc, tirc, self_lb_block)) { err =mlx5_ib_enable_lb (dev, false, true);
@@ -1458,6 +1460,7 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,int err;u32 tdn = mucontext->tdn;u16 uid =to_mpd (pd)->uid; +u32 out[MLX5_ST_SZ_DW (create_tir_out)] = {};if (qp->sq.wqe_cnt) { err =create_raw_packet_qp_tis (dev, qp, sq, tdn, pd);
@@ -1490,7 +1493,9 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,if (err) goto err_destroy_sq; - err =create_raw_packet_qp_tir (dev, rq, tdn, &qp->flags_en, pd); + err =create_raw_packet_qp_tir ( + dev, rq, tdn, &qp->flags_en, pd, out, +MLX5_ST_SZ_BYTES (create_tir_out));if (err) goto err_destroy_rq;
@@ -1499,6 +1504,20 @@ static int create_raw_packet_qp(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, resp->comp_mask |=MLX5_IB_CREATE_QP_RESP_MASK_RQN ; resp->tirn = rq->tirn; resp->comp_mask |=MLX5_IB_CREATE_QP_RESP_MASK_TIRN ; +if (MLX5_CAP_FLOWTABLE_NIC_RX (dev->mdev, sw_owner)) { + resp->tir_icm_addr =MLX5_GET ( + create_tir_out, out, icm_address_31_0); + resp->tir_icm_addr |= + (u64 )MLX5_GET (create_tir_out, out, + icm_address_39_32) + <<32 ; + resp->tir_icm_addr |= + (u64 )MLX5_GET (create_tir_out, out, + icm_address_63_40) + <<40 ; + resp->comp_mask |= +MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR ; + } } }
@@ -1572,8 +1591,10 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, udata, struct mlx5_ib_ucontext, ibucontext);struct mlx5_ib_create_qp_resp resp = {};int inlen; +int outlen;int err;u32 *in; +u32 *out;void *tirc;void *hfso;u32 selected_fields =0 ;
@@ -1653,10 +1674,12 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp, } inlen =MLX5_ST_SZ_BYTES (create_tir_in); - in =kvzalloc (inlen,GFP_KERNEL ); + outlen =MLX5_ST_SZ_BYTES (create_tir_out); + in =kvzalloc (inlen + outlen,GFP_KERNEL );if (!in)return -ENOMEM ; + out = in +MLX5_ST_SZ_DW (create_tir_in);MLX5_SET (create_tir_in, in, uid,to_mpd (pd)->uid); tirc =MLX5_ADDR_OF (create_tir_in, in, ctx); MLX5_SET(tirc, tirc, disp_type,
@@ -1768,8 +1791,9 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,MLX5_SET (rx_hash_field_select, hfso, selected_fields, selected_fields); create_tir: - err =mlx5_core_create_tir (dev->mdev, in, inlen, &qp->rss_qp.tirn); + err =mlx5_core_create_tir_out (dev->mdev, in, inlen, out, outlen); + qp->rss_qp.tirn =MLX5_GET (create_tir_out, out, tirn);if (!err &&MLX5_GET (tirc, tirc, self_lb_block)) { err =mlx5_ib_enable_lb (dev, false, true);
@@ -1784,6 +1808,18 @@ static int create_rss_raw_qp_tir(struct mlx5_ib_dev *dev, struct mlx5_ib_qp *qp,if (mucontext->devx_uid) { resp.comp_mask |=MLX5_IB_CREATE_QP_RESP_MASK_TIRN ; resp.tirn = qp->rss_qp.tirn; +if (MLX5_CAP_FLOWTABLE_NIC_RX (dev->mdev, sw_owner)) { + resp.tir_icm_addr = +MLX5_GET (create_tir_out, out, icm_address_31_0); + resp.tir_icm_addr |= (u64 )MLX5_GET (create_tir_out, out, + icm_address_39_32) + <<32 ; + resp.tir_icm_addr |= (u64 )MLX5_GET (create_tir_out, out, + icm_address_63_40) + <<40 ; + resp.comp_mask |= +MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR ; + } } err =ib_copy_to_udata (udata, &resp,min (udata->outlen,sizeof (resp)));
include/uapi/rdma/mlx5-abi.h +2 -0
@@ -359,6 +359,7 @@ enum mlx5_ib_create_qp_resp_mask {MLX5_IB_CREATE_QP_RESP_MASK_TISN =1UL <<1 ,MLX5_IB_CREATE_QP_RESP_MASK_RQN =1UL <<2 ,MLX5_IB_CREATE_QP_RESP_MASK_SQN =1UL <<3 , + MLX5_IB_CREATE_QP_RESP_MASK_TIR_ICM_ADDR = 1UL << 4, }; struct mlx5_ib_create_qp_resp {
@@ -370,6 +371,7 @@ struct mlx5_ib_create_qp_resp {__u32 rqn;__u32 sqn;__u32 reserved1; +__u64 tir_icm_addr; };struct mlx5_ib_alloc_mw {
Signature · 1 line
2.20.1
Quoted 17 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.
Quoted 6 lines
The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
Quoted 21 lines · Jakub Kicinski
Quoted 20 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
Quoted 7 lines
Quoted 3 lines
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
mlx5 devices steering operates on packets without relation to specific protocol. The decision to steer is based on HW capabilities and user configuration for matching specific fields. Such configuration is done through kernel and through DEVX.
General description of DEVX can be found in manuals of libmlx5.
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/
Thanks
Quoted 30 lines · Leon Romanovsky
Quoted 21 lines · Jakub Kicinski
Quoted 20 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
Quoted 7 lines
Quoted 3 lines
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
Quoted 8 lines
mlx5 devices steering operates on packets without relation to specific protocol. The decision to steer is based on HW capabilities and user configuration for matching specific fields. Such configuration is done through kernel and through DEVX.
General description of DEVX can be found in manuals of libmlx5.
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/
Thanks for those, looks quite powerful. What's your primary use case here? Is there some flow steering and encapsulation going on in RDMA world?
Quoted 36 lines · Jakub Kicinski
Quoted 35 lines · Leon Romanovsky
Quoted 21 lines · Jakub Kicinski
Quoted 20 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
Quoted 7 lines
Quoted 3 lines
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
mlx5 devices steering operates on packets without relation to specific protocol. The decision to steer is based on HW capabilities and user configuration for matching specific fields. Such configuration is done through kernel and through DEVX.
Quoted 7 lines
Quoted 3 lines
General description of DEVX can be found in manuals of libmlx5.
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/Thanks for those, looks quite powerful. What's your primary use case here? Is there some flow steering and encapsulation going on in RDMA world?
Usual use case for that is in RoCE fabric, but as I said it is not limited to that. For example, flow steering is used in libpcap [1] and encapsulation is used for efficient VM to VM communication over different hosts (I assume exactly as in netdev world).
In regarding to this patch set, the combination of this steering functionality with DEVX allows us to speedup greatly buildup of rules by doing it in powerful host CPU instead of internal to NIC CPU.
[1]
https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42
Thanks
Quoted 50 lines · Leon Romanovsky
Quoted 36 lines · Jakub Kicinski
Quoted 35 lines · Leon Romanovsky
Quoted 21 lines · Jakub Kicinski
Quoted 20 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
Quoted 7 lines
Quoted 3 lines
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
mlx5 devices steering operates on packets without relation to specific protocol. The decision to steer is based on HW capabilities and user configuration for matching specific fields. Such configuration is done through kernel and through DEVX.
Quoted 7 lines
Quoted 3 lines
General description of DEVX can be found in manuals of libmlx5.
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md
https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/Thanks for those, looks quite powerful. What's your primary use case here? Is there some flow steering and encapsulation going on in RDMA world?
Usual use case for that is in RoCE fabric, but as I said it is not limited to that. For example, flow steering is used in libpcap [1] and encapsulation is used for efficient VM to VM communication over different hosts (I assume exactly as in netdev world).
Quoted 6 lines
In regarding to this patch set, the combination of this steering functionality with DEVX allows us to speedup greatly buildup of rules by doing it in powerful host CPU instead of internal to NIC CPU.
[1]
https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42
Interesting, thanks for the info!
Quoted 24 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Ariel,
This series of patches adds user space managed steering infrastructure to the
mlx5_ibdriver.User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space.
The changes are introducing the
mlx5_ibAPI to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations.
In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs.
Quoted 8 lines
Thanks
Ariel Levkovich (8):
net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX user
This doesn't apply, it conflicts with patches in mlx5-next, please resent
Quoted 3 lines
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
Is this actually two patch series? This one looks OK, lets apply it to mlx5-next please
Jason
Quoted 45 lines · Jason Gunthorpe
Quoted 35 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Ariel,
This series of patches adds user space managed steering infrastructure
to themlx5_ibdriver.User space managed steering requires the means to access a dedicated
memory space that is used by the device to store the packet steering
and header modification tables and rules in order to manage them directly
without the device's firmware involvement. This dedicated memory is part
of the ICM memory space.The changes are introducing the
mlx5_ibAPI to allocate, deallocate and
register this dedicated SW ICM memory via the existing device memory API
using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it
using remote RDMA operations.In addition, the series exposed the ICM address of the receive transport
interface (TIR) of Raw Packet and RSS QPs to user since they are required
to properly create and insert steering rules that direct flows to these QPs.Quoted 8 lines
Thanks
Ariel Levkovich (8):
net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX user
Quoted 7 lines
This doesn't apply, it conflicts with patches in mlx5-next, please resent
Quoted 3 lines
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
it is a 3 patch series:
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API net/mlx5: Expose SW ICM related device memory capabilities
LGTM, I can apply them if you want, or do you want to wait for leon ?
Quoted 54 lines · Saeed Mahameed
Quoted 45 lines · Jason Gunthorpe
Quoted 35 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Ariel,
This series of patches adds user space managed steering infrastructure
to themlx5_ibdriver.User space managed steering requires the means to access a dedicated
memory space that is used by the device to store the packet steering
and header modification tables and rules in order to manage them directly
without the device's firmware involvement. This dedicated memory is part
of the ICM memory space.The changes are introducing the
mlx5_ibAPI to allocate, deallocate and
register this dedicated SW ICM memory via the existing device memory API
using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it
using remote RDMA operations.In addition, the series exposed the ICM address of the receive transport
interface (TIR) of Raw Packet and RSS QPs to user since they are required
to properly create and insert steering rules that direct flows to these QPs.Quoted 8 lines
Thanks
Ariel Levkovich (8):
net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX userQuoted 7 lines
This doesn't apply, it conflicts with patches in mlx5-next, please resent
Quoted 3 lines
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
Quoted 6 lines
it is a 3 patch series:
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API net/mlx5: Expose SW ICM related device memory capabilitiesLGTM, I can apply them if you want, or do you want to wait for leon ?
Fine by me
Jason
Quoted 65 lines · Jason Gunthorpe
Quoted 64 lines · Saeed Mahameed
Quoted 50 lines · Jason Gunthorpe
Quoted 40 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Quoted 1 line
From Ariel,
This series of patches adds user space managed steering infrastructure
to themlx5_ibdriver.User space managed steering requires the means to access a dedicated
memory space that is used by the device to store the packet steering
and header modification tables and rules in order to manage them
directly
without the device's firmware involvement. This dedicated memory is
part
of the ICM memory space.The changes are introducing the
mlx5_ibAPI to allocate, deallocate
and
register this dedicated SW ICM memory via the existing device memory API
using a private attribute which specifies the memory type.The allocated memory itself is not IO mapped and user can only access it
using remote RDMA operations.In addition, the series exposed the ICM address of the receive transport
interface (TIR) of Raw Packet and RSS QPs to user since they are
required
to properly create and insert steering rules that direct flows to
these QPs.Quoted 8 lines
Thanks
Ariel Levkovich (8):
net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX userQuoted 7 lines
This doesn't apply, it conflicts with patches in mlx5-next, please
resentQuoted 3 lines
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
it is a 3 patch series:
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API net/mlx5: Expose SW ICM related device memory capabilities
Quoted 5 lines
Quoted 2 lines
LGTM, I can apply them if you want, or do you want to wait for leon ?
Fine by me
the above 3 mlx5-next patches applied.
Thanks,
Saeed.
Quoted 1 line
Jason
Quoted 3 lines · Leon Romanovsky
net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space
These are applied to for-next now
Thanks,
Jason