Dump and fill MKEY

From: Yonatan Cohen <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to boost performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

Meaning, instead of a PCI write access the HCA leaves the target memory untouched, and skips filling that packet section. Similar behavior is done upon send, the HCA skips data in memory relevant to this key and saves PCI bus access.

This functionality saves PCI read/write operations.

Signed-off-by Yonatan Cohen <[email protected]>

Reviewed-by Yishai Hadas <[email protected]>

Reviewed-by Guy Levi <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

 drivers/infiniband/hw/mlx5/main.c | 16 +++++++++++++++-
 include/uapi/rdma/mlx5-abi.h      |  3 ++-
 2 files changed, 17 insertions(+), 2 deletions(-)

drivers/infiniband/hw/mlx5/main.c +2 -0

index c29c7c838980..97113957398d 100644
+++ b/drivers/infiniband/hw/mlx5/main.c

@@ -1634,6 +1634,7 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev,
 	int err;
 	size_t min_req_v2 = offsetof(struct mlx5_ib_alloc_ucontext_req_v2,
 				     max_cqe_version);
+	u32 dump_fill_mkey;
 	bool lib_uar_4k;
 
 	if (!dev->ib_active)

@@ -1743,8 +1744,12 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev,
 		}

Quoted 37 lines · Leon Romanovsky

From: Yonatan Cohen <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to boost performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

Meaning, instead of a PCI write access the HCA leaves the target memory untouched, and skips filling that packet section. Similar behavior is done upon send, the HCA skips data in memory relevant to this key and saves PCI bus access.

This functionality saves PCI read/write operations.

Signed-off-by Yonatan Cohen <[email protected]>

Reviewed-by Yishai Hadas <[email protected]>

Reviewed-by Guy Levi <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

 drivers/infiniband/hw/mlx5/main.c | 16 +++++++++++++++-
 include/uapi/rdma/mlx5-abi.h      |  3 ++-
 2 files changed, 17 insertions(+), 2 deletions(-)

drivers/infiniband/hw/mlx5/main.c +2 -0

index c29c7c838980..97113957398d 100644
+++ b/drivers/infiniband/hw/mlx5/main.c

@@ -1634,6 +1634,7 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev,
 	int err;
 	size_t min_req_v2 = offsetof(struct mlx5_ib_alloc_ucontext_req_v2,
 				     max_cqe_version);
+	u32 dump_fill_mkey;
 	bool lib_uar_4k;
 
 	if (!dev->ib_active)

@@ -1743,8 +1744,12 @@ static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev,
 		}
 

drivers/infiniband/hw/mlx5/main.c static struct ib_ucontext *mlx5_ib_alloc_ucontext(struct ib_device *ibdev, · line 1753

Quoted 9 lines

drivers/infiniband/hw/mlx5/main.c +5 -1

 		err = mlx5_ib_devx_create(dev, context);
+	}
+
+	if (MLX5_CAP_GEN(dev->mdev, dump_fill_mkey)) {
+		err = mlx5_cmd_dump_fill_mkey(dev->mdev, &dump_fill_mkey);
 		if (err)
-			goto out_td;
+			goto out_mdev;
 	}

Dropping the if (err) after mlx5_ib_devx_create is a rebasing error, right?

Jason

From: Leon Romanovsky <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to increase performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

In this three patch series, we expose various bits in our HW spec file (mlx5_ifc.h), move unneeded for mlx5_core FW command and export such memory key to user space thought our mlx5-abi header file.

Quoted 5 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to increase performance, when used in a send or receive operations.

Quoted 6 lines

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

In this three patch series, we expose various bits in our HW spec file (mlx5_ifc.h), move unneeded for mlx5_core FW command and export such memory key to user space thought our mlx5-abi header file.

Where is the user space for this?

Jason

From: Leon Romanovsky <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to increase performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

In this three patch series, we expose various bits in our HW spec file (mlx5_ifc.h), move unneeded for mlx5_core FW command and export such memory key to user space thought our mlx5-abi header file.

Thanks

Quoted 8 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to increase performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

Quoted 5 lines

In this three patch series, we expose various bits in our HW spec file (mlx5_ifc.h), move unneeded for mlx5_core FW command and export such memory key to user space thought our mlx5-abi header file.

Thanks

This looks fine, can you send a pull request off the mlx5 branch please?

Thanks,
Jason

Quoted 13 lines · Jason Gunthorpe

Quoted 12 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

MLX5 IB HCA offers the memory key, dump_fill_mkey to increase performance, when used in a send or receive operations.

It is used to force local HCA operations to skip the PCI bus access, while keeping track of the processed length in the ibv_sge handling.

In this three patch series, we expose various bits in our HW spec file (mlx5_ifc.h), move unneeded for mlx5_core FW command and export such memory key to user space thought our mlx5-abi header file.

Quoted 4 lines

Quoted 1 line

Thanks

This looks fine, can you send a pull request off the mlx5 branch please?

Updated mlx5-next with first two commits,
b183ee27f5fb net/mlx5: Add hardware definitions for dump_fill_mkey 4d4fb5dc988a net/mlx5: Limit scope of dump_fill_mkey function