Skip to contentSkip to conversation

IB/mlx5: Test write combining support

PATCH Applied Leon Romanovsky 4 participants 5 messages Oct 27, 2019 – Oct 31, 2019

Subsystems (2)
InfiniBand Mellanox MLX5 IB driver
lore
Share & tools
lore

Leon Romanovsky

author
#0 lore raw

PATCH 1/1 IB/mlx5: Test write combining support

From: Michael Guralnik <[email protected]>

Add a test in mlx5_ib initialization process to test whether write-combining is supported on the machine.
The test will run as part of the enable_driver callback to ensure that the test runs after the device is setup and can create and modify the QP needed, but runs before the device is exposed to the users.

The test opens UD QP and posts NOP WQEs, the WQE written to the BlueFlame is differnet from the WQE in memory, requesting CQE only on the BlueFlame WQE. By checking whether we received a completion on one of these WQEs we can know if BlueFlame succeeded and whether write-combining is supported.

Change reporting of BlueFlame support to be dependent on write-combining support.

Signed-off-by Michael Guralnik <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>

---
Changelog:
v0 -> v1: https://lore.kernel.org/linux-rdma/[email protected]
* First patch was merged, resend second only.
* Changed spin_lock() to spin_lock_irqsave() in post_send_nop().
* Updated post_send_nop() current position math calculations.

---
 drivers/infiniband/hw/mlx5/main.c    |  15 +-
 drivers/infiniband/hw/mlx5/mem.c     | 199 +++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |   6 +
 drivers/infiniband/hw/mlx5/qp.c      |   6 +-
 4 files changed, 223 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/main.c +14 -1
index 20fabb368f61..fd3dd4523a22 100644
@@ -1796,7 +1796,7 @@ static int mlx5_ib_alloc_ucontext(struct ib_ucontext *uctx,
 		return -EINVAL;
 
 	resp.qp_tab_size = 1 << MLX5_CAP_GEN(dev->mdev, log_max_qp);
-	if (mlx5_core_is_pf(dev->mdev) && MLX5_CAP_GEN(dev->mdev, bf))
+	if (dev->wc_support)
 		resp.bf_reg_size = 1 << MLX5_CAP_GEN(dev->mdev, log_bf_reg_size);
 	resp.cache_line_size = cache_line_size();
 	resp.max_sq_desc_sz = MLX5_CAP_GEN(dev->mdev, max_wqe_sz_sq);

@@ -6257,6 +6257,7 @@ static const struct ib_device_ops mlx5_ib_dev_ops = {
 	.disassociate_ucontext = mlx5_ib_disassociate_ucontext,
 	.drain_rq = mlx5_ib_drain_rq,
 	.drain_sq = mlx5_ib_drain_sq,
+	.enable_driver = mlx5_ib_enable_driver,
 	.fill_res_entry = mlx5_ib_fill_res_entry,
 	.fill_stat_entry = mlx5_ib_fill_stat_entry,
 	.get_dev_fw_str = get_dev_fw_str,

@@ -6700,6 +6701,18 @@ static void mlx5_ib_stage_devx_cleanup(struct mlx5_ib_dev *dev)
 	}
 }
 
+int mlx5_ib_enable_driver(struct ib_device *dev)
+{
+	struct mlx5_ib_dev *mdev = to_mdev(dev);
+	int ret;
+
+	ret = mlx5_ib_test_wc(mdev);
+	mlx5_ib_dbg(mdev, "Write-Combining %s",
+		    mdev->wc_support ? "supported" : "not supported");
+
+	return ret;
+}
+
 void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 		      const struct mlx5_ib_profile *profile,
 		      int stage)
drivers/infiniband/hw/mlx5/mem.c +199 -0

208 lines not on this page · raw message

drivers/infiniband/hw/mlx5/mlx5_ib.h +6 -0
index d53739e43a93..d925ce0e46b5 100644
@@ -248,6 +248,7 @@ struct mlx5_ib_flow_db {
  * rely on the range reserved for that use in the ib_qp_create_flags enum.
  *./
 #define MLX5_IB_QP_CREATE_SQPN_QP1	IB_QP_CREATE_RESERVED_START
+#define MLX5_IB_QP_CREATE_WC_TEST	(IB_QP_CREATE_RESERVED_START << 1)
 
 struct wr_list {
 	u16	opcode;

@@ -962,6 +963,7 @@ struct mlx5_ib_dev {
 	u8				fill_delay:1;
 	u8				is_rep:1;
 	u8				lag_active:1;
+	u8				wc_support:1;
 	struct umr_common		umrc;
 	/* sync used page count stats
 	 */

@@ -987,6 +989,7 @@ struct mlx5_ib_dev {
 	/* Array with num_ports elements */
 	struct mlx5_ib_port	*port;
 	struct mlx5_sq_bfreg	bfreg;
+	struct mlx5_sq_bfreg	wc_bfreg;
 	struct mlx5_sq_bfreg	fp_bfreg;
 	struct mlx5_ib_delay_drop	delay_drop;
 	const struct mlx5_ib_profile	*profile;

@@ -1495,4 +1498,7 @@ static inline bool mlx5_ib_can_use_umr(struct mlx5_ib_dev *dev,
 
 	return true;
 }
+
+int mlx5_ib_enable_driver(struct ib_device *dev);
+int mlx5_ib_test_wc(struct mlx5_ib_dev *dev);
 #endif /* MLX5_IB_H */
drivers/infiniband/hw/mlx5/qp.c +4 -2
index bb3f432e2fb6..c831b455ffa8 100644
@@ -1041,11 +1041,14 @@ static int create_kernel_qp(struct mlx5_ib_dev *dev,
 					IB_QP_CREATE_BLOCK_MULTICAST_LOOPBACK |
 					IB_QP_CREATE_IPOIB_UD_LSO |
 					IB_QP_CREATE_NETIF_QP |
-					MLX5_IB_QP_CREATE_SQPN_QP1))
+					MLX5_IB_QP_CREATE_SQPN_QP1 |
+					MLX5_IB_QP_CREATE_WC_TEST))
 		return -EINVAL;
 
 	if (init_attr->qp_type == MLX5_IB_QPT_REG_UMR)
 		qp->bf.bfreg = &dev->fp_bfreg;
+	else if (init_attr->create_flags & MLX5_IB_QP_CREATE_WC_TEST)
+		qp->bf.bfreg = &dev->wc_bfreg;
 	else
 		qp->bf.bfreg = &dev->bfreg;
 

@@ -5328,7 +5331,6 @@ static int _mlx5_ib_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr,
 		 * we hit doorbell */
 		wmb();
 
-		/* currently we support only regular doorbells */
 		mlx5_write64((__be32 *)ctrl, bf->bfreg->map + bf->offset);
 		/* Make sure doorbells don't leak out of SQ spinlock
 		 * and reach the HCA out of order.

--
2.20.1

#1 lore raw
Quoted 11 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

Add a test in mlx5_ib initialization process to test whether write-combining is supported on the machine.
The test will run as part of the enable_driver callback to ensure that the test runs after the device is setup and can create and modify the QP needed, but runs before the device is exposed to the users.

The test opens UD QP and posts NOP WQEs, the WQE written to the BlueFlame is differnet from the WQE in memory, requesting CQE only on the BlueFlame

nit: different

Quoted 5 lines

WQE. By checking whether we received a completion on one of these WQEs we can know if BlueFlame succeeded and whether write-combining is supported.

Change reporting of BlueFlame support to be dependent on write-combining support.

author
#2 lore raw
Quoted 10 lines · Or Gerlitz
Quoted 9 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

Add a test in mlx5_ib initialization process to test whether write-combining is supported on the machine.
The test will run as part of the enable_driver callback to ensure that the test runs after the device is setup and can create and modify the QP needed, but runs before the device is exposed to the users.

Quoted 4 lines
Quoted 2 lines

The test opens UD QP and posts NOP WQEs, the WQE written to the BlueFlame is differnet from the WQE in memory, requesting CQE only on the BlueFlame

nit: different

Thanks, Jason/Doug should I resend?

Quoted 6 lines
Quoted 6 lines

WQE. By checking whether we received a completion on one of these WQEs we can know if BlueFlame succeeded and whether write-combining is supported.

Change reporting of BlueFlame support to be dependent on write-combining support.

#3 lore raw
Quoted 11 lines · Leon Romanovsky
Quoted 10 lines · Or Gerlitz
Quoted 9 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

Add a test in mlx5_ib initialization process to test whether write-combining is supported on the machine.
The test will run as part of the enable_driver callback to ensure that the test runs after the device is setup and can create and modify the QP needed, but runs before the device is exposed to the users.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

The test opens UD QP and posts NOP WQEs, the WQE written to the BlueFlame is differnet from the WQE in memory, requesting CQE only on the BlueFlame

nit: different

Thanks, Jason/Doug should I resend?

No, I usually fix stuff like that anyhow

Jason

Applied
#4 lore raw
In reply to Leon Romanovsky
Quoted 16 lines · Leon Romanovsky

From: Michael Guralnik <[email protected]>

Add a test in mlx5_ib initialization process to test whether write-combining is supported on the machine.
The test will run as part of the enable_driver callback to ensure that the test runs after the device is setup and can create and modify the QP needed, but runs before the device is exposed to the users.

The test opens UD QP and posts NOP WQEs, the WQE written to the BlueFlame is differnet from the WQE in memory, requesting CQE only on the BlueFlame WQE. By checking whether we received a completion on one of these WQEs we can know if BlueFlame succeeded and whether write-combining is supported.

Change reporting of BlueFlame support to be dependent on write-combining support.

Quoted 7 lines
Signed-off-by Michael Guralnik <[email protected]>
Signed-off-by Leon Romanovsky <[email protected]>

---
Changelog:
v0 -> v1: https://lore.kernel.org/linux-rdma/[email protected]
* First patch was merged, resend second only.
* Changed spin_lock() to spin_lock_irqsave() in post_send_nop().

Probably this should just be spin_lock_irq as we are definitely not already in a IRQ or atomic context at this point.

Quoted 7 lines

* Updated post_send_nop() current position math calculations.

---
 drivers/infiniband/hw/mlx5/main.c    |  15 +-
 drivers/infiniband/hw/mlx5/mem.c     | 199 +++++++++++++++++++++++++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |   6 +
 drivers/infiniband/hw/mlx5/qp.c      |   6 +-
 4 files changed, 223 insertions(+), 3 deletions(-)

Applied to for-next, thanks

Jason