Remove SRQ code from mlx5_core

PATCH Applied Leon Romanovsky 2 participants 13 messages Nov 28, 2018 – Dec 4, 2018

Subsystems (5)

Networking Network drivers InfiniBand Mellanox mlx5 core +1 more subsystems

lore

Share & tools

lore

Leon Romanovsky

author

2018-11-28 18:53 UTC #0 lore raw

cover Remove SRQ code from mlx5_core

From: Leon Romanovsky <[email protected]>

Hi Doug and Jason,

This series is intended to be applied on mlx5-next after Saeed's infrastructure changes series are accepted:
https://patchwork.kernel.org/project/linux-rdma/list/?series=47809

----------------------------------------------------------------------
The following patchset brings mlx5 SRQ functionality to the module which is actually using it - mlx5_ib. The original code being not used in mlx5_core/mlx5_en, but located there, caused to unneeded maintenance overhead.

By moving mlx5 SRQ code to mlx5_ib, we successfully reduced ~200 LOC codes and minimized exports and headers pollution.

The proposed patches are divided to three logical groups: move from netdev, fix IB part and cleanup.

Thanks

Leon Romanovsky (11):
net/mlx5: Align SRQ licenses and copyright information net/mlx5: Remove dead transobj code
net/mlx5: Remove not-used lib/eq.h header file net/mlx5: Remove references to local mlx5_core functions net/mlx5: Move SRQ functions to RDMA part RDMA/mlx5: Remove SRQ signature global flag RDMA/mlx5: Use stages for callback to setup and release DEVX RDMA/mlx5: Update SRQ functions signatures to mlx5_ib format RDMA/mlx5: Initialize SRQ tables on mlx5_ib RDMA/mlx5: Unfold create RMP function
RDMA/mlx5: Unfold modify RMP function

 drivers/infiniband/hw/mlx5/Makefile           |   4 +-
 drivers/infiniband/hw/mlx5/cq.c               |   4 +-
 drivers/infiniband/hw/mlx5/ib_rep.c           |   4 +
 drivers/infiniband/hw/mlx5/main.c             |  33 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |   6 +-
 drivers/infiniband/hw/mlx5/srq.c              |  49 +--
 drivers/infiniband/hw/mlx5/srq.h              |  73 ++++
 .../srq.c => infiniband/hw/mlx5/srq_cmd.c}    | 321 ++++++++----------
 .../net/ethernet/mellanox/mlx5/core/Makefile  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/main.c    |   5 -
 .../ethernet/mellanox/mlx5/core/transobj.c    | 109 ------
 include/linux/mlx5/driver.h                   |  33 --
 include/linux/mlx5/srq.h                      |  72 ----
 include/linux/mlx5/transobj.h                 |  11 -
 14 files changed, 274 insertions(+), 452 deletions(-)
 create mode 100644 drivers/infiniband/hw/mlx5/srq.h
 rename drivers/{net/ethernet/mellanox/mlx5/core/srq.c => infiniband/hw/mlx5/srq_cmd.c} (71%)
 delete mode 100644 include/linux/mlx5/srq.h

Leon Romanovsky

author

2018-11-28 18:53 UTC #1 lore raw

PATCH 1/11 net/mlx5: Align SRQ licenses and copyright information

From: Leon Romanovsky <[email protected]>

Ensure that both RDMA and netdev parts of SRQ implementation has same copyright and license information annotated by SPDX tags.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/srq.c              | 31 ++-----------------
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 31 ++-----------------
 include/linux/mlx5/srq.h                      | 31 ++-----------------
 3 files changed, 6 insertions(+), 87 deletions(-)

drivers/infiniband/hw/mlx5/srq.c +2 ~~-29~~

index d012e7dbcc38..28794780062e 100644
@@ -1,33 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
 /*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
+ * Copyright (c) 2013-2018, Mellanox Technologies inc.  All rights reserved.
  */
 
 #include <linux/module.h>

drivers/net/ethernet/mellanox/mlx5/core/srq.c +2 ~~-29~~

index 79c5f0d57956..10036aaa200a 100644
@@ -1,33 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
 /*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
+ * Copyright (c) 2013-2018, Mellanox Technologies inc.  All rights reserved.
  */
 
 #include <linux/kernel.h>

include/linux/mlx5/srq.h +2 ~~-29~~

index 1b1f3c20c6a3..77bc4264066d 100644
@@ -1,33 +1,6 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
 /*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
+ * Copyright (c) 2013-2018, Mellanox Technologies. All rights reserved.
  */
 
 #ifndef MLX5_SRQ_H

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #2 lore raw

PATCH 2/11 net/mlx5: Remove dead transobj code

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Delete functions which are not called and not needed.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 .../ethernet/mellanox/mlx5/core/transobj.c    | 66 -------------------
 include/linux/mlx5/transobj.h                 |  5 --
 2 files changed, 71 deletions(-)

drivers/net/ethernet/mellanox/mlx5/core/transobj.c +0 ~~-66~~

index a1ee9a8a769e..ab482124e901 100644
@@ -301,72 +301,6 @@ int mlx5_core_query_rmp(struct mlx5_core_dev *dev, u32 rmpn, u32 *out)
 	return mlx5_cmd_exec(dev, in, sizeof(in), out, outlen);
 }
 
-int mlx5_core_arm_rmp(struct mlx5_core_dev *dev, u32 rmpn, u16 lwm)
-{
-	void *in;
-	void *rmpc;
-	void *wq;
-	void *bitmask;
-	int  err;
-
-	in = kvzalloc(MLX5_ST_SZ_BYTES(modify_rmp_in), GFP_KERNEL);
-	if (!in)
-		return -ENOMEM;
-
-	rmpc    = MLX5_ADDR_OF(modify_rmp_in,   in,   ctx);
-	bitmask = MLX5_ADDR_OF(modify_rmp_in,   in,   bitmask);
-	wq      = MLX5_ADDR_OF(rmpc,	        rmpc, wq);
-
-	MLX5_SET(modify_rmp_in, in,	 rmp_state, MLX5_RMPC_STATE_RDY);
-	MLX5_SET(modify_rmp_in, in,	 rmpn,      rmpn);
-	MLX5_SET(wq,		wq,	 lwm,	    lwm);
-	MLX5_SET(rmp_bitmask,	bitmask, lwm,	    1);
-	MLX5_SET(rmpc,		rmpc,	 state,	    MLX5_RMPC_STATE_RDY);
-
-	err =  mlx5_core_modify_rmp(dev, in, MLX5_ST_SZ_BYTES(modify_rmp_in));
-
-	kvfree(in);
-
-	return err;
-}
-
-int mlx5_core_create_xsrq(struct mlx5_core_dev *dev, u32 *in, int inlen,
-			  u32 *xsrqn)
-{
-	u32 out[MLX5_ST_SZ_DW(create_xrc_srq_out)] = {0};
-	int err;
-
-	MLX5_SET(create_xrc_srq_in, in, opcode,     MLX5_CMD_OP_CREATE_XRC_SRQ);
-	err = mlx5_cmd_exec(dev, in, inlen, out, sizeof(out));
-	if (!err)
-		*xsrqn = MLX5_GET(create_xrc_srq_out, out, xrc_srqn);
-
-	return err;
-}
-
-int mlx5_core_destroy_xsrq(struct mlx5_core_dev *dev, u32 xsrqn)
-{
-	u32 in[MLX5_ST_SZ_DW(destroy_xrc_srq_in)]   = {0};
-	u32 out[MLX5_ST_SZ_DW(destroy_xrc_srq_out)] = {0};
-
-	MLX5_SET(destroy_xrc_srq_in, in, opcode,   MLX5_CMD_OP_DESTROY_XRC_SRQ);
-	MLX5_SET(destroy_xrc_srq_in, in, xrc_srqn, xsrqn);
-	return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
-}
-
-int mlx5_core_arm_xsrq(struct mlx5_core_dev *dev, u32 xsrqn, u16 lwm)
-{
-	u32 in[MLX5_ST_SZ_DW(arm_xrc_srq_in)]   = {0};
-	u32 out[MLX5_ST_SZ_DW(arm_xrc_srq_out)] = {0};
-
-	MLX5_SET(arm_xrc_srq_in, in, opcode,   MLX5_CMD_OP_ARM_XRC_SRQ);
-	MLX5_SET(arm_xrc_srq_in, in, xrc_srqn, xsrqn);
-	MLX5_SET(arm_xrc_srq_in, in, lwm,      lwm);
-	MLX5_SET(arm_xrc_srq_in, in, op_mod,
-		 MLX5_ARM_XRC_SRQ_IN_OP_MOD_XRC_SRQ);
-	return mlx5_cmd_exec(dev, in, sizeof(in), out, sizeof(out));
-}
-
 int mlx5_core_create_rqt(struct mlx5_core_dev *dev, u32 *in, int inlen,
 			 u32 *rqtn)
 {

include/linux/mlx5/transobj.h +0 -5

index 7f5ca2cd3a32..39ebb699875b 100644
@@ -63,11 +63,6 @@ int mlx5_core_create_rmp(struct mlx5_core_dev *dev, u32 *in, int inlen,
 int mlx5_core_modify_rmp(struct mlx5_core_dev *dev, u32 *in, int inlen);
 int mlx5_core_destroy_rmp(struct mlx5_core_dev *dev, u32 rmpn);
 int mlx5_core_query_rmp(struct mlx5_core_dev *dev, u32 rmpn, u32 *out);
-int mlx5_core_arm_rmp(struct mlx5_core_dev *dev, u32 rmpn, u16 lwm);
-int mlx5_core_create_xsrq(struct mlx5_core_dev *dev, u32 *in, int inlen,
-			  u32 *rmpn);
-int mlx5_core_destroy_xsrq(struct mlx5_core_dev *dev, u32 rmpn);
-int mlx5_core_arm_xsrq(struct mlx5_core_dev *dev, u32 rmpn, u16 lwm);
 
 int mlx5_core_create_rqt(struct mlx5_core_dev *dev, u32 *in, int inlen,
 			 u32 *rqtn);

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #3 lore raw

PATCH 3/11 net/mlx5: Remove not-used lib/eq.h header file

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

lib/eq.h is needed for EQ manipulation which are not performed in SRQ.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 1 -
 1 file changed, 1 deletion(-)

drivers/net/ethernet/mellanox/mlx5/core/srq.c +0 -1

index 10036aaa200a..248f1c8dd5d3 100644
@@ -11,7 +11,6 @@
 #include <rdma/ib_verbs.h>
 #include <linux/mlx5/transobj.h>
 #include "mlx5_core.h"
-#include "lib/eq.h"
 
 static int srq_event_notifier(struct notifier_block *nb,
 			      unsigned long type, void *data)

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #4 lore raw

PATCH 4/11 net/mlx5: Remove references to local mlx5_core functions

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

As a preparation to move SRQ functionality to RDMA, drop all references to mlx5_core logic and make SRQ be dependent on shared code only.

Most of the time, we are interested to know if events are working/not working and it is possible with previous commit ("net/mlx5: Debug print for forwarded async events").

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 22 +++----------------
 1 file changed, 3 insertions(+), 19 deletions(-)

drivers/net/ethernet/mellanox/mlx5/core/srq.c +3 ~~-19~~

index 248f1c8dd5d3..690815234838 100644
@@ -10,15 +10,12 @@
 #include <linux/mlx5/srq.h>
 #include <rdma/ib_verbs.h>
 #include <linux/mlx5/transobj.h>
-#include "mlx5_core.h"
 
 static int srq_event_notifier(struct notifier_block *nb,
 			      unsigned long type, void *data)
 {
 	struct mlx5_srq_table *table;
-	struct mlx5_core_dev *dev;
 	struct mlx5_core_srq *srq;
-	struct mlx5_priv *priv;
 	struct mlx5_eqe *eqe;
 	u32 srqn;

@@ -27,12 +24,9 @@ static int srq_event_notifier(struct notifier_block *nb,
 		return NOTIFY_DONE;
 
 	table = container_of(nb, struct mlx5_srq_table, nb);
-	priv  = container_of(table, struct mlx5_priv, srq_table);
-	dev   = container_of(priv, struct mlx5_core_dev, priv);
 
 	eqe = data;
 	srqn = be32_to_cpu(eqe->data.qp_srq.qp_srq_n) & 0xffffff;
-	mlx5_core_dbg(dev, "SRQ event (%d): srqn 0x%x\n", eqe->type, srqn);
 
 	spin_lock(&table->lock);

@@ -42,10 +36,8 @@ static int srq_event_notifier(struct notifier_block *nb,
 
 	spin_unlock(&table->lock);
 
-	if (!srq) {
-		mlx5_core_warn(dev, "Async event for bogus SRQ 0x%08x\n", srqn);
+	if (!srq)
 		return NOTIFY_OK;
-	}
 
 	srq->event(srq, eqe->type);

@@ -617,10 +609,8 @@ int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
 	spin_lock_irq(&table->lock);
 	err = radix_tree_insert(&table->tree, srq->srqn, srq);
 	spin_unlock_irq(&table->lock);
-	if (err) {
-		mlx5_core_warn(dev, "err %d, srqn 0x%x\n", err, srq->srqn);
+	if (err)
 		goto err_destroy_srq_split;
-	}
 
 	return 0;

@@ -640,14 +630,8 @@ int mlx5_core_destroy_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq)
 	spin_lock_irq(&table->lock);
 	tmp = radix_tree_delete(&table->tree, srq->srqn);
 	spin_unlock_irq(&table->lock);
-	if (!tmp) {
-		mlx5_core_warn(dev, "srq 0x%x not found in tree\n", srq->srqn);
+	if (!tmp || tmp != srq)
 		return -EINVAL;
-	}
-	if (tmp != srq) {
-		mlx5_core_warn(dev, "corruption on srqn 0x%x\n", srq->srqn);
-		return -EINVAL;
-	}
 
 	err = destroy_srq_split(dev, srq);
 	if (err)

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #5 lore raw

PATCH 5/11 net/mlx5: Move SRQ functions to RDMA part

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

There is no need to keep SRQ which is RDMA object in mlx5_core. In this patch, we partially move the execution code, while next patches will move table initialization/release logic too.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/Makefile           |   4 +-
 drivers/infiniband/hw/mlx5/cq.c               |   1 +
 drivers/infiniband/hw/mlx5/srq.c              |   2 +-
 .../infiniband/hw}/mlx5/srq.h                 |  21 +-
 .../srq.c => infiniband/hw/mlx5/srq_cmd.c}    |  97 +--
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 634 ------------------
 .../ethernet/mellanox/mlx5/core/transobj.c    |  43 --
 include/linux/mlx5/driver.h                   |   8 -
 include/linux/mlx5/srq.h                      |  31 -
 include/linux/mlx5/transobj.h                 |   6 -
 10 files changed, 49 insertions(+), 798 deletions(-)
 copy {include/linux => drivers/infiniband/hw}/mlx5/srq.h (50%)
 copy drivers/{net/ethernet/mellanox/mlx5/core/srq.c => infiniband/hw/mlx5/srq_cmd.c} (92%)

drivers/infiniband/hw/mlx5/Makefile +3 -1

index b8e4b15e2674..33f5adb14e4e 100644
@@ -1,6 +1,8 @@
 obj-$(CONFIG_MLX5_INFINIBAND)	+= mlx5_ib.o
 
-mlx5_ib-y :=	main.o cq.o doorbell.o qp.o mem.o srq.o mr.o ah.o mad.o gsi.o ib_virt.o cmd.o cong.o
+mlx5_ib-y :=	main.o cq.o doorbell.o qp.o mem.o srq_cmd.o \
+		srq.o mr.o ah.o mad.o gsi.o ib_virt.o cmd.o \
+		cong.o
 mlx5_ib-$(CONFIG_INFINIBAND_ON_DEMAND_PAGING) += odp.o
 mlx5_ib-$(CONFIG_MLX5_ESWITCH) += ib_rep.o
 mlx5_ib-$(CONFIG_INFINIBAND_USER_ACCESS) += devx.o

drivers/infiniband/hw/mlx5/cq.c +1 -0

index c631bd057013..a91580edca92 100644
@@ -35,6 +35,7 @@
 #include <rdma/ib_user_verbs.h>
 #include <rdma/ib_cache.h>
 #include "mlx5_ib.h"
+#include "srq.h"
 
 static void mlx5_ib_cq_comp(struct mlx5_core_cq *cq)
 {

drivers/infiniband/hw/mlx5/srq.c +1 -1

index 28794780062e..a86d9f153805 100644
@@ -9,8 +9,8 @@
 #include <linux/slab.h>
 #include <rdma/ib_umem.h>
 #include <rdma/ib_user_verbs.h>
-
 #include "mlx5_ib.h"
+#include "srq.h"
 
 /* not supported currently */
 static int srq_signature;

drivers/infiniband/hw/mlx5/srq.h renamed from include/linux/mlx5/srq.h +11 ~~-10~~

similarity index 50%
copy from include/linux/mlx5/srq.h
copy to drivers/infiniband/hw/mlx5/srq.h
index 77bc4264066d..f23d5de12973 100644
@@ -3,10 +3,8 @@
  * Copyright (c) 2013-2018, Mellanox Technologies. All rights reserved.
  */
 
-#ifndef MLX5_SRQ_H
-#define MLX5_SRQ_H
-
-#include <linux/mlx5/driver.h>
+#ifndef MLX5_IB_SRQ_H
+#define MLX5_IB_SRQ_H
 
 enum {
 	MLX5_SRQ_FLAG_ERR    = (1 << 0),

@@ -37,9 +35,12 @@ struct mlx5_srq_attr {
 	u16 uid;
 };
 
-struct mlx5_core_dev;
-
-void mlx5_init_srq_table(struct mlx5_core_dev *dev);
-void mlx5_cleanup_srq_table(struct mlx5_core_dev *dev);
-
-#endif /* MLX5_SRQ_H */
+int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
+			 struct mlx5_srq_attr *in);
+int mlx5_core_destroy_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq);
+int mlx5_core_query_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
+			struct mlx5_srq_attr *out);
+int mlx5_core_arm_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
+		      u16 lwm, int is_srq);
+struct mlx5_core_srq *mlx5_core_get_srq(struct mlx5_core_dev *dev, u32 srqn);
+#endif /* MLX5_IB_SRQ_H */

drivers/infiniband/hw/mlx5/srq_cmd.c renamed from drivers/net/ethernet/mellanox/mlx5/core/srq.c +33 ~~-64~~

138 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/srq.c +0 ~~-634~~

648 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/transobj.c +0 ~~-43~~

49 lines not on this page · raw message

include/linux/mlx5/driver.h +0 -8

index 27a481b159ed..1096da4fb368 100644
@@ -904,13 +904,6 @@ struct mlx5_cmd_mailbox *mlx5_alloc_cmd_mailbox_chain(struct mlx5_core_dev *dev,
 						      gfp_t flags, int npages);
 void mlx5_free_cmd_mailbox_chain(struct mlx5_core_dev *dev,
 				 struct mlx5_cmd_mailbox *head);
-int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-			 struct mlx5_srq_attr *in);
-int mlx5_core_destroy_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq);
-int mlx5_core_query_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-			struct mlx5_srq_attr *out);
-int mlx5_core_arm_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-		      u16 lwm, int is_srq);
 void mlx5_init_mkey_table(struct mlx5_core_dev *dev);
 void mlx5_cleanup_mkey_table(struct mlx5_core_dev *dev);
 int mlx5_core_create_mkey_cb(struct mlx5_core_dev *dev,

@@ -942,7 +935,6 @@ void mlx5_unregister_debugfs(void);
 
 void mlx5_fill_page_array(struct mlx5_frag_buf *buf, __be64 *pas);
 void mlx5_fill_page_frag_array(struct mlx5_frag_buf *frag_buf, __be64 *pas);
-struct mlx5_core_srq *mlx5_core_get_srq(struct mlx5_core_dev *dev, u32 srqn);
 int mlx5_vector2eqn(struct mlx5_core_dev *dev, int vector, int *eqn,
 		    unsigned int *irqn);
 int mlx5_core_attach_mcg(struct mlx5_core_dev *dev, union ib_gid *mgid, u32 qpn);

include/linux/mlx5/srq.h +0 ~~-31~~

index 77bc4264066d..9343306cd188 100644
@@ -6,37 +6,6 @@
 #ifndef MLX5_SRQ_H
 #define MLX5_SRQ_H
 
-#include <linux/mlx5/driver.h>
-
-enum {
-	MLX5_SRQ_FLAG_ERR    = (1 << 0),
-	MLX5_SRQ_FLAG_WQ_SIG = (1 << 1),
-	MLX5_SRQ_FLAG_RNDV   = (1 << 2),
-};
-
-struct mlx5_srq_attr {
-	u32 type;
-	u32 flags;
-	u32 log_size;
-	u32 wqe_shift;
-	u32 log_page_size;
-	u32 wqe_cnt;
-	u32 srqn;
-	u32 xrcd;
-	u32 page_offset;
-	u32 cqn;
-	u32 pd;
-	u32 lwm;
-	u32 user_index;
-	u64 db_record;
-	__be64 *pas;
-	u32 tm_log_list_size;
-	u32 tm_next_tag;
-	u32 tm_hw_phase_cnt;
-	u32 tm_sw_phase_cnt;
-	u16 uid;
-};
-
 struct mlx5_core_dev;
 
 void mlx5_init_srq_table(struct mlx5_core_dev *dev);

include/linux/mlx5/transobj.h +0 -6

index 39ebb699875b..a261d5528ff7 100644
@@ -58,12 +58,6 @@ int mlx5_core_create_tis(struct mlx5_core_dev *dev, u32 *in, int inlen,
 int mlx5_core_modify_tis(struct mlx5_core_dev *dev, u32 tisn, u32 *in,
 			 int inlen);
 void mlx5_core_destroy_tis(struct mlx5_core_dev *dev, u32 tisn);
-int mlx5_core_create_rmp(struct mlx5_core_dev *dev, u32 *in, int inlen,
-			 u32 *rmpn);
-int mlx5_core_modify_rmp(struct mlx5_core_dev *dev, u32 *in, int inlen);
-int mlx5_core_destroy_rmp(struct mlx5_core_dev *dev, u32 rmpn);
-int mlx5_core_query_rmp(struct mlx5_core_dev *dev, u32 rmpn, u32 *out);
-
 int mlx5_core_create_rqt(struct mlx5_core_dev *dev, u32 *in, int inlen,
 			 u32 *rqtn);
 int mlx5_core_modify_rqt(struct mlx5_core_dev *dev, u32 rqtn, u32 *in,

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #6 lore raw

PATCH 6/11 RDMA/mlx5: Remove SRQ signature global flag

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

SRQ signature is not supported, hence no need for special static global variable to announce it.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/srq.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

drivers/infiniband/hw/mlx5/srq.c +1 -4

index a86d9f153805..154fd0d3718e 100644
@@ -12,9 +12,6 @@
 #include "mlx5_ib.h"
 #include "srq.h"
 
-/* not supported currently */
-static int srq_signature;
-
 static void *get_wqe(struct mlx5_ib_srq *srq, int n)
 {
 	return mlx5_buf_offset(&srq->buf, n << srq->msrq.wqe_shift);

@@ -175,7 +172,7 @@ static int create_srq_kernel(struct mlx5_ib_dev *dev, struct mlx5_ib_srq *srq,
 		err = -ENOMEM;
 		goto err_in;
 	}
-	srq->wq_sig = !!srq_signature;
+	srq->wq_sig = 0;
 
 	in->log_page_size = srq->buf.page_shift - MLX5_ADAPTER_PAGE_SHIFT;
 	if (MLX5_CAP_GEN(dev->mdev, cqe_version) == MLX5_CQE_VERSION_V1 &&

Leon Romanovsky

author

2018-11-28 18:53 UTC #7 lore raw

PATCH 7/11 RDMA/mlx5: Use stages for callback to setup and release DEVX

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Reuse existing infrastructure to initialize and release DEVX uid. The DevX interface is intended for user space access, so it is supposed to be initialized before ib_register_device(). Also it isn't supported in switchdev mode and don't need to initialize it in that mode.

Fixes 76dc5a8406bf ("IB/mlx5: Manage device uid for DEVX white list commands")

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/main.c    | 26 +++++++++++++++++++-------
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
 2 files changed, 20 insertions(+), 7 deletions(-)

drivers/infiniband/hw/mlx5/main.c +19 -7

index b3d9af0cafc2..b15f1a70d33b 100644
@@ -6218,6 +6218,22 @@ static void mlx5_ib_stage_dev_notifier_cleanup(struct mlx5_ib_dev *dev)
 	mlx5_notifier_unregister(dev->mdev, &dev->mdev_events);
 }
 
+static int mlx5_ib_stage_devx_init(struct mlx5_ib_dev *dev)
+{
+	int uid;
+
+	uid = mlx5_ib_devx_create(dev);
+	if (uid > 0)
+		dev->devx_whitelist_uid = uid;
+
+	return 0;
+}
+static void mlx5_ib_stage_devx_cleanup(struct mlx5_ib_dev *dev)
+{
+	if (dev->devx_whitelist_uid)
+		mlx5_ib_devx_destroy(dev, dev->devx_whitelist_uid);
+}
+
 void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 		      const struct mlx5_ib_profile *profile,
 		      int stage)

@@ -6229,8 +6245,6 @@ void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
 			profile->stage[stage].cleanup(dev);
 	}
 
-	if (dev->devx_whitelist_uid)
-		mlx5_ib_devx_destroy(dev, dev->devx_whitelist_uid);
 	ib_dealloc_device((struct ib_device *)dev);
 }

@@ -6239,7 +6253,6 @@ void *__mlx5_ib_add(struct mlx5_ib_dev *dev,
 {
 	int err;
 	int i;
-	int uid;
 
 	for (i = 0; i < MLX5_IB_STAGE_MAX; i++) {
 		if (profile->stage[i].init) {

@@ -6249,10 +6262,6 @@ void *__mlx5_ib_add(struct mlx5_ib_dev *dev,
 		}
 	}
 
-	uid = mlx5_ib_devx_create(dev);
-	if (uid > 0)
-		dev->devx_whitelist_uid = uid;
-
 	dev->profile = profile;
 	dev->ib_active = true;

@@ -6304,6 +6313,9 @@ static const struct mlx5_ib_profile pf_profile = {
 	STAGE_CREATE(MLX5_IB_STAGE_PRE_IB_REG_UMR,
 		     NULL,
 		     mlx5_ib_stage_pre_ib_reg_umr_cleanup),
+	STAGE_CREATE(MLX5_IB_STAGE_WHITELIST_UID,
+		     mlx5_ib_stage_devx_init,
+		     mlx5_ib_stage_devx_cleanup),
 	STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
 		     mlx5_ib_stage_ib_reg_init,
 		     mlx5_ib_stage_ib_reg_cleanup),

drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0

index 24af6fbb43e7..fe34129bb780 100644
@@ -782,6 +782,7 @@ enum mlx5_ib_stages {
 	MLX5_IB_STAGE_UAR,
 	MLX5_IB_STAGE_BFREG,
 	MLX5_IB_STAGE_PRE_IB_REG_UMR,
+	MLX5_IB_STAGE_WHITELIST_UID,
 	MLX5_IB_STAGE_IB_REG,
 	MLX5_IB_STAGE_POST_IB_REG_UMR,
 	MLX5_IB_STAGE_DELAY_DROP,

Leon Romanovsky

author

2018-11-28 18:53 UTC #8 lore raw

PATCH 8/11 RDMA/mlx5: Update SRQ functions signatures to mlx5_ib format

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Reflect the change of moving SRQ code from mlx5_core to mlx5_ib by updating function signatures do not require mlx5_core_dev as an input, because all operations in mlx5_ib are supposed to use mlx5_ib_dev.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/cq.c      |   3 +-
 drivers/infiniband/hw/mlx5/srq.c     |  10 +--
 drivers/infiniband/hw/mlx5/srq.h     |  18 ++--
 drivers/infiniband/hw/mlx5/srq_cmd.c | 129 ++++++++++++++-------------
 4 files changed, 83 insertions(+), 77 deletions(-)

drivers/infiniband/hw/mlx5/cq.c +1 -2

index a91580edca92..26ab9041f94a 100644
@@ -178,8 +178,7 @@ static void handle_responder(struct ib_wc *wc, struct mlx5_cqe64 *cqe,
 		struct mlx5_core_srq *msrq = NULL;
 
 		if (qp->ibqp.xrcd) {
-			msrq = mlx5_core_get_srq(dev->mdev,
-						 be32_to_cpu(cqe->srqn));
+			msrq = mlx5_cmd_get_srq(dev, be32_to_cpu(cqe->srqn));
 			srq = to_mibsrq(msrq);
 		} else {
 			srq = to_msrq(qp->ibqp.srq);

drivers/infiniband/hw/mlx5/srq.c +5 -5

index 154fd0d3718e..2b184c7f531a 100644
@@ -297,7 +297,7 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
 
 	in.pd = to_mpd(pd)->pdn;
 	in.db_record = srq->db.dma;
-	err = mlx5_core_create_srq(dev->mdev, &srq->msrq, &in);
+	err = mlx5_cmd_create_srq(dev, &srq->msrq, &in);
 	kvfree(in.pas);
 	if (err) {
 		mlx5_ib_dbg(dev, "create SRQ failed, err %d\n", err);

@@ -321,7 +321,7 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
 	return &srq->ibsrq;
 
 err_core:
-	mlx5_core_destroy_srq(dev->mdev, &srq->msrq);
+	mlx5_cmd_destroy_srq(dev, &srq->msrq);
 
 err_usr_kern_srq:
 	if (pd->uobject)

@@ -351,7 +351,7 @@ int mlx5_ib_modify_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr,
 			return -EINVAL;
 
 		mutex_lock(&srq->mutex);
-		ret = mlx5_core_arm_srq(dev->mdev, &srq->msrq, attr->srq_limit, 1);
+		ret = mlx5_cmd_arm_srq(dev, &srq->msrq, attr->srq_limit, 1);
 		mutex_unlock(&srq->mutex);
 
 		if (ret)

@@ -372,7 +372,7 @@ int mlx5_ib_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *srq_attr)
 	if (!out)
 		return -ENOMEM;
 
-	ret = mlx5_core_query_srq(dev->mdev, &srq->msrq, out);
+	ret = mlx5_cmd_query_srq(dev, &srq->msrq, out);
 	if (ret)
 		goto out_box;

@@ -390,7 +390,7 @@ int mlx5_ib_destroy_srq(struct ib_srq *srq)
 	struct mlx5_ib_dev *dev = to_mdev(srq->device);
 	struct mlx5_ib_srq *msrq = to_msrq(srq);
 
-	mlx5_core_destroy_srq(dev->mdev, &msrq->msrq);
+	mlx5_cmd_destroy_srq(dev, &msrq->msrq);
 
 	if (srq->uobject) {
 		mlx5_ib_db_unmap_user(to_mucontext(srq->uobject->context), &msrq->db);

drivers/infiniband/hw/mlx5/srq.h +10 -8

index f23d5de12973..1110aeaa775e 100644
@@ -35,12 +35,14 @@ struct mlx5_srq_attr {
 	u16 uid;
 };
 
-int mlx5_core_create_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-			 struct mlx5_srq_attr *in);
-int mlx5_core_destroy_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq);
-int mlx5_core_query_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-			struct mlx5_srq_attr *out);
-int mlx5_core_arm_srq(struct mlx5_core_dev *dev, struct mlx5_core_srq *srq,
-		      u16 lwm, int is_srq);
-struct mlx5_core_srq *mlx5_core_get_srq(struct mlx5_core_dev *dev, u32 srqn);
+struct mlx5_ib_dev;
+
+int mlx5_cmd_create_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
+			struct mlx5_srq_attr *in);
+int mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq);
+int mlx5_cmd_query_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
+		       struct mlx5_srq_attr *out);
+int mlx5_cmd_arm_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
+		     u16 lwm, int is_srq);
+struct mlx5_core_srq *mlx5_cmd_get_srq(struct mlx5_ib_dev *dev, u32 srqn);
 #endif /* MLX5_IB_SRQ_H */

drivers/infiniband/hw/mlx5/srq_cmd.c +67 ~~-62~~

365 lines not on this page · raw message

Leon Romanovsky

author

2018-11-28 18:53 UTC #9 lore raw

PATCH 9/11 RDMA/mlx5: Initialize SRQ tables on mlx5_ib

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Transfer initialization and cleanup from mlx5_priv struct of mlx5_core_dev to be part of mlx5_ib_dev. This completes removal of SRQ from mlx5_core.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/ib_rep.c           |  4 ++
 drivers/infiniband/hw/mlx5/main.c             |  7 ++
 drivers/infiniband/hw/mlx5/mlx5_ib.h          |  5 +-
 drivers/infiniband/hw/mlx5/srq.c              |  1 -
 drivers/infiniband/hw/mlx5/srq.h              | 25 +++++++
 drivers/infiniband/hw/mlx5/srq_cmd.c          | 72 +++++++++++++++----
 .../net/ethernet/mellanox/mlx5/core/Makefile  |  2 +-
 .../net/ethernet/mellanox/mlx5/core/main.c    |  5 --
 drivers/net/ethernet/mellanox/mlx5/core/srq.c | 63 ----------------
 include/linux/mlx5/driver.h                   | 25 -------
 include/linux/mlx5/srq.h                      | 14 ----
 11 files changed, 101 insertions(+), 122 deletions(-)
 delete mode 100644 drivers/net/ethernet/mellanox/mlx5/core/srq.c
 delete mode 100644 include/linux/mlx5/srq.h

drivers/infiniband/hw/mlx5/ib_rep.c +4 -0

index 584ff2ea7810..8a682d86d634 100644
@@ -4,6 +4,7 @@
  */
 
 #include "ib_rep.h"
+#include "srq.h"
 
 static const struct mlx5_ib_profile rep_profile = {
 	STAGE_CREATE(MLX5_IB_STAGE_INIT,

@@ -21,6 +22,9 @@ static const struct mlx5_ib_profile rep_profile = {
 	STAGE_CREATE(MLX5_IB_STAGE_ROCE,
 		     mlx5_ib_stage_rep_roce_init,
 		     mlx5_ib_stage_rep_roce_cleanup),
+	STAGE_CREATE(MLX5_IB_STAGE_SRQ,
+		     mlx5_init_srq_table,
+		     mlx5_cleanup_srq_table),
 	STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
 		     mlx5_ib_stage_dev_res_init,
 		     mlx5_ib_stage_dev_res_cleanup),

drivers/infiniband/hw/mlx5/main.c +7 -0

index b15f1a70d33b..24a0957ae630 100644
@@ -60,6 +60,7 @@
 #include "mlx5_ib.h"
 #include "ib_rep.h"
 #include "cmd.h"
+#include "srq.h"
 #include <linux/mlx5/fs_helpers.h>
 #include <linux/mlx5/accel.h>
 #include <rdma/uverbs_std_types.h>

@@ -6289,6 +6290,9 @@ static const struct mlx5_ib_profile pf_profile = {
 	STAGE_CREATE(MLX5_IB_STAGE_ROCE,
 		     mlx5_ib_stage_roce_init,
 		     mlx5_ib_stage_roce_cleanup),
+	STAGE_CREATE(MLX5_IB_STAGE_SRQ,
+		     mlx5_init_srq_table,
+		     mlx5_cleanup_srq_table),
 	STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
 		     mlx5_ib_stage_dev_res_init,
 		     mlx5_ib_stage_dev_res_cleanup),

@@ -6343,6 +6347,9 @@ static const struct mlx5_ib_profile nic_rep_profile = {
 	STAGE_CREATE(MLX5_IB_STAGE_ROCE,
 		     mlx5_ib_stage_rep_roce_init,
 		     mlx5_ib_stage_rep_roce_cleanup),
+	STAGE_CREATE(MLX5_IB_STAGE_SRQ,
+		     mlx5_init_srq_table,
+		     mlx5_cleanup_srq_table),
 	STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
 		     mlx5_ib_stage_dev_res_init,
 		     mlx5_ib_stage_dev_res_cleanup),

drivers/infiniband/hw/mlx5/mlx5_ib.h +4 -1

index fe34129bb780..4ed9f76b49de 100644
@@ -41,7 +41,6 @@
 #include <linux/mlx5/cq.h>
 #include <linux/mlx5/fs.h>
 #include <linux/mlx5/qp.h>
-#include <linux/mlx5/srq.h>
 #include <linux/mlx5/fs.h>
 #include <linux/types.h>
 #include <linux/mlx5/transobj.h>

@@ -50,6 +49,8 @@
 #include <rdma/uverbs_ioctl.h>
 #include <rdma/mlx5_user_ioctl_cmds.h>
 
+#include "srq.h"
+
 #define mlx5_ib_dbg(_dev, format, arg...)                                      \
 	dev_dbg(&(_dev)->ib_dev.dev, "%s:%d:(pid %d): " format, __func__,      \
 		__LINE__, current->pid, ##arg)

@@ -774,6 +775,7 @@ enum mlx5_ib_stages {
 	MLX5_IB_STAGE_CAPS,
 	MLX5_IB_STAGE_NON_DEFAULT_CB,
 	MLX5_IB_STAGE_ROCE,
+	MLX5_IB_STAGE_SRQ,
 	MLX5_IB_STAGE_DEVICE_RESOURCES,
 	MLX5_IB_STAGE_DEVICE_NOTIFIER,
 	MLX5_IB_STAGE_ODP,

@@ -940,6 +942,7 @@ struct mlx5_ib_dev {
 	u64			sys_image_guid;
 	struct mlx5_memic	memic;
 	u16			devx_whitelist_uid;
+	struct mlx5_srq_table   srq_table;
 };
 
 static inline struct mlx5_ib_cq *to_mibcq(struct mlx5_core_cq *mcq)

drivers/infiniband/hw/mlx5/srq.c +0 -1

index 2b184c7f531a..91dcd3918d96 100644
@@ -5,7 +5,6 @@
 
 #include <linux/module.h>
 #include <linux/mlx5/qp.h>
-#include <linux/mlx5/srq.h>
 #include <linux/slab.h>
 #include <rdma/ib_umem.h>
 #include <rdma/ib_user_verbs.h>

drivers/infiniband/hw/mlx5/srq.h +25 -0

index 1110aeaa775e..75eb5839ae95 100644
@@ -37,6 +37,28 @@ struct mlx5_srq_attr {
 
 struct mlx5_ib_dev;
 
+struct mlx5_core_srq {
+	struct mlx5_core_rsc_common common; /* must be first */
+	u32 srqn;
+	int max;
+	size_t max_gs;
+	size_t max_avail_gather;
+	int wqe_shift;
+	void (*event)(struct mlx5_core_srq *srq, enum mlx5_event e);
+
+	atomic_t refcount;
+	struct completion free;
+	u16 uid;
+};
+
+struct mlx5_srq_table {
+	struct notifier_block nb;
+	/* protect radix tree
+	 */
+	spinlock_t lock;
+	struct radix_tree_root tree;
+};
+
 int mlx5_cmd_create_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 			struct mlx5_srq_attr *in);
 int mlx5_cmd_destroy_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq);

@@ -45,4 +67,7 @@ int mlx5_cmd_query_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 int mlx5_cmd_arm_srq(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 		     u16 lwm, int is_srq);
 struct mlx5_core_srq *mlx5_cmd_get_srq(struct mlx5_ib_dev *dev, u32 srqn);
+
+int mlx5_init_srq_table(struct mlx5_ib_dev *dev);
+void mlx5_cleanup_srq_table(struct mlx5_ib_dev *dev);
 #endif /* MLX5_IB_SRQ_H */

drivers/infiniband/hw/mlx5/srq_cmd.c +60 ~~-12~~

100 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 -1

index 26afe0779a0c..d499b3d00348 100644
@@ -12,7 +12,7 @@ obj-$(CONFIG_MLX5_CORE) += mlx5_core.o
 # mlx5 core basic
 #
 mlx5_core-y :=	main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \
-		health.o mcg.o cq.o srq.o alloc.o qp.o port.o mr.o pd.o \
+		health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \
 		mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o \
 		fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \
 		diag/fs_tracepoint.o diag/fw_tracer.o

drivers/net/ethernet/mellanox/mlx5/core/main.c +0 -5

index 4bc27a073dc4..778995573812 100644
@@ -43,7 +43,6 @@
 #include <linux/mlx5/driver.h>
 #include <linux/mlx5/cq.h>
 #include <linux/mlx5/qp.h>
-#include <linux/mlx5/srq.h>
 #include <linux/debugfs.h>
 #include <linux/kmod.h>
 #include <linux/mlx5/mlx5_ifc.h>

@@ -749,8 +748,6 @@ static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
 
 	mlx5_init_qp_table(dev);
 
-	mlx5_init_srq_table(dev);
-
 	mlx5_init_mkey_table(dev);
 
 	mlx5_init_reserved_gids(dev);

@@ -804,7 +801,6 @@ static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
 err_tables_cleanup:
 	mlx5_vxlan_destroy(dev->vxlan);
 	mlx5_cleanup_mkey_table(dev);
-	mlx5_cleanup_srq_table(dev);
 	mlx5_cleanup_qp_table(dev);
 	mlx5_cq_debugfs_cleanup(dev);
 err_events_cleanup:

@@ -828,7 +824,6 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev)
 	mlx5_cleanup_clock(dev);
 	mlx5_cleanup_reserved_gids(dev);
 	mlx5_cleanup_mkey_table(dev);
-	mlx5_cleanup_srq_table(dev);
 	mlx5_cleanup_qp_table(dev);
 	mlx5_cq_debugfs_cleanup(dev);
 	mlx5_events_cleanup(dev);

drivers/net/ethernet/mellanox/mlx5/core/srq.c deleted +0 ~~-63~~

63 lines not on this page · raw message

include/linux/mlx5/driver.h +0 ~~-25~~

index 1096da4fb368..584d8a5df7eb 100644
@@ -50,7 +50,6 @@
 
 #include <linux/mlx5/device.h>
 #include <linux/mlx5/doorbell.h>
-#include <linux/mlx5/srq.h>
 #include <linux/mlx5/eq.h>
 #include <linux/timecounter.h>
 #include <linux/ptp_clock_kernel.h>

@@ -393,20 +392,6 @@ struct mlx5_core_rsc_common {
 	struct completion	free;
 };
 
-struct mlx5_core_srq {
-	struct mlx5_core_rsc_common	common; /* must be first */
-	u32		srqn;
-	int		max;
-	size_t		max_gs;
-	size_t		max_avail_gather;
-	int		wqe_shift;
-	void (*event)	(struct mlx5_core_srq *, enum mlx5_event);
-
-	atomic_t		refcount;
-	struct completion	free;
-	u16		uid;
-};
-
 struct mlx5_uars_page {
 	void __iomem	       *map;
 	bool			wc;

@@ -464,14 +449,6 @@ struct mlx5_qp_table {
 	struct radix_tree_root	tree;
 };
 
-struct mlx5_srq_table {
-	struct notifier_block   nb;
-	/* protect radix tree
-	 */
-	spinlock_t		lock;
-	struct radix_tree_root	tree;
-};
-
 struct mlx5_mkey_table {
 	/* protect radix tree
 	 */

@@ -547,8 +524,6 @@ struct mlx5_priv {
 
 	struct mlx5_core_health health;
 
-	struct mlx5_srq_table	srq_table;
-
 	/* start: qp staff */
 	struct mlx5_qp_table	qp_table;
 	struct dentry	       *qp_debugfs;

include/linux/mlx5/srq.h deleted +0 ~~-14~~

deleted file mode 100644
index 9343306cd188..000000000000
@@ -1,14 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
-/*
- * Copyright (c) 2013-2018, Mellanox Technologies. All rights reserved.
- */
-
-#ifndef MLX5_SRQ_H
-#define MLX5_SRQ_H
-
-struct mlx5_core_dev;
-
-void mlx5_init_srq_table(struct mlx5_core_dev *dev);
-void mlx5_cleanup_srq_table(struct mlx5_core_dev *dev);
-
-#endif /* MLX5_SRQ_H */

Signature · 1 line

2.19.1

Leon Romanovsky

author

2018-11-28 18:53 UTC #10 lore raw

PATCH 10/11 RDMA/mlx5: Unfold create RMP function

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

There is no need to perform create_rmp in two separate function, while one of them uses stack as a placeholder for data while other allocates it dynamically. Combine those two functions to one instead of two.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/srq_cmd.c | 35 +++++++++++++---------------
 1 file changed, 16 insertions(+), 19 deletions(-)

drivers/infiniband/hw/mlx5/srq_cmd.c +16 ~~-19~~

index 6be89c6be40f..955df638b416 100644
@@ -298,20 +298,6 @@ static int query_xrc_srq_cmd(struct mlx5_ib_dev *dev,
 	return err;
 }
 
-static int mlx5_core_create_rmp(struct mlx5_ib_dev *dev, u32 *in, int inlen,
-				u32 *rmpn)
-{
-	u32 out[MLX5_ST_SZ_DW(create_rmp_out)] = { 0 };
-	int err;
-
-	MLX5_SET(create_rmp_in, in, opcode, MLX5_CMD_OP_CREATE_RMP);
-	err = mlx5_cmd_exec(dev->mdev, in, inlen, out, sizeof(out));
-	if (!err)
-		*rmpn = MLX5_GET(create_rmp_out, out, rmpn);
-
-	return err;
-}
-
 static int mlx5_core_modify_rmp(struct mlx5_ib_dev *dev, u32 *in, int inlen)
 {
 	u32 out[MLX5_ST_SZ_DW(modify_rmp_out)] = {0};

@@ -333,18 +319,24 @@ static int mlx5_core_query_rmp(struct mlx5_ib_dev *dev, u32 rmpn, u32 *out)
 static int create_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 			  struct mlx5_srq_attr *in)
 {
-	void *create_in;
+	void *create_out = NULL;
+	void *create_in = NULL;
 	void *rmpc;
 	void *wq;
 	int pas_size;
+	int outlen;
 	int inlen;
 	int err;
 
 	pas_size = get_pas_size(in);
 	inlen = MLX5_ST_SZ_BYTES(create_rmp_in) + pas_size;
+	outlen = MLX5_ST_SZ_BYTES(create_rmp_out);
 	create_in = kvzalloc(inlen, GFP_KERNEL);
-	if (!create_in)
-		return -ENOMEM;
+	create_out = kvzalloc(outlen, GFP_KERNEL);
+	if (!create_in || !create_out) {
+		err = -ENOMEM;
+		goto out;
+	}
 
 	rmpc = MLX5_ADDR_OF(create_rmp_in, create_in, ctx);
 	wq = MLX5_ADDR_OF(rmpc, rmpc, wq);

@@ -354,11 +346,16 @@ static int create_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 	set_wq(wq, in);
 	memcpy(MLX5_ADDR_OF(rmpc, rmpc, wq.pas), in->pas, pas_size);
 
-	err = mlx5_core_create_rmp(dev, create_in, inlen, &srq->srqn);
-	if (!err)
+	MLX5_SET(create_rmp_in, create_in, opcode, MLX5_CMD_OP_CREATE_RMP);
+	err = mlx5_cmd_exec(dev->mdev, create_in, inlen, create_out, outlen);
+	if (!err) {
+		srq->srqn = MLX5_GET(create_rmp_out, create_out, rmpn);
 		srq->uid = in->uid;
+	}
 
+out:
 	kvfree(create_in);
+	kvfree(create_out);
 	return err;
 }

Leon Romanovsky

author

2018-11-28 18:53 UTC #11 lore raw

PATCH 11/11 RDMA/mlx5: Unfold modify RMP function

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

There is no need to perform modify_rmp in two separate function, while one of them uses stack as a placeholder for data while other allocates it dynamically. Combine those two functions to one call instead of two.

Reviewed-by Mark Bloch <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/srq_cmd.c | 62 +++++++++++++++-------------
 1 file changed, 34 insertions(+), 28 deletions(-)

drivers/infiniband/hw/mlx5/srq_cmd.c +34 ~~-28~~

index 955df638b416..7aaaffbd4afa 100644
@@ -298,24 +298,6 @@ static int query_xrc_srq_cmd(struct mlx5_ib_dev *dev,
 	return err;
 }
 
-static int mlx5_core_modify_rmp(struct mlx5_ib_dev *dev, u32 *in, int inlen)
-{
-	u32 out[MLX5_ST_SZ_DW(modify_rmp_out)] = {0};
-
-	MLX5_SET(modify_rmp_in, in, opcode, MLX5_CMD_OP_MODIFY_RMP);
-	return mlx5_cmd_exec(dev->mdev, in, inlen, out, sizeof(out));
-}
-
-static int mlx5_core_query_rmp(struct mlx5_ib_dev *dev, u32 rmpn, u32 *out)
-{
-	u32 in[MLX5_ST_SZ_DW(query_rmp_in)] = {0};
-	int outlen = MLX5_ST_SZ_BYTES(query_rmp_out);
-
-	MLX5_SET(query_rmp_in, in, opcode, MLX5_CMD_OP_QUERY_RMP);
-	MLX5_SET(query_rmp_in, in, rmpn,   rmpn);
-	return mlx5_cmd_exec(dev->mdev, in, sizeof(in), out, outlen);
-}
-
 static int create_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 			  struct mlx5_srq_attr *in)
 {

@@ -373,15 +355,24 @@ static int destroy_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq)
 static int arm_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 		       u16 lwm)
 {
-	void *in;
+	void *out = NULL;
+	void *in = NULL;
 	void *rmpc;
 	void *wq;
 	void *bitmask;
+	int outlen;
+	int inlen;
 	int err;
 
-	in = kvzalloc(MLX5_ST_SZ_BYTES(modify_rmp_in), GFP_KERNEL);
-	if (!in)
-		return -ENOMEM;
+	inlen = MLX5_ST_SZ_BYTES(modify_rmp_in);
+	outlen = MLX5_ST_SZ_BYTES(modify_rmp_out);
+
+	in = kvzalloc(inlen, GFP_KERNEL);
+	out = kvzalloc(outlen, GFP_KERNEL);
+	if (!in || !out) {
+		err = -ENOMEM;
+		goto out;
+	}
 
 	rmpc =	  MLX5_ADDR_OF(modify_rmp_in,   in,   ctx);
 	bitmask = MLX5_ADDR_OF(modify_rmp_in,   in,   bitmask);

@@ -393,25 +384,39 @@ static int arm_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 	MLX5_SET(wq,		wq,	 lwm,	    lwm);
 	MLX5_SET(rmp_bitmask,	bitmask, lwm,	    1);
 	MLX5_SET(rmpc, rmpc, state, MLX5_RMPC_STATE_RDY);
+	MLX5_SET(modify_rmp_in, in, opcode, MLX5_CMD_OP_MODIFY_RMP);
 
-	err = mlx5_core_modify_rmp(dev, in, MLX5_ST_SZ_BYTES(modify_rmp_in));
+	err = mlx5_cmd_exec(dev->mdev, in, inlen, out, outlen);
 
+out:
 	kvfree(in);
+	kvfree(out);
 	return err;
 }
 
 static int query_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 			 struct mlx5_srq_attr *out)
 {
-	u32 *rmp_out;
+	u32 *rmp_out = NULL;
+	u32 *rmp_in = NULL;
 	void *rmpc;
+	int outlen;
+	int inlen;
 	int err;
 
-	rmp_out =  kvzalloc(MLX5_ST_SZ_BYTES(query_rmp_out), GFP_KERNEL);
-	if (!rmp_out)
-		return -ENOMEM;
+	outlen = MLX5_ST_SZ_BYTES(query_rmp_out);
+	inlen = MLX5_ST_SZ_BYTES(query_rmp_in);
+
+	rmp_out = kvzalloc(outlen, GFP_KERNEL);
+	rmp_in = kvzalloc(inlen, GFP_KERNEL);
+	if (!rmp_out || !rmp_in) {
+		err = -ENOMEM;
+		goto out;
+	}
 
-	err = mlx5_core_query_rmp(dev, srq->srqn, rmp_out);
+	MLX5_SET(query_rmp_in, rmp_in, opcode, MLX5_CMD_OP_QUERY_RMP);
+	MLX5_SET(query_rmp_in, rmp_in, rmpn,   srq->srqn);
+	err = mlx5_cmd_exec(dev->mdev, rmp_in, inlen, rmp_out, outlen);
 	if (err)
 		goto out;

@@ -422,6 +427,7 @@ static int query_rmp_cmd(struct mlx5_ib_dev *dev, struct mlx5_core_srq *srq,
 
 out:
 	kvfree(rmp_out);
+	kvfree(rmp_in);
 	return err;
 }

Signature · 1 line

2.19.1

Leon Romanovsky

Applied

2018-12-04 07:29 UTC #12 lore raw

In reply to Leon Romanovsky

Quoted 22 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>
Hi Doug and Jason,
This series is intended to be applied on mlx5-next after Saeed's infrastructure changes series are accepted:
https://patchwork.kernel.org/project/linux-rdma/list/?series=47809
----------------------------------------------------------------------
The following patchset brings mlx5 SRQ functionality to the module which is actually using it - mlx5_ib. The original code being not used in mlx5_core/mlx5_en, but located there, caused to unneeded maintenance overhead.
By moving mlx5 SRQ code to mlx5_ib, we successfully reduced ~200 LOC codes and minimized exports and headers pollution.
The proposed patches are divided to three logical groups: move from netdev, fix IB part and cleanup.
Thanks

Quoted 13 lines

Leon Romanovsky (11):
net/mlx5: Align SRQ licenses and copyright information net/mlx5: Remove dead transobj code
net/mlx5: Remove not-used lib/eq.h header file net/mlx5: Remove references to local mlx5_core functions net/mlx5: Move SRQ functions to RDMA part RDMA/mlx5: Remove SRQ signature global flag RDMA/mlx5: Use stages for callback to setup and release DEVX RDMA/mlx5: Update SRQ functions signatures to mlx5_ib format RDMA/mlx5: Initialize SRQ tables on mlx5_ib RDMA/mlx5: Unfold create RMP function
RDMA/mlx5: Unfold modify RMP function

Applied to mlx5-next.

Thanks