Skip to contentSkip to conversation

New hfi1 feature: Accelerated IP

PATCH Applied Dennis Dalessandro 5 participants 29 messages May 11, 2020 – Jun 17, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover New hfi1 feature: Accelerated IP

This patch series is an accelerated ipoib using the rdma netdev mechanism already present in ipoib. A new device capability bit, IB_DEVICE_RDMA_NETDEV_OPA, triggers ipoib to create a datagram QP using the IB_QP_CREATE_NETDEV_USE.

The highlights include:
- Sharing send and receive resources with VNIC
- Allows for switching between connected mode and datagram mode
- Increases the maximum datagram MTU for opa devices to 10k

The same spreading capability exploited by VNIC is used here to vary the receive context that receives the packet.

The patches are fully bisectable and stepwise implement the capability.

changes since v2
----------------
*Rebased ontop of latest rdma/for-next

Changes since v1
----------------
*Fix incorrect parameter to xa_find() in patch 9 *Address Erez comments and try to hide opa from ipoib in patch 7 *Fix some typos in RB lines

---

Gary Leshner (6):
IB/hfi1: Add functions to transmit datagram ipoib packets IB/hfi1: Add the transmit side of a datagram ipoib RDMA netdev IB/hfi1: Remove module parameter for KDETH qpns IB/{rdmavt,hfi1}: Implement creation of accelerated UD QPs IB/{hfi1,ipoib,rdma}: Broadcast ping sent packets which exceeded mtu size IB/ipoib: Add capability to switch between datagram and connected mode

Grzegorz Andrejczuk (6):
IB/hfi1: RSM rules for AIP
IB/hfi1: Rename num_vnic_contexts as num_netdev_contexts IB/hfi1: Add interrupt handler functions for accelerated ipoib IB/hfi1: Add rx functions for dummy netdev
IB/hfi1: Activate the dummy netdev
IB/hfi1: Add packet histogram trace event

Kaike Wan (3):
IB/hfi1: Add accelerated IP capability bit IB/ipoib: Increase ipoib Datagram mode MTU's upper limit IB/hfi1: Add functions to receive accelerated ipoib packets

Piotr Stankiewicz (1):
IB/hfi1: Enable the transmit side of the datagram ipoib netdev

 drivers/infiniband/hw/hfi1/Makefile            |    4 
 drivers/infiniband/hw/hfi1/affinity.c          |   12 
 drivers/infiniband/hw/hfi1/affinity.h          |    3 
 drivers/infiniband/hw/hfi1/chip.c              |  303 ++++++---
 drivers/infiniband/hw/hfi1/chip.h              |    5 
 drivers/infiniband/hw/hfi1/common.h            |   13 
 drivers/infiniband/hw/hfi1/driver.c            |  231 ++++++-
 drivers/infiniband/hw/hfi1/file_ops.c          |    4 
 drivers/infiniband/hw/hfi1/hfi.h               |   38 -
 drivers/infiniband/hw/hfi1/init.c              |   13 
 drivers/infiniband/hw/hfi1/ipoib.h             |  171 +++++
 drivers/infiniband/hw/hfi1/ipoib_main.c        |  309 +++++++++
 drivers/infiniband/hw/hfi1/ipoib_rx.c          |   95 +++
 drivers/infiniband/hw/hfi1/ipoib_tx.c          |  828 ++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/msix.c              |   36 +
 drivers/infiniband/hw/hfi1/msix.h              |    7 
 drivers/infiniband/hw/hfi1/netdev.h            |  118 +++
 drivers/infiniband/hw/hfi1/netdev_rx.c         |  484 ++++++++++++++
 drivers/infiniband/hw/hfi1/qp.c                |   18 -
 drivers/infiniband/hw/hfi1/tid_rdma.c          |    4 
 drivers/infiniband/hw/hfi1/trace.c             |   42 +
 drivers/infiniband/hw/hfi1/trace_ctxts.h       |   11 
 drivers/infiniband/hw/hfi1/verbs.c             |   13 
 drivers/infiniband/hw/hfi1/vnic.h              |    5 
 drivers/infiniband/hw/hfi1/vnic_main.c         |  318 ++-------
 drivers/infiniband/sw/rdmavt/qp.c              |   24 +
 drivers/infiniband/ulp/ipoib/ipoib_main.c      |   22 -
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |   12 
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c     |    3 
 drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |    3 
 include/rdma/ib_verbs.h                        |   82 ++
 include/rdma/opa_port_info.h                   |   10 
 include/rdma/opa_vnic.h                        |    4 
 include/rdma/rdmavt_qp.h                       |   29 +
 include/uapi/rdma/hfi/hfi1_user.h              |    3 
 35 files changed, 2784 insertions(+), 493 deletions(-)
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib.h
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_main.c
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_rx.c
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_tx.c
 create mode 100644 drivers/infiniband/hw/hfi1/netdev.h
 create mode 100644 drivers/infiniband/hw/hfi1/netdev_rx.c

--
-Denny

#1 lore raw

PATCH v3 1/16 IB/hfi1: Add accelerated IP capability bit

From: Kaike Wan <[email protected]>

The accelerated IP capability bit is added to allow users to control which feature is enabled and disabled.

---
 drivers/infiniband/hw/hfi1/common.h |    5 +++--
 include/uapi/rdma/hfi/hfi1_user.h   |    3 ++-
 2 files changed, 5 insertions(+), 3 deletions(-)
drivers/infiniband/hw/hfi1/common.h +3 -2
index 40a1ff0..1f7107e 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -149,7 +149,8 @@
 				   HFI1_CAP_NO_INTEGRITY |		\
 				   HFI1_CAP_PKEY_CHECK |		\
 				   HFI1_CAP_TID_RDMA |			\
-				   HFI1_CAP_OPFN) <<			\
+				   HFI1_CAP_OPFN |			\
+				   HFI1_CAP_AIP) <<			\
 				  HFI1_CAP_USER_SHIFT)
 /*
  * Set of capabilities that need to be enabled for kernel context in
include/uapi/rdma/hfi/hfi1_user.h +2 -1
index 01ac585..d95ef9a 100644
@@ -6,7 +6,7 @@
  *
  * GPL LICENSE SUMMARY
  *
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of version 2 of the GNU General Public License as
@@ -109,6 +109,7 @@
 #define HFI1_CAP_OPFN             (1UL << 16) /* Enable the OPFN protocol */
 #define HFI1_CAP_SDMA_HEAD_CHECK  (1UL << 17) /* SDMA head checking */
 #define HFI1_CAP_EARLY_CREDIT_RETURN (1UL << 18) /* early credit return */
+#define HFI1_CAP_AIP              (1UL << 19) /* Enable accelerated IP */
 
 #define HFI1_RCVHDR_ENTSIZE_2    (1UL << 0)
 #define HFI1_RCVHDR_ENTSIZE_16   (1UL << 1)
#2 lore raw

PATCH v3 2/16 IB/hfi1: Add functions to transmit datagram ipoib packets

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

This patch implements the mechanism to accelerate the transmit side of a multiple transmit queue RDMA netdev by submitting the packets to the SDMA engine directly instead of sending through the verbs layer.

This patch also changes the UD/SEND_ONLY op to output the entropy value in byte 0 of deth[1]. UD/SEND_ONLY_WITH_IMMEDIATE uses the previous behavior with no entropy value being output.

The code in the ipoib rdma netdev which submits tx requests upon successful submission will call trace_sdma_output_ibhdr to output the ibhdr to the trace buffer.

---
 drivers/infiniband/hw/hfi1/Makefile   |    1 
 drivers/infiniband/hw/hfi1/ipoib.h    |  145 ++++++
 drivers/infiniband/hw/hfi1/ipoib_tx.c |  828 +++++++++++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/trace.c    |   10 
 include/rdma/ib_verbs.h               |    1 
 5 files changed, 984 insertions(+), 1 deletion(-)
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib.h
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_tx.c
drivers/infiniband/hw/hfi1/Makefile +1 -0
index 0405d26..09ef0b8 100644
@@ -22,6 +22,7 @@ hfi1-y := \
 	init.o \
 	intr.o \
 	iowait.o \
+	ipoib_tx.o \
 	mad.o \
 	mmu_rb.o \
 	msix.o \
drivers/infiniband/hw/hfi1/ipoib.h new file +145 -0

145 lines not on this page · raw message

drivers/infiniband/hw/hfi1/ipoib_tx.c new file +828 -0

828 lines not on this page · raw message

drivers/infiniband/hw/hfi1/trace.c +9 -1
index 9a3d236..c8a9988 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -47,6 +47,7 @@
 #define CREATE_TRACE_POINTS
 #include "trace.h"
 #include "exp_rcv.h"
+#include "ipoib.h"
 
 static u8 __get_ib_hdr_len(struct ib_header *hdr)
 {
@@ -126,6 +127,7 @@ const char *hfi1_trace_get_packet_l2_str(u8 l2)
 #define RETH_PRN "reth vaddr:0x%.16llx rkey:0x%.8x dlen:0x%.8x"
 #define AETH_PRN "aeth syn:0x%.2x %s msn:0x%.8x"
 #define DETH_PRN "deth qkey:0x%.8x sqpn:0x%.6x"
+#define DETH_ENTROPY_PRN "deth qkey:0x%.8x sqpn:0x%.6x entropy:0x%.2x"
 #define IETH_PRN "ieth rkey:0x%.8x"
 #define ATOMICACKETH_PRN "origdata:%llx"
 #define ATOMICETH_PRN "vaddr:0x%llx rkey:0x%.8x sdata:%llx cdata:%llx"
@@ -444,6 +446,12 @@ const char *parse_everbs_hdrs(
 		break;
 	/* deth */
 	case OP(UD, SEND_ONLY):
+		trace_seq_printf(p, DETH_ENTROPY_PRN,
+				 be32_to_cpu(eh->ud.deth[0]),
+				 be32_to_cpu(eh->ud.deth[1]) & RVT_QPN_MASK,
+				 be32_to_cpu(eh->ud.deth[1]) >>
+					     HFI1_IPOIB_ENTROPY_SHIFT);
+		break;
 	case OP(UD, SEND_ONLY_WITH_IMMEDIATE):
 		trace_seq_printf(p, DETH_PRN,
 				 be32_to_cpu(eh->ud.deth[0]),
include/rdma/ib_verbs.h +1 -0
index 4c488ca..76651db 100644
@@ -2205,6 +2205,7 @@ struct rdma_netdev {
 	void              *clnt_priv;
 	struct ib_device  *hca;
 	u8                 port_num;
+	int                mtu;
 
 	/*
 	 * cleanup function must be specified.
#3 lore raw

PATCH v3 3/16 IB/hfi1: Add the transmit side of a datagram ipoib RDMA netdev

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

This implements the transmit side of the multiple transmit queue RDMA netdev used to accelerate ipoib. The receive side remains the ipoib internal implementation.

The init/unint/open/stop netdev operations are saved off and called by the versions within the hfi1 netdev in order to initialize the connected mode resources present in ipoib thus allowing us to switch modes between datagram and connected.

The datagram queue pair instantiated by the ipoib ulp is used by this implementation for its queue pair number and to register with multicast.

The above queue pair is not used on transmit other than its qpn as the verbs layer is skipped and packets are directly submitted to the sdma engines.

---
 drivers/infiniband/hw/hfi1/Makefile     |    1 
 drivers/infiniband/hw/hfi1/ipoib.h      |    5 +
 drivers/infiniband/hw/hfi1/ipoib_main.c |  283 +++++++++++++++++++++++++++++++
 3 files changed, 289 insertions(+)
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_main.c
drivers/infiniband/hw/hfi1/Makefile +1 -0
index 09ef0b8..0b25713 100644
@@ -22,6 +22,7 @@ hfi1-y := \
 	init.o \
 	intr.o \
 	iowait.o \
+	ipoib_main.o \
 	ipoib_tx.o \
 	mad.o \
 	mmu_rb.o \
drivers/infiniband/hw/hfi1/ipoib.h +5 -0
index 2b541ab..c2e63ca 100644
@@ -142,4 +142,9 @@ int hfi1_ipoib_send_dma(struct net_device *dev,
 void hfi1_ipoib_napi_tx_enable(struct net_device *dev);
 void hfi1_ipoib_napi_tx_disable(struct net_device *dev);
 
+int hfi1_ipoib_rn_get_params(struct ib_device *device,
+			     u8 port_num,
+			     enum rdma_netdev_t type,
+			     struct rdma_netdev_alloc_params *params);
+
 #endif /* _IPOIB_H */
drivers/infiniband/hw/hfi1/ipoib_main.c new file +283 -0

283 lines not on this page · raw message

#4 lore raw

PATCH v3 4/16 IB/hfi1: Remove module parameter for KDETH qpns

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

The module parameter for KDETH qpns is being removed in favor of always using the default value of 0x80 as the qpn prefix. Defines have been added for various KDETH values including the prefix of 0x80.
The reserved range now starts at the base value for KDETH qpns (0x80) and extends up to and including the last qpn for other reserved QP prefixed types.
Adjust other QP prefixed define names to match KDETH defined names.

---
 drivers/infiniband/hw/hfi1/chip.c     |   19 +++----------------
 drivers/infiniband/hw/hfi1/common.h   |    7 -------
 drivers/infiniband/hw/hfi1/file_ops.c |    4 ++--
 drivers/infiniband/hw/hfi1/hfi.h      |    3 +--
 drivers/infiniband/hw/hfi1/tid_rdma.c |    4 ++--
 drivers/infiniband/hw/hfi1/verbs.c    |    7 +++----
 include/rdma/rdmavt_qp.h              |   29 ++++++++++++++++++++++++++++-
 7 files changed, 39 insertions(+), 34 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +3 -16
index e0b1238..c08bf81 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -67,10 +67,6 @@
 #include "debugfs.h"
 #include "fault.h"
 
-uint kdeth_qp;
-module_param_named(kdeth_qp, kdeth_qp, uint, S_IRUGO);
-MODULE_PARM_DESC(kdeth_qp, "Set the KDETH queue pair prefix");
-
 uint num_vls = HFI1_MAX_VLS_SUPPORTED;
 module_param(num_vls, uint, S_IRUGO);
 MODULE_PARM_DESC(num_vls, "Set number of Virtual Lanes to use (1-8)");
@@ -14119,21 +14115,12 @@ static void init_early_variables(struct hfi1_devdata *dd)
 
 static void init_kdeth_qp(struct hfi1_devdata *dd)
 {
-	/* user changed the KDETH_QP */
-	if (kdeth_qp != 0 && kdeth_qp >= 0xff) {
-		/* out of range or illegal value */
-		dd_dev_err(dd, "Invalid KDETH queue pair prefix, ignoring");
-		kdeth_qp = 0;
-	}
-	if (kdeth_qp == 0)	/* not set, or failed range check */
-		kdeth_qp = DEFAULT_KDETH_QP;
-
 	write_csr(dd, SEND_BTH_QP,
-		  (kdeth_qp & SEND_BTH_QP_KDETH_QP_MASK) <<
+		  (RVT_KDETH_QP_PREFIX & SEND_BTH_QP_KDETH_QP_MASK) <<
 		  SEND_BTH_QP_KDETH_QP_SHIFT);
 
 	write_csr(dd, RCV_BTH_QP,
-		  (kdeth_qp & RCV_BTH_QP_KDETH_QP_MASK) <<
+		  (RVT_KDETH_QP_PREFIX & RCV_BTH_QP_KDETH_QP_MASK) <<
 		  RCV_BTH_QP_KDETH_QP_SHIFT);
 }
 
drivers/infiniband/hw/hfi1/common.h +0 -7
index 1f7107e..6062545 100644
@@ -72,13 +72,6 @@
  * compilation unit
  */
 
-/*
- * If a packet's QP[23:16] bits match this value, then it is
- * a PSM packet and the hardware will expect a KDETH header
- * following the BTH.
- */
-#define DEFAULT_KDETH_QP 0x80
-
 /* driver/hw feature set bitmask */
 #define HFI1_CAP_USER_SHIFT      24
 #define HFI1_CAP_MASK            ((1UL << HFI1_CAP_USER_SHIFT) - 1)
drivers/infiniband/hw/hfi1/file_ops.c +2 -2
index e7fdd70..8ca51e4 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015-2017 Intel Corporation.
+ * Copyright(c) 2015-2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -1264,7 +1264,7 @@ static int get_base_info(struct hfi1_filedata *fd, unsigned long arg, u32 len)
 	memset(&binfo, 0, sizeof(binfo));
 	binfo.hw_version = dd->revision;
 	binfo.sw_version = HFI1_KERN_SWVERSION;
-	binfo.bthqp = kdeth_qp;
+	binfo.bthqp = RVT_KDETH_QP_PREFIX;
 	binfo.jkey = uctxt->jkey;
 	/*
 	 * If more than 64 contexts are enabled the allocated credit
drivers/infiniband/hw/hfi1/hfi.h +1 -2
index b06c259..ed13051 100644
@@ -1,7 +1,7 @@
 #ifndef _HFI1_KERNEL_H
 #define _HFI1_KERNEL_H
 /*
- * Copyright(c) 2015-2018 Intel Corporation.
+ * Copyright(c) 2015-2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -2250,7 +2250,6 @@ static inline void flush_wc(void)
 extern unsigned long n_krcvqs;
 extern uint krcvqs[];
 extern int krcvqsset;
-extern uint kdeth_qp;
 extern uint loopback;
 extern uint quick_linkup;
 extern uint rcv_intr_timeout;
drivers/infiniband/hw/hfi1/tid_rdma.c +2 -2
index 8a2e0d9..243b4ba 100644
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
 /*
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018 - 2020 Intel Corporation.
  *
  */
 
@@ -194,7 +194,7 @@ void tid_rdma_opfn_init(struct rvt_qp *qp, struct tid_rdma_params *p)
 {
 	struct hfi1_qp_priv *priv = qp->priv;
 
-	p->qp = (kdeth_qp << 16) | priv->rcd->ctxt;
+	p->qp = (RVT_KDETH_QP_PREFIX << 16) | priv->rcd->ctxt;
 	p->max_len = TID_RDMA_MAX_SEGMENT_SIZE;
 	p->jkey = priv->rcd->jkey;
 	p->max_read = TID_RDMA_MAX_READ_SEGS_PER_REQ;
drivers/infiniband/hw/hfi1/verbs.c +3 -4
index 2f6323a..c1c6fa98 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -1863,9 +1863,8 @@ int hfi1_register_ib_device(struct hfi1_devdata *dd)
 	dd->verbs_dev.rdi.dparms.qpn_start = 0;
 	dd->verbs_dev.rdi.dparms.qpn_inc = 1;
 	dd->verbs_dev.rdi.dparms.qos_shift = dd->qos_shift;
-	dd->verbs_dev.rdi.dparms.qpn_res_start = kdeth_qp << 16;
-	dd->verbs_dev.rdi.dparms.qpn_res_end =
-	dd->verbs_dev.rdi.dparms.qpn_res_start + 65535;
+	dd->verbs_dev.rdi.dparms.qpn_res_start = RVT_KDETH_QP_BASE;
+	dd->verbs_dev.rdi.dparms.qpn_res_end = RVT_AIP_QP_MAX;
 	dd->verbs_dev.rdi.dparms.max_rdma_atomic = HFI1_MAX_RDMA_ATOMIC;
 	dd->verbs_dev.rdi.dparms.psn_mask = PSN_MASK;
 	dd->verbs_dev.rdi.dparms.psn_shift = PSN_SHIFT;
include/rdma/rdmavt_qp.h +28 -1
index 5fc1010..4487af8 100644
@@ -2,7 +2,7 @@
 #define DEF_RDMAVT_INCQP_H
 
 /*
- * Copyright(c) 2016 - 2019 Intel Corporation.
+ * Copyright(c) 2016 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -69,6 +69,33 @@
 #define RVT_R_COMM_EST  0x10
 
 /*
+ * If a packet's QP[23:16] bits match this value, then it is
+ * a PSM packet and the hardware will expect a KDETH header
+ * following the BTH.
+ */
+#define RVT_KDETH_QP_PREFIX       0x80
+#define RVT_KDETH_QP_SUFFIX       0xffff
+#define RVT_KDETH_QP_PREFIX_MASK  0x00ff0000
+#define RVT_KDETH_QP_PREFIX_SHIFT 16
+#define RVT_KDETH_QP_BASE         (u32)(RVT_KDETH_QP_PREFIX << \
+					RVT_KDETH_QP_PREFIX_SHIFT)
+#define RVT_KDETH_QP_MAX          (u32)(RVT_KDETH_QP_BASE + RVT_KDETH_QP_SUFFIX)
+
+/*
+ * If a packet's LNH == BTH and DEST QPN[23:16] in the BTH match this
+ * prefix value, then it is an AIP packet with a DETH containing the entropy
+ * value in byte 4 following the BTH.
+ */
+#define RVT_AIP_QP_PREFIX       0x81
+#define RVT_AIP_QP_SUFFIX       0xffff
+#define RVT_AIP_QP_PREFIX_MASK  0x00ff0000
+#define RVT_AIP_QP_PREFIX_SHIFT 16
+#define RVT_AIP_QP_BASE         (u32)(RVT_AIP_QP_PREFIX << \
+				      RVT_AIP_QP_PREFIX_SHIFT)
+#define RVT_AIP_QPN_MAX         BIT(RVT_AIP_QP_PREFIX_SHIFT)
+#define RVT_AIP_QP_MAX          (u32)(RVT_AIP_QP_BASE + RVT_AIP_QPN_MAX - 1)
+
+/*
  * Bit definitions for s_flags.
  *
  * RVT_S_SIGNAL_REQ_WR - set if QP send WRs contain completion signaled
#5 lore raw

PATCH v3 5/16 IB/{rdmavt, hfi1}: Implement creation of accelerated UD QPs

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

Adds capability to create a qpn to be recognized as an accelerated UD QP for ipoib.

This is accomplished by reserving 0x81 in byte[0] of the qpn as the prefix for these qp types and reserving qpns between 0x810000 and 0x81ffff.

The hfi1 capability mask already contained a flag for the VNIC netdev. This has been renamed and extended to include both VNIC and ipoib.

The rvt code to allocate qps now recognizes this flag and sets 0x81 into byte[0] of the qpn.

The code to allocate qpns is modified to reset the qpn numbering when it is detected that a value is located in byte[0] for a UD QP and it is a qpn being requested for net dev use. If it is a regular UD QP then it is allowable to have bits set in byte[0] of the qpn and provide the previously normal behavior.

The code to free the qpn now checks for the AIP prefix value of 0x81 and removes it from the qpn before being freed so that the lower 16 bit number can be reused.

This patch requires minor changes in the IB core and ipoib to facilitate the creation of accelerated UP QPs.

---
 drivers/infiniband/hw/hfi1/verbs.c         |    2 +-
 drivers/infiniband/sw/rdmavt/qp.c          |   24 +++++++++++++++++++-----
 drivers/infiniband/ulp/ipoib/ipoib_verbs.c |    3 +++
 include/rdma/ib_verbs.h                    |    4 ++--
 include/rdma/opa_vnic.h                    |    4 ++--
 5 files changed, 27 insertions(+), 10 deletions(-)
drivers/infiniband/hw/hfi1/verbs.c +1 -1
index c1c6fa98..c61b291 100644
@@ -1342,7 +1342,7 @@ static void hfi1_fill_device_attr(struct hfi1_devdata *dd)
 			IB_DEVICE_SYS_IMAGE_GUID | IB_DEVICE_RC_RNR_NAK_GEN |
 			IB_DEVICE_PORT_ACTIVE_EVENT | IB_DEVICE_SRQ_RESIZE |
 			IB_DEVICE_MEM_MGT_EXTENSIONS |
-			IB_DEVICE_RDMA_NETDEV_OPA_VNIC;
+			IB_DEVICE_RDMA_NETDEV_OPA;
 	rdi->dparms.props.page_size_cap = PAGE_SIZE;
 	rdi->dparms.props.vendor_id = dd->oui1 << 16 | dd->oui2 << 8 | dd->oui3;
 	rdi->dparms.props.vendor_part_id = dd->pcidev->device;
drivers/infiniband/sw/rdmavt/qp.c +19 -5
index 0e1b291..91ad6c5 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2016 - 2019 Intel Corporation.
+ * Copyright(c) 2016 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -525,15 +525,18 @@ static inline unsigned mk_qpn(struct rvt_qpn_table *qpt,
  * @rdi: rvt device info structure
  * @qpt: queue pair number table pointer
  * @port_num: IB port number, 1 based, comes from core
+ * @exclude_prefix: prefix of special queue pair number being allocated
  *
  * Return: The queue pair number
  */
 static int alloc_qpn(struct rvt_dev_info *rdi, struct rvt_qpn_table *qpt,
-		     enum ib_qp_type type, u8 port_num)
+		     enum ib_qp_type type, u8 port_num, u8 exclude_prefix)
 {
 	u32 i, offset, max_scan, qpn;
 	struct rvt_qpn_map *map;
 	u32 ret;
+	u32 max_qpn = exclude_prefix == RVT_AIP_QP_PREFIX ?
+		RVT_AIP_QPN_MAX : RVT_QPN_MAX;
 
 	if (rdi->driver_f.alloc_qpn)
 		return rdi->driver_f.alloc_qpn(rdi, qpt, type, port_num);
@@ -553,7 +556,7 @@ static int alloc_qpn(struct rvt_dev_info *rdi, struct rvt_qpn_table *qpt,
 	}
 
 	qpn = qpt->last + qpt->incr;
-	if (qpn >= RVT_QPN_MAX)
+	if (qpn >= max_qpn)
 		qpn = qpt->incr | ((qpt->last & 1) ^ 1);
 	/* offset carries bit 0 */
 	offset = qpn & RVT_BITS_PER_PAGE_MASK;
@@ -987,6 +990,9 @@ static void rvt_free_qpn(struct rvt_qpn_table *qpt, u32 qpn)
 {
 	struct rvt_qpn_map *map;
 
+	if ((qpn & RVT_AIP_QP_PREFIX_MASK) == RVT_AIP_QP_BASE)
+		qpn &= RVT_AIP_QP_SUFFIX;
+
 	map = qpt->map + (qpn & RVT_QPN_MASK) / RVT_BITS_PER_PAGE;
 	if (map->page)
 		clear_bit(qpn & RVT_BITS_PER_PAGE_MASK, map->page);
@@ -1074,13 +1080,15 @@ struct ib_qp *rvt_create_qp(struct ib_pd *ibpd,
 	struct rvt_dev_info *rdi = ib_to_rvt(ibpd->device);
 	void *priv = NULL;
 	size_t sqsize;
+	u8 exclude_prefix = 0;
 
 	if (!rdi)
 		return ERR_PTR(-EINVAL);
 
 	if (init_attr->cap.max_send_sge > rdi->dparms.props.max_send_sge ||
 	    init_attr->cap.max_send_wr > rdi->dparms.props.max_qp_wr ||
-	    init_attr->create_flags)
+	    (init_attr->create_flags &&
+	     init_attr->create_flags != IB_QP_CREATE_NETDEV_USE))
 		return ERR_PTR(-EINVAL);
 
 	/* Check receive queue parameters if no SRQ is specified. */
@@ -1199,14 +1207,20 @@ struct ib_qp *rvt_create_qp(struct ib_pd *ibpd,
 			goto bail_driver_priv;
 		}
 
+		if (init_attr->create_flags & IB_QP_CREATE_NETDEV_USE)
+			exclude_prefix = RVT_AIP_QP_PREFIX;
+
 		err = alloc_qpn(rdi, &rdi->qp_dev->qpn_table,
 				init_attr->qp_type,
-				init_attr->port_num);
+				init_attr->port_num,
+				exclude_prefix);
 		if (err < 0) {
 			ret = ERR_PTR(err);
 			goto bail_rq_wq;
 		}
 		qp->ibqp.qp_num = err;
+		if (init_attr->create_flags & IB_QP_CREATE_NETDEV_USE)
+			qp->ibqp.qp_num |= RVT_AIP_QP_BASE;
 		qp->port_num = init_attr->port_num;
 		rvt_init_qp(rdi, qp, init_attr->qp_type);
 		if (rdi->driver_f.qp_priv_init) {
drivers/infiniband/ulp/ipoib/ipoib_verbs.c +3 -0
index b69304d..587252f 100644
@@ -206,6 +206,9 @@ int ipoib_transport_dev_init(struct net_device *dev, struct ib_device *ca)
 	if (priv->hca_caps & IB_DEVICE_MANAGED_FLOW_STEERING)
 		init_attr.create_flags |= IB_QP_CREATE_NETIF_QP;
 
+	if (priv->hca_caps & IB_DEVICE_RDMA_NETDEV_OPA)
+		init_attr.create_flags |= IB_QP_CREATE_NETDEV_USE;
+
 	priv->qp = ib_create_qp(priv->pd, &init_attr);
 	if (IS_ERR(priv->qp)) {
 		pr_warn("%s: failed to create QP\n", ca->name);
include/rdma/ib_verbs.h +2 -2
index 76651db..4b23ee5 100644
@@ -305,7 +305,7 @@ enum ib_device_cap_flags {
 	IB_DEVICE_VIRTUAL_FUNCTION		= (1ULL << 33),
 	/* Deprecated. Please use IB_RAW_PACKET_CAP_SCATTER_FCS. */
 	IB_DEVICE_RAW_SCATTER_FCS		= (1ULL << 34),
-	IB_DEVICE_RDMA_NETDEV_OPA_VNIC		= (1ULL << 35),
+	IB_DEVICE_RDMA_NETDEV_OPA		= (1ULL << 35),
 	/* The device supports padding incoming writes to cacheline. */
 	IB_DEVICE_PCI_WRITE_END_PADDING		= (1ULL << 36),
 	IB_DEVICE_ALLOW_USER_UNREG		= (1ULL << 37),
@@ -1117,7 +1117,7 @@ enum ib_qp_create_flags {
 	IB_QP_CREATE_MANAGED_RECV               = 1 << 4,
 	IB_QP_CREATE_NETIF_QP			= 1 << 5,
 	IB_QP_CREATE_INTEGRITY_EN		= 1 << 6,
-	/* FREE					= 1 << 7, */
+	IB_QP_CREATE_NETDEV_USE			= 1 << 7,
 	IB_QP_CREATE_SCATTER_FCS		= 1 << 8,
 	IB_QP_CREATE_CVLAN_STRIPPING		= 1 << 9,
 	IB_QP_CREATE_SOURCE_QPN			= 1 << 10,
include/rdma/opa_vnic.h +2 -2
index e90b149..6f244e7 100644
@@ -1,7 +1,7 @@
 #ifndef _OPA_VNIC_H
 #define _OPA_VNIC_H
 /*
- * Copyright(c) 2017 Intel Corporation.
+ * Copyright(c) 2017 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -132,7 +132,7 @@ struct opa_vnic_stats {
 static inline bool rdma_cap_opa_vnic(struct ib_device *device)
 {
 	return !!(device->attrs.device_cap_flags &
-		  IB_DEVICE_RDMA_NETDEV_OPA_VNIC);
+		  IB_DEVICE_RDMA_NETDEV_OPA);
 }
 
 #endif /* _OPA_VNIC_H */
#6 lore raw

PATCH v3 6/16 IB/hfi1: RSM rules for AIP

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

This is implementation of RSM rule for AIP packets. AIP rule will use rule RSM2 and will match standard Infiniband packet containg BTH (LNH==BTH) and having Dest QPN prefixed with value 0x81. Spread between receive contexts will be done using source QPN bits.

VNIC and AIP will share receive contexts, so their rules will point to the same RMT entries and their shared code is moved to separate functions.
If any of the rules is active RMT mapping will be skipped for latter.

Changed function hfi1_vnic_is_rsm_full to be more general and moved it from main header to chip.c.

Changed the order of RSM rules because AIP rule as more specific one is needed to be placed before more general QOS rule. Rules are occupying two last RSM registers.

---
 drivers/infiniband/hw/hfi1/chip.c |  171 ++++++++++++++++++++++++++++---------
 drivers/infiniband/hw/hfi1/chip.h |    4 +
 drivers/infiniband/hw/hfi1/hfi.h  |    8 +-
 drivers/infiniband/hw/hfi1/init.c |    3 -
 4 files changed, 136 insertions(+), 50 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +128 -43
index c08bf81..be1fb29 100644
@@ -124,13 +124,15 @@ struct flag_table {
 
 /*
  * RSM instance allocation
- *   0 - Verbs
- *   1 - User Fecn Handling
- *   2 - Vnic
+ *   0 - User Fecn Handling
+ *   1 - Vnic
+ *   2 - AIP
+ *   3 - Verbs
  */
-#define RSM_INS_VERBS             0
-#define RSM_INS_FECN              1
-#define RSM_INS_VNIC              2
+#define RSM_INS_FECN              0
+#define RSM_INS_VNIC              1
+#define RSM_INS_AIP               2
+#define RSM_INS_VERBS             3
 
 /* Bit offset into the GUID which carries HFI id information */
 #define GUID_HFI_INDEX_SHIFT     39
@@ -171,6 +173,25 @@ struct flag_table {
 /* QPN[m+n:1] QW 1, OFFSET 1 */
 #define QPN_SELECT_OFFSET      ((1ull << QW_SHIFT) | (1ull))
 
+/* RSM fields for AIP */
+/* LRH.BTH above is reused for this rule */
+
+/* BTH.DESTQP: QW 1, OFFSET 16 for match */
+#define BTH_DESTQP_QW           1ull
+#define BTH_DESTQP_BIT_OFFSET   16ull
+#define BTH_DESTQP_OFFSET(off) ((BTH_DESTQP_QW << QW_SHIFT) | (off))
+#define BTH_DESTQP_MATCH_OFFSET BTH_DESTQP_OFFSET(BTH_DESTQP_BIT_OFFSET)
+#define BTH_DESTQP_MASK         0xFFull
+#define BTH_DESTQP_VALUE        0x81ull
+
+/* DETH.SQPN: QW 1 Offset 56 for select */
+/* We use 8 most significant Soure QPN bits as entropy fpr AIP */
+#define DETH_AIP_SQPN_QW 3ull
+#define DETH_AIP_SQPN_BIT_OFFSET 56ull
+#define DETH_AIP_SQPN_OFFSET(off) ((DETH_AIP_SQPN_QW << QW_SHIFT) | (off))
+#define DETH_AIP_SQPN_SELECT_OFFSET \
+	DETH_AIP_SQPN_OFFSET(DETH_AIP_SQPN_BIT_OFFSET)
+
 /* RSM fields for Vnic */
 /* L2_TYPE: QW 0, OFFSET 61 - for match */
 #define L2_TYPE_QW             0ull
@@ -14236,6 +14257,12 @@ static void complete_rsm_map_table(struct hfi1_devdata *dd,
 	}
 }
 
+/* Is a receive side mapping rule */
+static bool has_rsm_rule(struct hfi1_devdata *dd, u8 rule_index)
+{
+	return read_csr(dd, RCV_RSM_CFG + (8 * rule_index)) != 0;
+}
+
 /*
  * Add a receive side mapping rule.
  */
@@ -14472,39 +14499,49 @@ static void init_fecn_handling(struct hfi1_devdata *dd,
 	rmt->used += total_cnt;
 }
 
-/* Initialize RSM for VNIC */
-void hfi1_init_vnic_rsm(struct hfi1_devdata *dd)
+static inline bool hfi1_is_rmt_full(int start, int spare)
+{
+	return (start + spare) > NUM_MAP_ENTRIES;
+}
+
+static bool hfi1_netdev_update_rmt(struct hfi1_devdata *dd)
 {
 	u8 i, j;
 	u8 ctx_id = 0;
 	u64 reg;
 	u32 regoff;
-	struct rsm_rule_data rrd;
+	int rmt_start = dd->vnic.rmt_start;
 
-	if (hfi1_vnic_is_rsm_full(dd, NUM_VNIC_MAP_ENTRIES)) {
-		dd_dev_err(dd, "Vnic RSM disabled, rmt entries used = %d\n",
-			   dd->vnic.rmt_start);
-		return;
+	/* We already have contexts mapped in RMT */
+	if (has_rsm_rule(dd, RSM_INS_VNIC) || has_rsm_rule(dd, RSM_INS_AIP)) {
+		dd_dev_info(dd, "Contexts are already mapped in RMT\n");
+		return true;
+	}
+
+	if (hfi1_is_rmt_full(rmt_start, NUM_VNIC_MAP_ENTRIES)) {
+		dd_dev_err(dd, "Not enought RMT entries used = %d\n",
+			   rmt_start);
+		return false;
 	}
 
-	dev_dbg(&(dd)->pcidev->dev, "Vnic rsm start = %d, end %d\n",
-		dd->vnic.rmt_start,
-		dd->vnic.rmt_start + NUM_VNIC_MAP_ENTRIES);
+	dev_dbg(&(dd)->pcidev->dev, "RMT start = %d, end %d\n",
+		rmt_start,
+		rmt_start + NUM_VNIC_MAP_ENTRIES);
 
 	/* Update RSM mapping table, 32 regs, 256 entries - 1 ctx per byte */
-	regoff = RCV_RSM_MAP_TABLE + (dd->vnic.rmt_start / 8) * 8;
+	regoff = RCV_RSM_MAP_TABLE + (rmt_start / 8) * 8;
 	reg = read_csr(dd, regoff);
 	for (i = 0; i < NUM_VNIC_MAP_ENTRIES; i++) {
-		/* Update map register with vnic context */
-		j = (dd->vnic.rmt_start + i) % 8;
+		/* Update map register with netdev context */
+		j = (rmt_start + i) % 8;
 		reg &= ~(0xffllu << (j * 8));
 		reg |= (u64)dd->vnic.ctxt[ctx_id++]->ctxt << (j * 8);
-		/* Wrap up vnic ctx index */
+		/* Wrap up netdev ctx index */
 		ctx_id %= dd->vnic.num_ctxt;
 		/* Write back map register */
 		if (j == 7 || ((i + 1) == NUM_VNIC_MAP_ENTRIES)) {
 			dev_dbg(&(dd)->pcidev->dev,
-				"Vnic rsm map reg[%d] =0x%llx\n",
+				"RMT[%d] =0x%llx\n",
 				regoff - RCV_RSM_MAP_TABLE, reg);
 
 			write_csr(dd, regoff, reg);
@@ -14514,35 +14551,83 @@ void hfi1_init_vnic_rsm(struct hfi1_devdata *dd)
 		}
 	}
 
-	/* Add rule for vnic */
-	rrd.offset = dd->vnic.rmt_start;
-	rrd.pkt_type = 4;
-	/* Match 16B packets */
-	rrd.field1_off = L2_TYPE_MATCH_OFFSET;
-	rrd.mask1 = L2_TYPE_MASK;
-	rrd.value1 = L2_16B_VALUE;
-	/* Match ETH L4 packets */
-	rrd.field2_off = L4_TYPE_MATCH_OFFSET;
-	rrd.mask2 = L4_16B_TYPE_MASK;
-	rrd.value2 = L4_16B_ETH_VALUE;
-	/* Calc context from veswid and entropy */
-	rrd.index1_off = L4_16B_HDR_VESWID_OFFSET;
-	rrd.index1_width = ilog2(NUM_VNIC_MAP_ENTRIES);
-	rrd.index2_off = L2_16B_ENTROPY_OFFSET;
-	rrd.index2_width = ilog2(NUM_VNIC_MAP_ENTRIES);
-	add_rsm_rule(dd, RSM_INS_VNIC, &rrd);
-
-	/* Enable RSM if not already enabled */
+	return true;
+}
+
+static void hfi1_enable_rsm_rule(struct hfi1_devdata *dd,
+				 int rule, struct rsm_rule_data *rrd)
+{
+	if (!hfi1_netdev_update_rmt(dd)) {
+		dd_dev_err(dd, "Failed to update RMT for RSM%d rule\n", rule);
+		return;
+	}
+
+	add_rsm_rule(dd, rule, rrd);
 	add_rcvctrl(dd, RCV_CTRL_RCV_RSM_ENABLE_SMASK);
 }
 
+void hfi1_init_aip_rsm(struct hfi1_devdata *dd)
+{
+	/*
+	 * go through with the initialisation only if this rule actually doesn't
+	 * exist yet
+	 */
+	if (atomic_fetch_inc(&dd->ipoib_rsm_usr_num) == 0) {
+		struct rsm_rule_data rrd = {
+			.offset = dd->vnic.rmt_start,
+			.pkt_type = IB_PACKET_TYPE,
+			.field1_off = LRH_BTH_MATCH_OFFSET,
+			.mask1 = LRH_BTH_MASK,
+			.value1 = LRH_BTH_VALUE,
+			.field2_off = BTH_DESTQP_MATCH_OFFSET,
+			.mask2 = BTH_DESTQP_MASK,
+			.value2 = BTH_DESTQP_VALUE,
+			.index1_off = DETH_AIP_SQPN_SELECT_OFFSET +
+					ilog2(NUM_VNIC_MAP_ENTRIES),
+			.index1_width = ilog2(NUM_VNIC_MAP_ENTRIES),
+			.index2_off = DETH_AIP_SQPN_SELECT_OFFSET,
+			.index2_width = ilog2(NUM_VNIC_MAP_ENTRIES)
+		};
+
+		hfi1_enable_rsm_rule(dd, RSM_INS_AIP, &rrd);
+	}
+}
+
+/* Initialize RSM for VNIC */
+void hfi1_init_vnic_rsm(struct hfi1_devdata *dd)
+{
+	struct rsm_rule_data rrd = {
+		/* Add rule for vnic */
+		.offset = dd->vnic.rmt_start,
+		.pkt_type = 4,
+		/* Match 16B packets */
+		.field1_off = L2_TYPE_MATCH_OFFSET,
+		.mask1 = L2_TYPE_MASK,
+		.value1 = L2_16B_VALUE,
+		/* Match ETH L4 packets */
+		.field2_off = L4_TYPE_MATCH_OFFSET,
+		.mask2 = L4_16B_TYPE_MASK,
+		.value2 = L4_16B_ETH_VALUE,
+		/* Calc context from veswid and entropy */
+		.index1_off = L4_16B_HDR_VESWID_OFFSET,
+		.index1_width = ilog2(NUM_VNIC_MAP_ENTRIES),
+		.index2_off = L2_16B_ENTROPY_OFFSET,
+		.index2_width = ilog2(NUM_VNIC_MAP_ENTRIES)
+	};
+
+	hfi1_enable_rsm_rule(dd, RSM_INS_VNIC, &rrd);
+}
+
 void hfi1_deinit_vnic_rsm(struct hfi1_devdata *dd)
 {
 	clear_rsm_rule(dd, RSM_INS_VNIC);
+}
 
-	/* Disable RSM if used only by vnic */
-	if (dd->vnic.rmt_start == 0)
-		clear_rcvctrl(dd, RCV_CTRL_RCV_RSM_ENABLE_SMASK);
+void hfi1_deinit_aip_rsm(struct hfi1_devdata *dd)
+{
+	/* only actually clear the rule if it's the last user asking to do so */
+	if (atomic_fetch_add_unless(&dd->ipoib_rsm_usr_num, -1, 0) == 1)
+		clear_rsm_rule(dd, RSM_INS_AIP);
 }
 
 static int init_rxe(struct hfi1_devdata *dd)
drivers/infiniband/hw/hfi1/chip.h +3 -1
index 7255092..b10e0bf 100644
@@ -1,7 +1,7 @@
 #ifndef _CHIP_H
 #define _CHIP_H
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -1455,6 +1455,8 @@ int hfi1_set_ctxt_pkey(struct hfi1_devdata *dd, struct hfi1_ctxtdata *ctxt,
 void remap_sdma_interrupts(struct hfi1_devdata *dd, int engine, int msix_intr);
 void reset_interrupts(struct hfi1_devdata *dd);
 u8 hfi1_get_qp_map(struct hfi1_devdata *dd, u8 idx);
+void hfi1_init_aip_rsm(struct hfi1_devdata *dd);
+void hfi1_deinit_aip_rsm(struct hfi1_devdata *dd);
 
 /*
  * Interrupt source table.
drivers/infiniband/hw/hfi1/hfi.h +3 -5
index ed13051..c61e56a 100644
@@ -1417,12 +1417,10 @@ struct hfi1_devdata {
 	struct hfi1_vnic_data vnic;
 	/* Lock to protect IRQ SRC register access */
 	spinlock_t irq_src_lock;
-};
 
-static inline bool hfi1_vnic_is_rsm_full(struct hfi1_devdata *dd, int spare)
-{
-	return (dd->vnic.rmt_start + spare) > NUM_MAP_ENTRIES;
-}
+	/* Keeps track of IPoIB RSM rule users */
+	atomic_t ipoib_rsm_usr_num;
+};
 
 /* 8051 firmware version helper */
 #define dc8051_ver(a, b, c) ((a) << 16 | (b) << 8 | (c))
drivers/infiniband/hw/hfi1/init.c +2 -1
index 3759d92..8c6b96a 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -1316,6 +1316,7 @@ static struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev,
 		goto bail;
 	}
 
+	atomic_set(&dd->ipoib_rsm_usr_num, 0);
 	return dd;
 
 bail:
#7 lore raw

PATCH v3 7/16 IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

In reply to Dennis Dalessandro

From: Kaike Wan <[email protected]>

Currently the ipoib UD mtu is restricted to 4K bytes. Remove this limitation so that the IPOIB module can potentially use an MTU (in UD mode) that is bounded by the MTU of the underlying device. A field is added to the ib_port_attr structure to indicate the maximum physical MTU the underlying device supports.

---
 drivers/infiniband/hw/hfi1/qp.c                |   18 +-----
 drivers/infiniband/hw/hfi1/verbs.c             |    2 +
 drivers/infiniband/ulp/ipoib/ipoib_main.c      |    2 -
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |   11 ++-
 include/rdma/ib_verbs.h                        |   77 ++++++++++++++++++++++++
 include/rdma/opa_port_info.h                   |   10 ---
 6 files changed, 88 insertions(+), 32 deletions(-)
drivers/infiniband/hw/hfi1/qp.c +2 -16
index f8e733a..0c2ae9f 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2019 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -186,15 +186,6 @@ static void flush_iowait(struct rvt_qp *qp)
 	write_sequnlock_irqrestore(lock, flags);
 }
 
-static inline int opa_mtu_enum_to_int(int mtu)
-{
-	switch (mtu) {
-	case OPA_MTU_8192:  return 8192;
-	case OPA_MTU_10240: return 10240;
-	default:            return -1;
-	}
-}
-
 /**
  * This function is what we would push to the core layer if we wanted to be a
  * "first class citizen".  Instead we hide this here and rely on Verbs ULPs
@@ -202,15 +193,10 @@ static inline int opa_mtu_enum_to_int(int mtu)
  */
 static inline int verbs_mtu_enum_to_int(struct ib_device *dev, enum ib_mtu mtu)
 {
-	int val;
-
 	/* Constraining 10KB packets to 8KB packets */
 	if (mtu == (enum ib_mtu)OPA_MTU_10240)
 		mtu = OPA_MTU_8192;
-	val = opa_mtu_enum_to_int((int)mtu);
-	if (val > 0)
-		return val;
-	return ib_mtu_enum_to_int(mtu);
+	return opa_mtu_enum_to_int((enum opa_mtu)mtu);
 }
 
 int hfi1_check_modify_qp(struct rvt_qp *qp, struct ib_qp_attr *attr,
drivers/infiniband/hw/hfi1/verbs.c +2 -0
index c61b291..19d5d00 100644
@@ -1439,6 +1439,8 @@ static int query_port(struct rvt_dev_info *rdi, u8 port_num,
 				      4096 : hfi1_max_mtu), IB_MTU_4096);
 	props->active_mtu = !valid_ib_mtu(ppd->ibmtu) ? props->max_mtu :
 		mtu_to_enum(ppd->ibmtu, IB_MTU_4096);
+	props->phys_mtu = HFI1_CAP_IS_KSET(AIP) ? hfi1_max_mtu :
+				ib_mtu_enum_to_int(props->max_mtu);
 
 	return 0;
 }
drivers/infiniband/ulp/ipoib/ipoib_main.c +1 -1
index d4c6a97..22216f1 100644
@@ -1855,7 +1855,7 @@ static int ipoib_parent_init(struct net_device *ndev)
 			priv->port);
 		return result;
 	}
-	priv->max_ib_mtu = ib_mtu_enum_to_int(attr.max_mtu);
+	priv->max_ib_mtu = rdma_mtu_from_attr(priv->ca, priv->port, &attr);
 
 	result = ib_query_pkey(priv->ca, priv->port, 0, &priv->pkey);
 	if (result) {
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5 -6
index b9e9562..7166ee9b 100644
@@ -218,6 +218,7 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 	struct rdma_ah_attr av;
 	int ret;
 	int set_qkey = 0;
+	int mtu;
 
 	mcast->mcmember = *mcmember;
 
@@ -240,13 +241,11 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 		priv->broadcast->mcmember.flow_label = mcmember->flow_label;
 		priv->broadcast->mcmember.hop_limit = mcmember->hop_limit;
 		/* assume if the admin and the mcast are the same both can be changed */
+		mtu = rdma_mtu_enum_to_int(priv->ca,  priv->port,
+					   priv->broadcast->mcmember.mtu);
 		if (priv->mcast_mtu == priv->admin_mtu)
-			priv->admin_mtu =
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
-		else
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
+			priv->admin_mtu = IPOIB_UD_MTU(mtu);
+		priv->mcast_mtu = IPOIB_UD_MTU(mtu);
 
 		priv->qkey = be32_to_cpu(priv->broadcast->mcmember.qkey);
 		spin_unlock_irq(&priv->lock);
include/rdma/ib_verbs.h +77 -0

101 lines not on this page · raw message

include/rdma/opa_port_info.h +1 -9
index bdbfe25..0d9e6d7 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2014-2017 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2014-2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -139,14 +139,6 @@
 #define OPA_CAP_MASK3_IsVLMarkerSupported         (1 << 1)
 #define OPA_CAP_MASK3_IsVLrSupported              (1 << 0)
 
-/**
- * new MTU values
- */
-enum {
-	OPA_MTU_8192  = 6,
-	OPA_MTU_10240 = 7,
-};
-
 enum {
 	OPA_PORT_PHYS_CONF_DISCONNECTED = 0,
 	OPA_PORT_PHYS_CONF_STANDARD     = 1,
#8 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 248 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

Currently the ipoib UD mtu is restricted to 4K bytes. Remove this limitation so that the IPOIB module can potentially use an MTU (in UD mode) that is bounded by the MTU of the underlying device. A field is added to the ib_port_attr structure to indicate the maximum physical MTU the underlying device supports.

---
 drivers/infiniband/hw/hfi1/qp.c                |   18 +-----
 drivers/infiniband/hw/hfi1/verbs.c             |    2 +
 drivers/infiniband/ulp/ipoib/ipoib_main.c      |    2 -
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |   11 ++-
 include/rdma/ib_verbs.h                        |   77 ++++++++++++++++++++++++
 include/rdma/opa_port_info.h                   |   10 ---
 6 files changed, 88 insertions(+), 32 deletions(-)
drivers/infiniband/hw/hfi1/qp.c +2 -16
index f8e733a..0c2ae9f 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2019 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -186,15 +186,6 @@ static void flush_iowait(struct rvt_qp *qp)
 	write_sequnlock_irqrestore(lock, flags);
 }
 
-static inline int opa_mtu_enum_to_int(int mtu)
-{
-	switch (mtu) {
-	case OPA_MTU_8192:  return 8192;
-	case OPA_MTU_10240: return 10240;
-	default:            return -1;
-	}
-}
-
 /**
  * This function is what we would push to the core layer if we wanted to be a
  * "first class citizen".  Instead we hide this here and rely on Verbs ULPs
@@ -202,15 +193,10 @@ static inline int opa_mtu_enum_to_int(int mtu)
  */
 static inline int verbs_mtu_enum_to_int(struct ib_device *dev, enum ib_mtu mtu)
 {
-	int val;
-
 	/* Constraining 10KB packets to 8KB packets */
 	if (mtu == (enum ib_mtu)OPA_MTU_10240)
 		mtu = OPA_MTU_8192;
-	val = opa_mtu_enum_to_int((int)mtu);
-	if (val > 0)
-		return val;
-	return ib_mtu_enum_to_int(mtu);
+	return opa_mtu_enum_to_int((enum opa_mtu)mtu);
 }
 
 int hfi1_check_modify_qp(struct rvt_qp *qp, struct ib_qp_attr *attr,
drivers/infiniband/hw/hfi1/verbs.c +2 -0
index c61b291..19d5d00 100644
@@ -1439,6 +1439,8 @@ static int query_port(struct rvt_dev_info *rdi, u8 port_num,
 				      4096 : hfi1_max_mtu), IB_MTU_4096);
 	props->active_mtu = !valid_ib_mtu(ppd->ibmtu) ? props->max_mtu :
 		mtu_to_enum(ppd->ibmtu, IB_MTU_4096);
+	props->phys_mtu = HFI1_CAP_IS_KSET(AIP) ? hfi1_max_mtu :
+				ib_mtu_enum_to_int(props->max_mtu);
 
 	return 0;
 }
drivers/infiniband/ulp/ipoib/ipoib_main.c +1 -1
index d4c6a97..22216f1 100644
@@ -1855,7 +1855,7 @@ static int ipoib_parent_init(struct net_device *ndev)
 			priv->port);
 		return result;
 	}
-	priv->max_ib_mtu = ib_mtu_enum_to_int(attr.max_mtu);
+	priv->max_ib_mtu = rdma_mtu_from_attr(priv->ca, priv->port, &attr);
 
 	result = ib_query_pkey(priv->ca, priv->port, 0, &priv->pkey);
 	if (result) {
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5 -6
index b9e9562..7166ee9b 100644
@@ -218,6 +218,7 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 	struct rdma_ah_attr av;
 	int ret;
 	int set_qkey = 0;
+	int mtu;
 
 	mcast->mcmember = *mcmember;
 
@@ -240,13 +241,11 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 		priv->broadcast->mcmember.flow_label = mcmember->flow_label;
 		priv->broadcast->mcmember.hop_limit = mcmember->hop_limit;
 		/* assume if the admin and the mcast are the same both can be changed */
+		mtu = rdma_mtu_enum_to_int(priv->ca,  priv->port,
+					   priv->broadcast->mcmember.mtu);
 		if (priv->mcast_mtu == priv->admin_mtu)
-			priv->admin_mtu =
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
-		else
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
+			priv->admin_mtu = IPOIB_UD_MTU(mtu);
+		priv->mcast_mtu = IPOIB_UD_MTU(mtu);
 
 		priv->qkey = be32_to_cpu(priv->broadcast->mcmember.qkey);
 		spin_unlock_irq(&priv->lock);
include/rdma/ib_verbs.h +77 -0
index 4b23ee5..792fb72 100644
@@ -462,6 +462,11 @@ enum ib_mtu {
 	IB_MTU_4096 = 5
 };
 
+enum opa_mtu {
+	OPA_MTU_8192 = 6,
+	OPA_MTU_10240 = 7
+};
+
 static inline int ib_mtu_enum_to_int(enum ib_mtu mtu)
 {
 	switch (mtu) {
@@ -488,6 +493,28 @@ static inline enum ib_mtu ib_mtu_int_to_enum(int mtu)
 		return IB_MTU_256;
 }
 
+static inline int opa_mtu_enum_to_int(enum opa_mtu mtu)
+{
+	switch (mtu) {
+	case OPA_MTU_8192:
+		return 8192;
+	case OPA_MTU_10240:
+		return 10240;
+	default:
+		return(ib_mtu_enum_to_int((enum ib_mtu)mtu));
+	}
+}
+
+static inline enum opa_mtu opa_mtu_int_to_enum(int mtu)
+{
+	if (mtu >= 10240)
+		return OPA_MTU_10240;
+	else if (mtu >= 8192)
+		return OPA_MTU_8192;
+	else
+		return ((enum opa_mtu)ib_mtu_int_to_enum(mtu));
+}
+
 enum ib_port_state {
 	IB_PORT_NOP		= 0,
 	IB_PORT_DOWN		= 1,
@@ -651,6 +678,7 @@ struct ib_port_attr {
 	enum ib_port_state	state;
 	enum ib_mtu		max_mtu;
 	enum ib_mtu		active_mtu;
+	u32                     phys_mtu;
 	int			gid_tbl_len;
 	unsigned int		ip_gids:1;
 	/* This is the value from PortInfo CapabilityMask, defined by IBA */
@@ -3364,6 +3392,55 @@ static inline unsigned int rdma_find_pg_bit(unsigned long addr,
 	return __fls(pgsz);
 }
 
+/**
+ * rdma_core_cap_opa_port - Return whether the RDMA Port is OPA or not.
+ * @device: Device
+ * @port_num: 1 based Port number
+ *
+ * Return true if port is an Intel OPA port , false if not
+ */
+static inline bool rdma_core_cap_opa_port(struct ib_device *device,
+					  u32 port_num)
+{
+	return (device->port_data[port_num].immutable.core_cap_flags &
+		RDMA_CORE_PORT_INTEL_OPA) == RDMA_CORE_PORT_INTEL_OPA;
+}
+
+/**
+ * rdma_mtu_enum_to_int - Return the mtu of the port as an integer value.
+ * @device: Device
+ * @port_num: Port number
+ * @mtu: enum value of MTU
+ *
+ * Return the MTU size supported by the port as an integer value. Will return
+ * -1 if enum value of mtu is not supported.
+ */
+static inline int rdma_mtu_enum_to_int(struct ib_device *device, u8 port,
+				       int mtu)
+{
+	if (rdma_core_cap_opa_port(device, port))
+		return opa_mtu_enum_to_int((enum opa_mtu)mtu);
+	else
+		return ib_mtu_enum_to_int((enum ib_mtu)mtu);
+}
+
+/**
+ * rdma_mtu_from_attr - Return the mtu of the port from the port attribute.
+ * @device: Device
+ * @port_num: Port number
+ * @attr: port attribute
+ *
+ * Return the MTU size supported by the port as an integer value.
+ */
+static inline int rdma_mtu_from_attr(struct ib_device *device, u8 port,
+				     struct ib_port_attr *attr)
+{
+	if (rdma_core_cap_opa_port(device, port))
+		return attr->phys_mtu;
+	else
+		return ib_mtu_enum_to_int(attr->max_mtu);
+}
+
 int ib_set_vf_link_state(struct ib_device *device, int vf, u8 port,
 			 int state);
 int ib_get_vf_config(struct ib_device *device, int vf, u8 port,
include/rdma/opa_port_info.h +1 -2
index bdbfe25..0d9e6d7 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2014-2017 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2014-2020 Intel Corporation.  All rights reserved.
  *
  * This software is available to you under a choice of one of two
  * licenses.  You may choose to be licensed under the terms of the GNU
@@ -139,14 +139,6 @@
 #define OPA_CAP_MASK3_IsVLMarkerSupported         (1 << 1)
 #define OPA_CAP_MASK3_IsVLrSupported              (1 << 0)
 
-/**

include/rdma/opa_port_info.h line 144

Quoted 10 lines
include/rdma/opa_port_info.h +0 -7
- * new MTU values
- */
-enum {
-	OPA_MTU_8192  = 6,
-	OPA_MTU_10240 = 7,
-};
-
 enum {
 	OPA_PORT_PHYS_CONF_DISCONNECTED = 0,
 	OPA_PORT_PHYS_CONF_STANDARD     = 1,

This patch introduces a new clang warning:

drivers/infiniband/hw/hfi1/qp.c:198:9: warning: implicit conversion from enumeration type 'enum opa_mtu' to different enumeration type 'enum ib_mtu' [-Wenum-conversion]

                mtu = OPA_MTU_8192;
                    ~ ^~~~~~~~~~~~

1 warning generated.

The simple solution is obviously to just remove the cast but then I noticed all of the explicit casts along with the casts in the conversion functions. What is the point of having these values as enums then explicitly casting them to others? Doesn't that just remove the type safety that enums offer? Wouldn't it just be better to have preprocessor defines? I noticed that OPA_MTU_0 to OPA_MTU_4096 are done that way but then OPA_MTU_8192 and OPA_MTU_10240 are defined as enumerators... I spent at least 15 minutes looking into how to untangle all of this and just decided to sent this email instead. I am happy to just send a patch shutting up this warning with a cast but I think the better long term solution is to just consolidate these enums/functions into something less unweildy.

Cheers,
Nathan

#9 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 280 lines · Nathan Chancellor
Quoted 248 lines · Dennis Dalessandro

From: Kaike Wan <[email protected]>

Currently the ipoib UD mtu is restricted to 4K bytes. Remove this limitation so that the IPOIB module can potentially use an MTU (in UD mode) that is bounded by the MTU of the underlying device. A field is added to the ib_port_attr structure to indicate the maximum physical MTU the underlying device supports.

---
  drivers/infiniband/hw/hfi1/qp.c                |   18 +-----
  drivers/infiniband/hw/hfi1/verbs.c             |    2 +
  drivers/infiniband/ulp/ipoib/ipoib_main.c      |    2 -
  drivers/infiniband/ulp/ipoib/ipoib_multicast.c |   11 ++-
  include/rdma/ib_verbs.h                        |   77 ++++++++++++++++++++++++
  include/rdma/opa_port_info.h                   |   10 ---
  6 files changed, 88 insertions(+), 32 deletions(-)
drivers/infiniband/hw/hfi1/qp.c +2 -16
index f8e733a..0c2ae9f 100644
@@ -1,5 +1,5 @@
  /*
- * Copyright(c) 2015 - 2019 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
   *
   * This file is provided under a dual BSD/GPLv2 license.  When using or
   * redistributing this file, you may do so under either license.
@@ -186,15 +186,6 @@ static void flush_iowait(struct rvt_qp *qp)
  	write_sequnlock_irqrestore(lock, flags);
  }
 
-static inline int opa_mtu_enum_to_int(int mtu)
-{
-	switch (mtu) {
-	case OPA_MTU_8192:  return 8192;
-	case OPA_MTU_10240: return 10240;
-	default:            return -1;
-	}
-}
-
  /**
   * This function is what we would push to the core layer if we wanted to be a
   * "first class citizen".  Instead we hide this here and rely on Verbs ULPs
@@ -202,15 +193,10 @@ static inline int opa_mtu_enum_to_int(int mtu)
   */
  static inline int verbs_mtu_enum_to_int(struct ib_device *dev, enum ib_mtu mtu)
  {
-	int val;
-
  	/* Constraining 10KB packets to 8KB packets */
  	if (mtu == (enum ib_mtu)OPA_MTU_10240)
  		mtu = OPA_MTU_8192;
-	val = opa_mtu_enum_to_int((int)mtu);
-	if (val > 0)
-		return val;
-	return ib_mtu_enum_to_int(mtu);
+	return opa_mtu_enum_to_int((enum opa_mtu)mtu);
  }
 
  int hfi1_check_modify_qp(struct rvt_qp *qp, struct ib_qp_attr *attr,
drivers/infiniband/hw/hfi1/verbs.c +2 -0
index c61b291..19d5d00 100644
@@ -1439,6 +1439,8 @@ static int query_port(struct rvt_dev_info *rdi, u8 port_num,
  				      4096 : hfi1_max_mtu), IB_MTU_4096);
  	props->active_mtu = !valid_ib_mtu(ppd->ibmtu) ? props->max_mtu :
  		mtu_to_enum(ppd->ibmtu, IB_MTU_4096);
+	props->phys_mtu = HFI1_CAP_IS_KSET(AIP) ? hfi1_max_mtu :
+				ib_mtu_enum_to_int(props->max_mtu);
 
  	return 0;
  }
drivers/infiniband/ulp/ipoib/ipoib_main.c +1 -1
index d4c6a97..22216f1 100644
@@ -1855,7 +1855,7 @@ static int ipoib_parent_init(struct net_device *ndev)
  			priv->port);
  		return result;
  	}
-	priv->max_ib_mtu = ib_mtu_enum_to_int(attr.max_mtu);
+	priv->max_ib_mtu = rdma_mtu_from_attr(priv->ca, priv->port, &attr);
 
  	result = ib_query_pkey(priv->ca, priv->port, 0, &priv->pkey);
  	if (result) {
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5 -6
index b9e9562..7166ee9b 100644
@@ -218,6 +218,7 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
  	struct rdma_ah_attr av;
  	int ret;
  	int set_qkey = 0;
+	int mtu;
 
  	mcast->mcmember = *mcmember;
 
@@ -240,13 +241,11 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
  		priv->broadcast->mcmember.flow_label = mcmember->flow_label;
  		priv->broadcast->mcmember.hop_limit = mcmember->hop_limit;
  		/* assume if the admin and the mcast are the same both can be changed */
+		mtu = rdma_mtu_enum_to_int(priv->ca,  priv->port,
+					   priv->broadcast->mcmember.mtu);
  		if (priv->mcast_mtu == priv->admin_mtu)
-			priv->admin_mtu =
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
-		else
-			priv->mcast_mtu =
-			IPOIB_UD_MTU(ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu));
+			priv->admin_mtu = IPOIB_UD_MTU(mtu);
+		priv->mcast_mtu = IPOIB_UD_MTU(mtu);
 
  		priv->qkey = be32_to_cpu(priv->broadcast->mcmember.qkey);
  		spin_unlock_irq(&priv->lock);
include/rdma/ib_verbs.h +77 -0
index 4b23ee5..792fb72 100644
@@ -462,6 +462,11 @@ enum ib_mtu {
  	IB_MTU_4096 = 5
  };
 
+enum opa_mtu {
+	OPA_MTU_8192 = 6,
+	OPA_MTU_10240 = 7
+};
+
  static inline int ib_mtu_enum_to_int(enum ib_mtu mtu)
  {
  	switch (mtu) {
@@ -488,6 +493,28 @@ static inline enum ib_mtu ib_mtu_int_to_enum(int mtu)
  		return IB_MTU_256;
  }
 
+static inline int opa_mtu_enum_to_int(enum opa_mtu mtu)
+{
+	switch (mtu) {
+	case OPA_MTU_8192:
+		return 8192;
+	case OPA_MTU_10240:
+		return 10240;
+	default:
+		return(ib_mtu_enum_to_int((enum ib_mtu)mtu));
+	}
+}
+
+static inline enum opa_mtu opa_mtu_int_to_enum(int mtu)
+{
+	if (mtu >= 10240)
+		return OPA_MTU_10240;
+	else if (mtu >= 8192)
+		return OPA_MTU_8192;
+	else
+		return ((enum opa_mtu)ib_mtu_int_to_enum(mtu));
+}
+
  enum ib_port_state {
  	IB_PORT_NOP		= 0,
  	IB_PORT_DOWN		= 1,
@@ -651,6 +678,7 @@ struct ib_port_attr {
  	enum ib_port_state	state;
  	enum ib_mtu		max_mtu;
  	enum ib_mtu		active_mtu;
+	u32                     phys_mtu;
  	int			gid_tbl_len;
  	unsigned int		ip_gids:1;
  	/* This is the value from PortInfo CapabilityMask, defined by IBA */
@@ -3364,6 +3392,55 @@ static inline unsigned int rdma_find_pg_bit(unsigned long addr,
  	return __fls(pgsz);
  }
 
+/**
+ * rdma_core_cap_opa_port - Return whether the RDMA Port is OPA or not.
+ * @device: Device
+ * @port_num: 1 based Port number
+ *
+ * Return true if port is an Intel OPA port , false if not
+ */
+static inline bool rdma_core_cap_opa_port(struct ib_device *device,
+					  u32 port_num)
+{
+	return (device->port_data[port_num].immutable.core_cap_flags &
+		RDMA_CORE_PORT_INTEL_OPA) == RDMA_CORE_PORT_INTEL_OPA;
+}
+
+/**
+ * rdma_mtu_enum_to_int - Return the mtu of the port as an integer value.
+ * @device: Device
+ * @port_num: Port number
+ * @mtu: enum value of MTU
+ *
+ * Return the MTU size supported by the port as an integer value. Will return
+ * -1 if enum value of mtu is not supported.
+ */
+static inline int rdma_mtu_enum_to_int(struct ib_device *device, u8 port,
+				       int mtu)
+{
+	if (rdma_core_cap_opa_port(device, port))
+		return opa_mtu_enum_to_int((enum opa_mtu)mtu);
+	else
+		return ib_mtu_enum_to_int((enum ib_mtu)mtu);
+}
+
+/**
+ * rdma_mtu_from_attr - Return the mtu of the port from the port attribute.
+ * @device: Device
+ * @port_num: Port number
+ * @attr: port attribute
+ *
+ * Return the MTU size supported by the port as an integer value.
+ */
+static inline int rdma_mtu_from_attr(struct ib_device *device, u8 port,
+				     struct ib_port_attr *attr)
+{
+	if (rdma_core_cap_opa_port(device, port))
+		return attr->phys_mtu;
+	else
+		return ib_mtu_enum_to_int(attr->max_mtu);
+}
+
  int ib_set_vf_link_state(struct ib_device *device, int vf, u8 port,
  			 int state);
  int ib_get_vf_config(struct ib_device *device, int vf, u8 port,
include/rdma/opa_port_info.h +1 -2
index bdbfe25..0d9e6d7 100644
@@ -1,5 +1,5 @@
  /*
- * Copyright (c) 2014-2017 Intel Corporation.  All rights reserved.
+ * Copyright (c) 2014-2020 Intel Corporation.  All rights reserved.
   *
   * This software is available to you under a choice of one of two
   * licenses.  You may choose to be licensed under the terms of the GNU
@@ -139,14 +139,6 @@
  #define OPA_CAP_MASK3_IsVLMarkerSupported         (1 << 1)
  #define OPA_CAP_MASK3_IsVLrSupported              (1 << 0)
 
-/**

include/rdma/opa_port_info.h line 144

Quoted 10 lines
include/rdma/opa_port_info.h +0 -7
- * new MTU values
- */
-enum {
-	OPA_MTU_8192  = 6,
-	OPA_MTU_10240 = 7,
-};
-
  enum {
  	OPA_PORT_PHYS_CONF_DISCONNECTED = 0,
  	OPA_PORT_PHYS_CONF_STANDARD     = 1,

This patch introduces a new clang warning:

drivers/infiniband/hw/hfi1/qp.c:198:9: warning: implicit conversion from enumeration type 'enum opa_mtu' to different enumeration type 'enum ib_mtu' [-Wenum-conversion]

                 mtu = OPA_MTU_8192;
                     ~ ^~~~~~~~~~~~

1 warning generated.

The simple solution is obviously to just remove the cast but then I noticed all of the explicit casts along with the casts in the conversion functions. What is the point of having these values as enums then explicitly casting them to others? Doesn't that just remove the type safety that enums offer? Wouldn't it just be better to have preprocessor defines? I noticed that OPA_MTU_0 to OPA_MTU_4096 are done that way but then OPA_MTU_8192 and OPA_MTU_10240 are defined as enumerators... I spent at least 15 minutes looking into how to untangle all of this and just decided to sent this email instead. I am happy to just send a patch shutting up this warning with a cast but I think the better long term solution is to just consolidate these enums/functions into something less unweildy.

Quoted 3 lines

Cheers,
Nathan

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

-Denny

#10 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

#11 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

Cheers,
Nathan

#12 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

We rather keep separation due to logic separation.

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

Thanks

Quoted 3 lines

Cheers,
Nathan

#13 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 22 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

Quoted 11 lines

We rather keep separation due to logic separation.

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

Thanks

Quoted 3 lines

Cheers,
Nathan

Fair enough, could someone take care of properly fixing the warning that was introduced by this patch then? I can send a patch that just adds an explicit cast but that looks like an eye sore to me. Otherwise, I am not familiar enough with this code to know what is an appropriate fix or not.

Cheers,
Nathan

#14 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 32 lines · Nathan Chancellor
Quoted 31 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

We rather keep separation due to logic separation.

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

Thanks

Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Cheers,
Nathan

Fair enough, could someone take care of properly fixing the warning that was introduced by this patch then? I can send a patch that just adds an explicit cast but that looks like an eye sore to me. Otherwise, I am not familiar enough with this code to know what is an appropriate fix or not.

Leon is correct. opa_* values should only be used on devices which are opa.

IPoIB needs to be 'opa aware' to do the right thing (be more efficient in this case) when running on an OPA device.

Is that the case with the current code?

Ira

Quoted 3 lines

Cheers,
Nathan

#15 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

In reply to Leon Romanovsky
Quoted 12 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines
Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

My opinion is yes.

Quoted 1 line

We rather keep separation due to logic separation.

To be fair, "you" rather. Not we. I'd like some others to weigh in here. Increasing the available MTUs an an enum just makes sense. Why does it matter if IB doesn't need them right now. Maybe someday.

Quoted 3 lines

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

But does it hurt to have more potentially available? Can you please explain the technical reason here?

-Denny

#16 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 31 lines · Dennis Dalessandro
Quoted 12 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines
Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

My opinion is yes.

Quoted 1 line

We rather keep separation due to logic separation.

To be fair, "you" rather. Not we. I'd like some others to weigh in here. Increasing the available MTUs an an enum just makes sense. Why does it matter if IB doesn't need them right now. Maybe someday.

Quoted 6 lines
Quoted 3 lines

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

But does it hurt to have more potentially available? Can you please explain the technical reason here?

The problem is that the IBTA _may_ define those enums to mean something different in the future. Hopefully, the Intel representatives within the IBTA would try to make them compatible but I don't know that 10K 'fits' with the IBTA's future. 8K seems like a reasonable extension for the IBTA but again we as Linux developers can't say that will happen for sure. We just don't control what the IBTA does in that regard.

Ira

Quoted 3 lines

-Denny

#17 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 39 lines · Ira Weiny
Quoted 31 lines · Dennis Dalessandro
Quoted 12 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines
Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

My opinion is yes.

Quoted 1 line

We rather keep separation due to logic separation.

To be fair, "you" rather. Not we. I'd like some others to weigh in here. Increasing the available MTUs an an enum just makes sense. Why does it matter if IB doesn't need them right now. Maybe someday.

Quoted 6 lines
Quoted 3 lines

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

But does it hurt to have more potentially available? Can you please explain the technical reason here?

Quoted 7 lines

The problem is that the IBTA _may_ define those enums to mean something different in the future. Hopefully, the Intel representatives within the IBTA would try to make them compatible but I don't know that 10K 'fits' with the IBTA's future. 8K seems like a reasonable extension for the IBTA but again we as Linux developers can't say that will happen for sure. We just don't control what the IBTA does in that regard.

I guess I buy that. However I believe I have seen it claimed in the past that we aren't the IBTA, we are the Linux kernel and can do what we think makes sense. But to be honest I don't feel strongly, and I'm not gonna argue strongly one way or the other.

-Denny

#18 lore raw

Re: [PATCH v3 for-next 07/16] IB/ipoib: Increase ipoib Datagram mode MTU's upper limit

Quoted 48 lines · Dennis Dalessandro
Quoted 39 lines · Ira Weiny
Quoted 31 lines · Dennis Dalessandro
Quoted 12 lines · Leon Romanovsky
Quoted 11 lines · Nathan Chancellor
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Dennis Dalessandro

They should probably all be in "enum ib_mtu". Jason any issues with us donig that? I can't for certain recall the real reason they were kept separate in the first place.

It is probably OK

Jason

Quoted 9 lines
Quoted 9 lines

I don't mind taking a wack at this if you guys are too busy (I'm rather tired of seeing the warning across all of my builds). However, I am wondering how far should this be unwound? Should 'enum opa_mtu' be collapsed into 'enum ib_mtu' and then all of the opa conversion functions be eliminated in favor of the ib ones? It looks like OPA_MTU_8192 and OPA_MTU_10240 are used in a few places within drivers/infiniband/hw/hfi1, should all of those instances be converted over to IB_MTU_* and the defines at the top of drivers/infiniband/hw/hfi1/hfi.h be eliminated?

My opinion is yes.

Quoted 1 line

We rather keep separation due to logic separation.

To be fair, "you" rather. Not we. I'd like some others to weigh in here. Increasing the available MTUs an an enum just makes sense. Why does it matter if IB doesn't need them right now. Maybe someday.

Quoted 6 lines
Quoted 3 lines

While ib_* defines come from IBTA and interoperable across different devices and vendors, opa_* definitions are Intel proprietary ones used for the product that was canceled.

But does it hurt to have more potentially available? Can you please explain the technical reason here?

Quoted 7 lines

The problem is that the IBTA _may_ define those enums to mean something different in the future. Hopefully, the Intel representatives within the IBTA would try to make them compatible but I don't know that 10K 'fits' with the IBTA's future. 8K seems like a reasonable extension for the IBTA but again we as Linux developers can't say that will happen for sure. We just don't control what the IBTA does in that regard.

Quoted 4 lines

I guess I buy that. However I believe I have seen it claimed in the past that we aren't the IBTA, we are the Linux kernel and can do what we think makes sense. But to be honest I don't feel strongly, and I'm not gonna argue strongly one way or the other.

Thanks Ira for the explanation.

Regarding "we are the Linux kernel and can do what we think makes sense" sentence, it is correct for SW interfaces and implementation only. Everything that touches already defined HW interfaces and wire protocol should follow the spec or should be separated.

Thanks

Quoted 3 lines

-Denny

#19 lore raw

PATCH v3 8/16 IB/hfi1: Rename num_vnic_contexts as num_netdev_contexts

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

Rename num_vnic_contexts as num_ndetdev_contexts since VNIC and ipoib will share the same set of receive contexts.

---
 drivers/infiniband/hw/hfi1/chip.c      |   24 ++++++++++++------------
 drivers/infiniband/hw/hfi1/hfi.h       |    4 ++--
 drivers/infiniband/hw/hfi1/msix.c      |    4 ++--
 drivers/infiniband/hw/hfi1/vnic_main.c |    8 ++++----
 4 files changed, 20 insertions(+), 20 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +12 -12
index be1fb29..07eec35 100644
@@ -13347,12 +13347,12 @@ static int set_up_interrupts(struct hfi1_devdata *dd)
  *                             in array of contexts
  *	freectxts  - number of free user contexts
  *	num_send_contexts - number of PIO send contexts being used
- *	num_vnic_contexts - number of contexts reserved for VNIC
+ *	num_netdev_contexts - number of contexts reserved for netdev
  */
 static int set_up_context_variables(struct hfi1_devdata *dd)
 {
 	unsigned long num_kernel_contexts;
-	u16 num_vnic_contexts = HFI1_NUM_VNIC_CTXT;
+	u16 num_netdev_contexts = HFI1_NUM_VNIC_CTXT;
 	int total_contexts;
 	int ret;
 	unsigned ngroups;
@@ -13391,11 +13391,11 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 	}
 
 	/* Accommodate VNIC contexts if possible */
-	if ((num_kernel_contexts + num_vnic_contexts) > rcv_contexts) {
+	if ((num_kernel_contexts + num_netdev_contexts) > rcv_contexts) {
 		dd_dev_err(dd, "No receive contexts available for VNIC\n");
-		num_vnic_contexts = 0;
+		num_netdev_contexts = 0;
 	}
-	total_contexts = num_kernel_contexts + num_vnic_contexts;
+	total_contexts = num_kernel_contexts + num_netdev_contexts;
 
 	/*
 	 * User contexts:
@@ -13422,15 +13422,15 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 	 * The RMT entries are currently allocated as shown below:
 	 * 1. QOS (0 to 128 entries);
 	 * 2. FECN (num_kernel_context - 1 + num_user_contexts +
-	 *    num_vnic_contexts);
-	 * 3. VNIC (num_vnic_contexts).
-	 * It should be noted that FECN oversubscribe num_vnic_contexts
-	 * entries of RMT because both VNIC and PSM could allocate any receive
+	 *    num_netdev_contexts);
+	 * 3. netdev (num_netdev_contexts).
+	 * It should be noted that FECN oversubscribe num_netdev_contexts
+	 * entries of RMT because both netdev and PSM could allocate any receive
 	 * context between dd->first_dyn_alloc_text and dd->num_rcv_contexts,
 	 * and PSM FECN must reserve an RMT entry for each possible PSM receive
 	 * context.
 	 */
-	rmt_count = qos_rmt_entries(dd, NULL, NULL) + (num_vnic_contexts * 2);
+	rmt_count = qos_rmt_entries(dd, NULL, NULL) + (num_netdev_contexts * 2);
 	if (HFI1_CAP_IS_KSET(TID_RDMA))
 		rmt_count += num_kernel_contexts - 1;
 	if (rmt_count + n_usr_ctxts > NUM_MAP_ENTRIES) {
@@ -13449,7 +13449,7 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 	dd->num_rcv_contexts = total_contexts;
 	dd->n_krcv_queues = num_kernel_contexts;
 	dd->first_dyn_alloc_ctxt = num_kernel_contexts;
-	dd->num_vnic_contexts = num_vnic_contexts;
+	dd->num_netdev_contexts = num_netdev_contexts;
 	dd->num_user_contexts = n_usr_ctxts;
 	dd->freectxts = n_usr_ctxts;
 	dd_dev_info(dd,
@@ -13457,7 +13457,7 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 		    rcv_contexts,
 		    (int)dd->num_rcv_contexts,
 		    (int)dd->n_krcv_queues,
-		    dd->num_vnic_contexts,
+		    dd->num_netdev_contexts,
 		    dd->num_user_contexts);
 
 	/*
drivers/infiniband/hw/hfi1/hfi.h +2 -2
index c61e56a..5a9276c 100644
@@ -1167,8 +1167,8 @@ struct hfi1_devdata {
 	u64 z_send_schedule;
 
 	u64 __percpu *send_schedule;
-	/* number of reserved contexts for VNIC usage */
-	u16 num_vnic_contexts;
+	/* number of reserved contexts for netdev usage */
+	u16 num_netdev_contexts;
 	/* number of receive contexts in use by the driver */
 	u32 num_rcv_contexts;
 	/* number of pio send contexts in use by the driver */
drivers/infiniband/hw/hfi1/msix.c +2 -2
index db82db4..7574f2b 100644
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
 /*
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -69,7 +69,7 @@ int msix_initialize(struct hfi1_devdata *dd)
 	 *	one for each VNIC context
 	 *      ...any new IRQs should be added here.
 	 */
-	total = 1 + dd->num_sdma + dd->n_krcv_queues + dd->num_vnic_contexts;
+	total = 1 + dd->num_sdma + dd->n_krcv_queues + dd->num_netdev_contexts;
 
 	if (total >= CCE_NUM_MSIX_VECTORS)
 		return -EINVAL;
drivers/infiniband/hw/hfi1/vnic_main.c +4 -4
index 6b14581..db7624c 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2017 - 2018 Intel Corporation.
+ * Copyright(c) 2017 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -804,7 +804,7 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 	struct rdma_netdev *rn;
 	int i, size, rc;
 
-	if (!dd->num_vnic_contexts)
+	if (!dd->num_netdev_contexts)
 		return ERR_PTR(-ENOMEM);
 
 	if (!port_num || (port_num > dd->num_pports))
@@ -815,7 +815,7 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 
 	size = sizeof(struct opa_vnic_rdma_netdev) + sizeof(*vinfo);
 	netdev = alloc_netdev_mqs(size, name, name_assign_type, setup,
-				  dd->num_sdma, dd->num_vnic_contexts);
+				  dd->num_sdma, dd->num_netdev_contexts);
 	if (!netdev)
 		return ERR_PTR(-ENOMEM);
 
@@ -823,7 +823,7 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 	vinfo = opa_vnic_dev_priv(netdev);
 	vinfo->dd = dd;
 	vinfo->num_tx_q = dd->num_sdma;
-	vinfo->num_rx_q = dd->num_vnic_contexts;
+	vinfo->num_rx_q = dd->num_netdev_contexts;
 	vinfo->netdev = netdev;
 	rn->free_rdma_netdev = hfi1_vnic_free_rn;
 	rn->set_id = hfi1_vnic_set_vesw_id;
#20 lore raw

PATCH v3 9/16 IB/hfi1: Add functions to receive accelerated ipoib packets

In reply to Dennis Dalessandro

From: Kaike Wan <[email protected]>

Ipoib netdev will share receive contexts with existing VNIC netdev. To achieve that, a dummy netdev is allocated with hfi1_devdata to own the receive contexts, and ipoib and VNIC netdevs will be put on top of it. Each receive context is associated with a single NAPI object.

This patch adds the functions to receive incoming packets for accelerated ipoib.

---
 drivers/infiniband/hw/hfi1/Makefile    |    2 +
 drivers/infiniband/hw/hfi1/driver.c    |   92 ++++++++++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/hfi.h       |    5 +-
 drivers/infiniband/hw/hfi1/ipoib.h     |   18 ++++++
 drivers/infiniband/hw/hfi1/ipoib_rx.c  |   71 +++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/netdev.h    |   90 +++++++++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/netdev_rx.c |   79 +++++++++++++++++++++++++++
 7 files changed, 355 insertions(+), 2 deletions(-)
 create mode 100644 drivers/infiniband/hw/hfi1/ipoib_rx.c
 create mode 100644 drivers/infiniband/hw/hfi1/netdev.h
 create mode 100644 drivers/infiniband/hw/hfi1/netdev_rx.c
drivers/infiniband/hw/hfi1/Makefile +2 -0
index 0b25713..2e89ec1 100644
@@ -23,10 +23,12 @@ hfi1-y := \
 	intr.o \
 	iowait.o \
 	ipoib_main.o \
+	ipoib_rx.o \
 	ipoib_tx.o \
 	mad.o \
 	mmu_rb.o \
 	msix.o \
+	netdev_rx.o \
 	opfn.o \
 	pcie.o \
 	pio.o \
drivers/infiniband/hw/hfi1/driver.c +91 -1

117 lines not on this page · raw message

drivers/infiniband/hw/hfi1/hfi.h +4 -1
index 5a9276c..c7d0aad 100644
@@ -233,6 +233,8 @@ struct hfi1_ctxtdata {
 	intr_handler fast_handler;
 	/** slow handler */
 	intr_handler slow_handler;
+	/* napi pointer assiociated with netdev */
+	struct napi_struct *napi;
 	/* verbs rx_stats per rcd */
 	struct hfi1_opcode_stats_perctx *opstats;
 	/* clear interrupt mask */
@@ -985,7 +987,7 @@ typedef void (*hfi1_make_req)(struct rvt_qp *qp,
 			      struct hfi1_pkt_state *ps,
 			      struct rvt_swqe *wqe);
 extern const rhf_rcv_function_ptr normal_rhf_rcv_functions[];
-
+extern const rhf_rcv_function_ptr netdev_rhf_rcv_functions[];
 
 /* return values for the RHF receive functions */
 #define RHF_RCV_CONTINUE  0	/* keep going */
@@ -1417,6 +1419,7 @@ struct hfi1_devdata {
 	struct hfi1_vnic_data vnic;
 	/* Lock to protect IRQ SRC register access */
 	spinlock_t irq_src_lock;
+	struct net_device *dummy_netdev;
 
 	/* Keeps track of IPoIB RSM rule users */
 	atomic_t ipoib_rsm_usr_num;
drivers/infiniband/hw/hfi1/ipoib.h +18 -0
index c2e63ca..ca00f6c 100644
@@ -22,6 +22,7 @@
 
 #include "hfi.h"
 #include "iowait.h"
+#include "netdev.h"
 
 #include <rdma/ib_verbs.h>
 
@@ -29,6 +30,7 @@
 
 #define HFI1_IPOIB_TXREQ_NAME_LEN   32
 
+#define HFI1_IPOIB_PSEUDO_LEN 20
 #define HFI1_IPOIB_ENCAP_LEN 4
 
 struct hfi1_ipoib_dev_priv;
@@ -119,6 +121,19 @@ struct hfi1_ipoib_rdma_netdev {
 }
 
 static inline void
+hfi1_ipoib_update_rx_netstats(struct hfi1_ipoib_dev_priv *priv,
+			      u64 packets,
+			      u64 bytes)
+{
+	struct pcpu_sw_netstats *netstats = this_cpu_ptr(priv->netstats);
+
+	u64_stats_update_begin(&netstats->syncp);
+	netstats->rx_packets += packets;
+	netstats->rx_bytes += bytes;
+	u64_stats_update_end(&netstats->syncp);
+}
+
+static inline void
 hfi1_ipoib_update_tx_netstats(struct hfi1_ipoib_dev_priv *priv,
 			      u64 packets,
 			      u64 bytes)
@@ -142,6 +157,9 @@ int hfi1_ipoib_send_dma(struct net_device *dev,
 void hfi1_ipoib_napi_tx_enable(struct net_device *dev);
 void hfi1_ipoib_napi_tx_disable(struct net_device *dev);
 
+struct sk_buff *hfi1_ipoib_prepare_skb(struct hfi1_netdev_rxq *rxq,
+				       int size, void *data);
+
 int hfi1_ipoib_rn_get_params(struct ib_device *device,
 			     u8 port_num,
 			     enum rdma_netdev_t type,
drivers/infiniband/hw/hfi1/ipoib_rx.c new file +71 -0

71 lines not on this page · raw message

drivers/infiniband/hw/hfi1/netdev.h new file +90 -0

90 lines not on this page · raw message

drivers/infiniband/hw/hfi1/netdev_rx.c new file +79 -0

79 lines not on this page · raw message

#21 lore raw

PATCH v3 10/16 IB/hfi1: Add interrupt handler functions for accelerated ipoib

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

This patch adds the interrupt handler function, the NAPI poll function, and its associated helper functions for receiving accelerated ipoib packets. While we are here, fix the formats of two error printouts.

---
 drivers/infiniband/hw/hfi1/affinity.c |   12 +++
 drivers/infiniband/hw/hfi1/affinity.h |    3 +
 drivers/infiniband/hw/hfi1/chip.c     |   44 ++++++++++++
 drivers/infiniband/hw/hfi1/chip.h     |    1 
 drivers/infiniband/hw/hfi1/driver.c   |  120 +++++++++++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/hfi.h      |    4 +
 drivers/infiniband/hw/hfi1/init.c     |    1 
 drivers/infiniband/hw/hfi1/msix.c     |   20 +++++-
 drivers/infiniband/hw/hfi1/msix.h     |    5 +
 drivers/infiniband/hw/hfi1/netdev.h   |    3 +
 10 files changed, 206 insertions(+), 7 deletions(-)
drivers/infiniband/hw/hfi1/affinity.c +11 -1
index 1aeea5d..2a91b8d 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -64,6 +64,7 @@ struct hfi1_affinity_node_list node_affinity = {
 static const char * const irq_type_names[] = {
 	"SDMA",
 	"RCVCTXT",
+	"NETDEVCTXT",
 	"GENERAL",
 	"OTHER",
 };
@@ -915,6 +916,11 @@ static int get_irq_affinity(struct hfi1_devdata *dd,
 			set = &entry->rcv_intr;
 		scnprintf(extra, 64, "ctxt %u", rcd->ctxt);
 		break;
+	case IRQ_NETDEVCTXT:
+		rcd = (struct hfi1_ctxtdata *)msix->arg;
+		set = &entry->def_intr;
+		scnprintf(extra, 64, "ctxt %u", rcd->ctxt);
+		break;
 	default:
 		dd_dev_err(dd, "Invalid IRQ type %d\n", msix->type);
 		return -EINVAL;
@@ -987,6 +993,10 @@ void hfi1_put_irq_affinity(struct hfi1_devdata *dd,
 		if (rcd->ctxt != HFI1_CTRL_CTXT)
 			set = &entry->rcv_intr;
 		break;
+	case IRQ_NETDEVCTXT:
+		rcd = (struct hfi1_ctxtdata *)msix->arg;
+		set = &entry->def_intr;
+		break;
 	default:
 		mutex_unlock(&node_affinity.lock);
 		return;
drivers/infiniband/hw/hfi1/affinity.h +2 -1
index 6a7e6ea..f94ed5d 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015 - 2018 Intel Corporation.
+ * Copyright(c) 2015 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -52,6 +52,7 @@
 enum irq_type {
 	IRQ_SDMA,
 	IRQ_RCVCTXT,
+	IRQ_NETDEVCTXT,
 	IRQ_GENERAL,
 	IRQ_OTHER
 };
drivers/infiniband/hw/hfi1/chip.c +44 -0

56 lines not on this page · raw message

drivers/infiniband/hw/hfi1/chip.h +1 -0
index b10e0bf..2c6f2de 100644
@@ -1447,6 +1447,7 @@ int hfi1_set_ctxt_pkey(struct hfi1_devdata *dd, struct hfi1_ctxtdata *ctxt,
 irqreturn_t sdma_interrupt(int irq, void *data);
 irqreturn_t receive_context_interrupt(int irq, void *data);
 irqreturn_t receive_context_thread(int irq, void *data);
+irqreturn_t receive_context_interrupt_napi(int irq, void *data);
 
 int set_intr_bits(struct hfi1_devdata *dd, u16 first, u16 last, bool set);
 void init_qsfp_int(struct hfi1_devdata *dd);
drivers/infiniband/hw/hfi1/driver.c +120 -0

138 lines not on this page · raw message

drivers/infiniband/hw/hfi1/hfi.h +3 -1
index c7d0aad..986d8c3 100644
@@ -385,11 +385,11 @@ struct hfi1_packet {
 	u32 rhqoff;
 	u32 dlid;
 	u32 slid;
+	int numpkt;
 	u16 tlen;
 	s16 etail;
 	u16 pkey;
 	u8 hlen;
-	u8 numpkt;
 	u8 rsize;
 	u8 updegr;
 	u8 etype;
@@ -1501,6 +1501,8 @@ struct hfi1_ctxtdata *hfi1_rcd_get_by_index_safe(struct hfi1_devdata *dd,
 int handle_receive_interrupt(struct hfi1_ctxtdata *rcd, int thread);
 int handle_receive_interrupt_nodma_rtail(struct hfi1_ctxtdata *rcd, int thread);
 int handle_receive_interrupt_dma_rtail(struct hfi1_ctxtdata *rcd, int thread);
+int handle_receive_interrupt_napi_fp(struct hfi1_ctxtdata *rcd, int budget);
+int handle_receive_interrupt_napi_sp(struct hfi1_ctxtdata *rcd, int budget);
 void set_all_slowpath(struct hfi1_devdata *dd);
 
 extern const struct pci_device_id hfi1_pci_tbl[];
drivers/infiniband/hw/hfi1/init.c +1 -0
index 8c6b96a..64279d0 100644
@@ -374,6 +374,7 @@ int hfi1_create_ctxtdata(struct hfi1_pportdata *ppd, int numa,
 		rcd->numa_id = numa;
 		rcd->rcv_array_groups = dd->rcv_entries.ngroups;
 		rcd->rhf_rcv_function_map = normal_rhf_rcv_functions;
+		rcd->msix_intr = CCE_NUM_MSIX_VECTORS;
 
 		mutex_init(&rcd->exp_mutex);
 		spin_lock_init(&rcd->exp_lock);
drivers/infiniband/hw/hfi1/msix.c +18 -2
index 7574f2b..7559875e 100644
@@ -49,6 +49,7 @@
 #include "hfi.h"
 #include "affinity.h"
 #include "sdma.h"
+#include "netdev.h"
 
 /**
  * msix_initialize() - Calculate, request and configure MSIx IRQs
@@ -140,7 +141,7 @@ static int msix_request_irq(struct hfi1_devdata *dd, void *arg,
 	ret = pci_request_irq(dd->pcidev, nr, handler, thread, arg, name);
 	if (ret) {
 		dd_dev_err(dd,
-			   "%s: request for IRQ %d failed, MSIx %lu, err %d\n",
+			   "%s: request for IRQ %d failed, MSIx %lx, err %d\n",
 			   name, irq, nr, ret);
 		spin_lock(&dd->msix_info.msix_lock);
 		__clear_bit(nr, dd->msix_info.in_use_msix);
@@ -160,7 +161,7 @@ static int msix_request_irq(struct hfi1_devdata *dd, void *arg,
 	/* This is a request, so a failure is not fatal */
 	ret = hfi1_get_irq_affinity(dd, me);
 	if (ret)
-		dd_dev_err(dd, "unable to pin IRQ %d\n", ret);
+		dd_dev_err(dd, "%s: unable to pin IRQ %d\n", name, ret);
 
 	return nr;
 }
@@ -204,6 +205,21 @@ int msix_request_rcd_irq(struct hfi1_ctxtdata *rcd)
 }
 
 /**
+ * msix_request_rcd_irq() - Helper function for RCVAVAIL IRQs
+ * for netdev context
+ * @rcd: valid netdev contexti
+ */
+int msix_netdev_request_rcd_irq(struct hfi1_ctxtdata *rcd)
+{
+	char name[MAX_NAME_SIZE];
+
+	snprintf(name, sizeof(name), DRIVER_NAME "_%d nd kctxt%d",
+		 rcd->dd->unit, rcd->ctxt);
+	return msix_request_rcd_irq_common(rcd, receive_context_interrupt_napi,
+					   NULL, name);
+}
+
+/**
  * msix_request_smda_ira() - Helper for getting SDMA IRQ resources
  * @sde: valid sdma engine
  *
drivers/infiniband/hw/hfi1/msix.h +3 -2
index 1a02ab7..42fab94 100644
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */
 /*
- * Copyright(c) 2018 Intel Corporation.
+ * Copyright(c) 2018 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -59,7 +59,8 @@
 int msix_request_sdma_irq(struct sdma_engine *sde);
 void msix_free_irq(struct hfi1_devdata *dd, u8 msix_intr);
 
-/* VNIC interface */
+/* Netdev interface */
 void msix_vnic_synchronize_irq(struct hfi1_devdata *dd);
+int msix_netdev_request_rcd_irq(struct hfi1_ctxtdata *rcd);
 
 #endif
drivers/infiniband/hw/hfi1/netdev.h +3 -0
index 8992dfe..6740ec3 100644
@@ -87,4 +87,7 @@ struct hfi1_ctxtdata *hfi1_netdev_get_ctxt(struct hfi1_devdata *dd, int ctxt)
 void *hfi1_netdev_get_data(struct hfi1_devdata *dd, int id);
 void *hfi1_netdev_get_first_data(struct hfi1_devdata *dd, int *start_id);
 
+/* chip.c  */
+int hfi1_netdev_rx_napi(struct napi_struct *napi, int budget);
+
 #endif /* HFI1_NETDEV_H */
#22 lore raw

PATCH v3 11/16 IB/hfi1: Add rx functions for dummy netdev

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

This patch adds the rx functions for the dummy netdev:
- Functions to allocate/free the dummy netdev.
- Functions to allocate/free receiving contexts for the netdev.
- Functions to initialize/de-initialize the receive queue.
- Functions to enable/disable the receive queue.

---
 drivers/infiniband/hw/hfi1/ipoib.h      |    3 
 drivers/infiniband/hw/hfi1/ipoib_main.c |   30 ++-
 drivers/infiniband/hw/hfi1/ipoib_rx.c   |   16 +
 drivers/infiniband/hw/hfi1/netdev.h     |    6 +
 drivers/infiniband/hw/hfi1/netdev_rx.c  |  361 +++++++++++++++++++++++++++++++
 5 files changed, 414 insertions(+), 2 deletions(-)
drivers/infiniband/hw/hfi1/ipoib.h +3 -0
index ca00f6c..185c9b0 100644
@@ -154,6 +154,9 @@ int hfi1_ipoib_send_dma(struct net_device *dev,
 int hfi1_ipoib_txreq_init(struct hfi1_ipoib_dev_priv *priv);
 void hfi1_ipoib_txreq_deinit(struct hfi1_ipoib_dev_priv *priv);
 
+int hfi1_ipoib_rxq_init(struct net_device *dev);
+void hfi1_ipoib_rxq_deinit(struct net_device *dev);
+
 void hfi1_ipoib_napi_tx_enable(struct net_device *dev);
 void hfi1_ipoib_napi_tx_disable(struct net_device *dev);
 
drivers/infiniband/hw/hfi1/ipoib_main.c +28 -2
index 304a5ac..014351e 100644
@@ -19,16 +19,31 @@ static u32 qpn_from_mac(u8 *mac_arr)
 static int hfi1_ipoib_dev_init(struct net_device *dev)
 {
 	struct hfi1_ipoib_dev_priv *priv = hfi1_ipoib_priv(dev);
+	int ret;
 
 	priv->netstats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats);
 
-	return priv->netdev_ops->ndo_init(dev);
+	ret = priv->netdev_ops->ndo_init(dev);
+	if (ret)
+		return ret;
+
+	ret = hfi1_netdev_add_data(priv->dd,
+				   qpn_from_mac(priv->netdev->dev_addr),
+				   dev);
+	if (ret < 0) {
+		priv->netdev_ops->ndo_uninit(dev);
+		return ret;
+	}
+
+	return 0;
 }
 
 static void hfi1_ipoib_dev_uninit(struct net_device *dev)
 {
 	struct hfi1_ipoib_dev_priv *priv = hfi1_ipoib_priv(dev);
 
+	hfi1_netdev_remove_data(priv->dd, qpn_from_mac(priv->netdev->dev_addr));
+
 	priv->netdev_ops->ndo_uninit(dev);
 }
 
@@ -55,6 +70,7 @@ static int hfi1_ipoib_dev_open(struct net_device *dev)
 		priv->qp = qp;
 		rcu_read_unlock();
 
+		hfi1_netdev_enable_queues(priv->dd);
 		hfi1_ipoib_napi_tx_enable(dev);
 	}
 
@@ -69,6 +85,7 @@ static int hfi1_ipoib_dev_stop(struct net_device *dev)
 		return 0;
 
 	hfi1_ipoib_napi_tx_disable(dev);
+	hfi1_netdev_disable_queues(priv->dd);
 
 	rvt_put_qp(priv->qp);
 	priv->qp = NULL;
@@ -195,6 +212,7 @@ static void hfi1_ipoib_netdev_dtor(struct net_device *dev)
 	struct hfi1_ipoib_dev_priv *priv = hfi1_ipoib_priv(dev);
 
 	hfi1_ipoib_txreq_deinit(priv);
+	hfi1_ipoib_rxq_deinit(priv->netdev);
 
 	free_percpu(priv->netstats);
 }
@@ -252,6 +270,13 @@ static int hfi1_ipoib_setup_rn(struct ib_device *device,
 		return rc;
 	}
 
+	rc = hfi1_ipoib_rxq_init(netdev);
+	if (rc) {
+		dd_dev_err(dd, "IPoIB netdev RX init - failed(%d)\n", rc);
+		hfi1_ipoib_free_rdma_netdev(netdev);
+		return rc;
+	}
+
 	netdev->priv_destructor = hfi1_ipoib_netdev_dtor;
 	netdev->needs_free_netdev = true;
 
@@ -268,7 +293,7 @@ int hfi1_ipoib_rn_get_params(struct ib_device *device,
 	if (type != RDMA_NETDEV_IPOIB)
 		return -EOPNOTSUPP;
 
-	if (!HFI1_CAP_IS_KSET(AIP))
+	if (!HFI1_CAP_IS_KSET(AIP) || !dd->num_netdev_contexts)
 		return -EOPNOTSUPP;
 
 	if (!port_num || port_num > dd->num_pports)
@@ -276,6 +301,7 @@ int hfi1_ipoib_rn_get_params(struct ib_device *device,
 
 	params->sizeof_priv = sizeof(struct hfi1_ipoib_rdma_netdev);
 	params->txqs = dd->num_sdma;
+	params->rxqs = dd->num_netdev_contexts;
 	params->param = NULL;
 	params->initialize_rdma_netdev = hfi1_ipoib_setup_rn;
 
drivers/infiniband/hw/hfi1/ipoib_rx.c +16 -0
index 2485663..606ac69 100644
@@ -69,3 +69,19 @@ struct sk_buff *hfi1_ipoib_prepare_skb(struct hfi1_netdev_rxq *rxq,
 
 	return skb;
 }
+
+int hfi1_ipoib_rxq_init(struct net_device *netdev)
+{
+	struct hfi1_ipoib_dev_priv *ipoib_priv = hfi1_ipoib_priv(netdev);
+	struct hfi1_devdata *dd = ipoib_priv->dd;
+
+	return hfi1_netdev_rx_init(dd);
+}
+
+void hfi1_ipoib_rxq_deinit(struct net_device *netdev)
+{
+	struct hfi1_ipoib_dev_priv *ipoib_priv = hfi1_ipoib_priv(netdev);
+	struct hfi1_devdata *dd = ipoib_priv->dd;
+
+	hfi1_netdev_rx_destroy(dd);
+}
drivers/infiniband/hw/hfi1/netdev.h +6 -0
index 6740ec3..edb936f 100644
@@ -82,6 +82,12 @@ struct hfi1_ctxtdata *hfi1_netdev_get_ctxt(struct hfi1_devdata *dd, int ctxt)
 	return priv->rxq[ctxt].rcd;
 }
 
+void hfi1_netdev_enable_queues(struct hfi1_devdata *dd);
+void hfi1_netdev_disable_queues(struct hfi1_devdata *dd);
+int hfi1_netdev_rx_init(struct hfi1_devdata *dd);
+int hfi1_netdev_rx_destroy(struct hfi1_devdata *dd);
+int hfi1_netdev_alloc(struct hfi1_devdata *dd);
+void hfi1_netdev_free(struct hfi1_devdata *dd);
 int hfi1_netdev_add_data(struct hfi1_devdata *dd, int id, void *data);
 void *hfi1_netdev_remove_data(struct hfi1_devdata *dd, int id);
 void *hfi1_netdev_get_data(struct hfi1_devdata *dd, int id);
drivers/infiniband/hw/hfi1/netdev_rx.c +361 -0

367 lines not on this page · raw message

#23 lore raw

PATCH v3 12/16 IB/hfi1: Activate the dummy netdev

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

As described in earlier patches, ipoib netdev will share receive contexts with existing VNIC netdev through a dummy netdev. The following changes are made to achieve that:
- Set up netdev receive contexts after user contexts. A function is added to count the available netdev receive contexts.
- Add functions to set/get receive map table free index.
- Rename NUM_VNIC_MAP_ENTRIES as NUM_NETDEV_MAP_ENTRIES.
- Let the dummy netdev own the receive contexts instead of VNIC.
- Allocate the dummy netdev when the hfi1 device is added and free it when the device is removed.
- Initialize AIP RSM rules when the IpoIb rxq is initialized and remove the rules when it is de-initialized.
- Convert VNIC to use the dummy netdev.

---
 drivers/infiniband/hw/hfi1/chip.c      |   73 ++++---
 drivers/infiniband/hw/hfi1/driver.c    |   18 --
 drivers/infiniband/hw/hfi1/hfi.h       |   14 -
 drivers/infiniband/hw/hfi1/init.c      |    9 -
 drivers/infiniband/hw/hfi1/ipoib_rx.c  |   10 +
 drivers/infiniband/hw/hfi1/msix.c      |   12 +
 drivers/infiniband/hw/hfi1/msix.h      |    2 
 drivers/infiniband/hw/hfi1/netdev.h    |   19 ++
 drivers/infiniband/hw/hfi1/netdev_rx.c |   46 +++++
 drivers/infiniband/hw/hfi1/vnic.h      |    5 -
 drivers/infiniband/hw/hfi1/vnic_main.c |  312 +++++---------------------------
 11 files changed, 178 insertions(+), 342 deletions(-)
drivers/infiniband/hw/hfi1/chip.c +38 -35
index 2117612..7f35b9e 100644
@@ -13396,8 +13396,7 @@ static int set_up_interrupts(struct hfi1_devdata *dd)
 static int set_up_context_variables(struct hfi1_devdata *dd)
 {
 	unsigned long num_kernel_contexts;
-	u16 num_netdev_contexts = HFI1_NUM_VNIC_CTXT;
-	int total_contexts;
+	u16 num_netdev_contexts;
 	int ret;
 	unsigned ngroups;
 	int rmt_count;
@@ -13434,13 +13433,6 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 		num_kernel_contexts = send_contexts - num_vls - 1;
 	}
 
-	/* Accommodate VNIC contexts if possible */
-	if ((num_kernel_contexts + num_netdev_contexts) > rcv_contexts) {
-		dd_dev_err(dd, "No receive contexts available for VNIC\n");
-		num_netdev_contexts = 0;
-	}
-	total_contexts = num_kernel_contexts + num_netdev_contexts;
-
 	/*
 	 * User contexts:
 	 *	- default to 1 user context per real (non-HT) CPU core if
@@ -13453,15 +13445,19 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 	/*
 	 * Adjust the counts given a global max.
 	 */
-	if (total_contexts + n_usr_ctxts > rcv_contexts) {
+	if (num_kernel_contexts + n_usr_ctxts > rcv_contexts) {
 		dd_dev_err(dd,
-			   "Reducing # user receive contexts to: %d, from %u\n",
-			   rcv_contexts - total_contexts,
+			   "Reducing # user receive contexts to: %u, from %u\n",
+			   (u32)(rcv_contexts - num_kernel_contexts),
 			   n_usr_ctxts);
 		/* recalculate */
-		n_usr_ctxts = rcv_contexts - total_contexts;
+		n_usr_ctxts = rcv_contexts - num_kernel_contexts;
 	}
 
+	num_netdev_contexts =
+		hfi1_num_netdev_contexts(dd, rcv_contexts -
+					 (num_kernel_contexts + n_usr_ctxts),
+					 &node_affinity.real_cpu_mask);
 	/*
 	 * The RMT entries are currently allocated as shown below:
 	 * 1. QOS (0 to 128 entries);
@@ -13487,17 +13483,16 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 		n_usr_ctxts = user_rmt_reduced;
 	}
 
-	total_contexts += n_usr_ctxts;
-
-	/* the first N are kernel contexts, the rest are user/vnic contexts */
-	dd->num_rcv_contexts = total_contexts;
+	/* the first N are kernel contexts, the rest are user/netdev contexts */
+	dd->num_rcv_contexts =
+		num_kernel_contexts + n_usr_ctxts + num_netdev_contexts;
 	dd->n_krcv_queues = num_kernel_contexts;
 	dd->first_dyn_alloc_ctxt = num_kernel_contexts;
 	dd->num_netdev_contexts = num_netdev_contexts;
 	dd->num_user_contexts = n_usr_ctxts;
 	dd->freectxts = n_usr_ctxts;
 	dd_dev_info(dd,
-		    "rcv contexts: chip %d, used %d (kernel %d, vnic %u, user %u)\n",
+		    "rcv contexts: chip %d, used %d (kernel %d, netdev %u, user %u)\n",
 		    rcv_contexts,
 		    (int)dd->num_rcv_contexts,
 		    (int)dd->n_krcv_queues,
@@ -14554,7 +14549,8 @@ static bool hfi1_netdev_update_rmt(struct hfi1_devdata *dd)
 	u8 ctx_id = 0;
 	u64 reg;
 	u32 regoff;
-	int rmt_start = dd->vnic.rmt_start;
+	int rmt_start = hfi1_netdev_get_free_rmt_idx(dd);
+	int ctxt_count = hfi1_netdev_ctxt_count(dd);
 
 	/* We already have contexts mapped in RMT */
 	if (has_rsm_rule(dd, RSM_INS_VNIC) || has_rsm_rule(dd, RSM_INS_AIP)) {
@@ -14562,7 +14558,7 @@ static bool hfi1_netdev_update_rmt(struct hfi1_devdata *dd)
 		return true;
 	}
 
-	if (hfi1_is_rmt_full(rmt_start, NUM_VNIC_MAP_ENTRIES)) {
+	if (hfi1_is_rmt_full(rmt_start, NUM_NETDEV_MAP_ENTRIES)) {
 		dd_dev_err(dd, "Not enought RMT entries used = %d\n",
 			   rmt_start);
 		return false;
@@ -14570,27 +14566,27 @@ static bool hfi1_netdev_update_rmt(struct hfi1_devdata *dd)
 
 	dev_dbg(&(dd)->pcidev->dev, "RMT start = %d, end %d\n",
 		rmt_start,
-		rmt_start + NUM_VNIC_MAP_ENTRIES);
+		rmt_start + NUM_NETDEV_MAP_ENTRIES);
 
 	/* Update RSM mapping table, 32 regs, 256 entries - 1 ctx per byte */
 	regoff = RCV_RSM_MAP_TABLE + (rmt_start / 8) * 8;
 	reg = read_csr(dd, regoff);
-	for (i = 0; i < NUM_VNIC_MAP_ENTRIES; i++) {
+	for (i = 0; i < NUM_NETDEV_MAP_ENTRIES; i++) {
 		/* Update map register with netdev context */
 		j = (rmt_start + i) % 8;
 		reg &= ~(0xffllu << (j * 8));
-		reg |= (u64)dd->vnic.ctxt[ctx_id++]->ctxt << (j * 8);
+		reg |= (u64)hfi1_netdev_get_ctxt(dd, ctx_id++)->ctxt << (j * 8);
 		/* Wrap up netdev ctx index */
-		ctx_id %= dd->vnic.num_ctxt;
+		ctx_id %= ctxt_count;
 		/* Write back map register */
-		if (j == 7 || ((i + 1) == NUM_VNIC_MAP_ENTRIES)) {
+		if (j == 7 || ((i + 1) == NUM_NETDEV_MAP_ENTRIES)) {
 			dev_dbg(&(dd)->pcidev->dev,
 				"RMT[%d] =0x%llx\n",
 				regoff - RCV_RSM_MAP_TABLE, reg);
 
 			write_csr(dd, regoff, reg);
 			regoff += 8;
-			if (i < (NUM_VNIC_MAP_ENTRIES - 1))
+			if (i < (NUM_NETDEV_MAP_ENTRIES - 1))
 				reg = read_csr(dd, regoff);
 		}
 	}
@@ -14617,8 +14613,9 @@ void hfi1_init_aip_rsm(struct hfi1_devdata *dd)
 	 * exist yet
 	 */
 	if (atomic_fetch_inc(&dd->ipoib_rsm_usr_num) == 0) {
+		int rmt_start = hfi1_netdev_get_free_rmt_idx(dd);
 		struct rsm_rule_data rrd = {
-			.offset = dd->vnic.rmt_start,
+			.offset = rmt_start,
 			.pkt_type = IB_PACKET_TYPE,
 			.field1_off = LRH_BTH_MATCH_OFFSET,
 			.mask1 = LRH_BTH_MASK,
@@ -14627,10 +14624,10 @@ void hfi1_init_aip_rsm(struct hfi1_devdata *dd)
 			.mask2 = BTH_DESTQP_MASK,
 			.value2 = BTH_DESTQP_VALUE,
 			.index1_off = DETH_AIP_SQPN_SELECT_OFFSET +
-					ilog2(NUM_VNIC_MAP_ENTRIES),
-			.index1_width = ilog2(NUM_VNIC_MAP_ENTRIES),
+					ilog2(NUM_NETDEV_MAP_ENTRIES),
+			.index1_width = ilog2(NUM_NETDEV_MAP_ENTRIES),
 			.index2_off = DETH_AIP_SQPN_SELECT_OFFSET,
-			.index2_width = ilog2(NUM_VNIC_MAP_ENTRIES)
+			.index2_width = ilog2(NUM_NETDEV_MAP_ENTRIES)
 		};
 
 		hfi1_enable_rsm_rule(dd, RSM_INS_AIP, &rrd);
@@ -14640,9 +14637,10 @@ void hfi1_init_aip_rsm(struct hfi1_devdata *dd)
 /* Initialize RSM for VNIC */
 void hfi1_init_vnic_rsm(struct hfi1_devdata *dd)
 {
+	int rmt_start = hfi1_netdev_get_free_rmt_idx(dd);
 	struct rsm_rule_data rrd = {
 		/* Add rule for vnic */
-		.offset = dd->vnic.rmt_start,
+		.offset = rmt_start,
 		.pkt_type = 4,
 		/* Match 16B packets */
 		.field1_off = L2_TYPE_MATCH_OFFSET,
@@ -14654,9 +14652,9 @@ void hfi1_init_vnic_rsm(struct hfi1_devdata *dd)
 		.value2 = L4_16B_ETH_VALUE,
 		/* Calc context from veswid and entropy */
 		.index1_off = L4_16B_HDR_VESWID_OFFSET,
-		.index1_width = ilog2(NUM_VNIC_MAP_ENTRIES),
+		.index1_width = ilog2(NUM_NETDEV_MAP_ENTRIES),
 		.index2_off = L2_16B_ENTROPY_OFFSET,
-		.index2_width = ilog2(NUM_VNIC_MAP_ENTRIES)
+		.index2_width = ilog2(NUM_NETDEV_MAP_ENTRIES)
 	};
 
 	hfi1_enable_rsm_rule(dd, RSM_INS_VNIC, &rrd);
@@ -14690,8 +14688,8 @@ static int init_rxe(struct hfi1_devdata *dd)
 	init_qos(dd, rmt);
 	init_fecn_handling(dd, rmt);
 	complete_rsm_map_table(dd, rmt);
-	/* record number of used rsm map entries for vnic */
-	dd->vnic.rmt_start = rmt->used;
+	/* record number of used rsm map entries for netdev */
+	hfi1_netdev_set_free_rmt_idx(dd, rmt->used);
 	kfree(rmt);
 
 	/*
@@ -15245,6 +15243,10 @@ int hfi1_init_dd(struct hfi1_devdata *dd)
 		 (dd->revision >> CCE_REVISION_SW_SHIFT)
 		    & CCE_REVISION_SW_MASK);
 
+	/* alloc netdev data */
+	if (hfi1_netdev_alloc(dd))
+		goto bail_cleanup;
+
 	ret = set_up_context_variables(dd);
 	if (ret)
 		goto bail_cleanup;
@@ -15345,6 +15347,7 @@ int hfi1_init_dd(struct hfi1_devdata *dd)
 	hfi1_comp_vectors_clean_up(dd);
 	msix_clean_up_interrupts(dd);
 bail_cleanup:
+	hfi1_netdev_free(dd);
 	hfi1_pcie_ddcleanup(dd);
 bail_free:
 	hfi1_free_devdata(dd);
drivers/infiniband/hw/hfi1/driver.c +0 -18
index d89fc8f..60ff6de 100644
@@ -1771,28 +1771,10 @@ static void process_receive_ib(struct hfi1_packet *packet)
 	hfi1_ib_rcv(packet);
 }
 
-static inline bool hfi1_is_vnic_packet(struct hfi1_packet *packet)
-{
-	/* Packet received in VNIC context via RSM */
-	if (packet->rcd->is_vnic)
-		return true;
-
-	if ((hfi1_16B_get_l2(packet->ebuf) == OPA_16B_L2_TYPE) &&
-	    (hfi1_16B_get_l4(packet->ebuf) == OPA_16B_L4_ETHR))
-		return true;
-
-	return false;
-}
-
 static void process_receive_bypass(struct hfi1_packet *packet)
 {
 	struct hfi1_devdata *dd = packet->rcd->dd;
 
-	if (hfi1_is_vnic_packet(packet)) {
-		hfi1_vnic_bypass_rcv(packet);
-		return;
-	}
-
 	if (hfi1_setup_bypass_packet(packet))
 		return;
 
drivers/infiniband/hw/hfi1/hfi.h +1 -13
index 986d8c3..b4c6bff 100644
@@ -1047,23 +1047,10 @@ struct hfi1_asic_data {
 #define NUM_MAP_ENTRIES	 256
 #define NUM_MAP_REGS      32
 
-/*
- * Number of VNIC contexts used. Ensure it is less than or equal to
- * max queues supported by VNIC (HFI1_VNIC_MAX_QUEUE).
- */
-#define HFI1_NUM_VNIC_CTXT   8
-
-/* Number of VNIC RSM entries */
-#define NUM_VNIC_MAP_ENTRIES 8
-
 /* Virtual NIC information */
 struct hfi1_vnic_data {
-	struct hfi1_ctxtdata *ctxt[HFI1_NUM_VNIC_CTXT];
 	struct kmem_cache *txreq_cache;
-	struct xarray vesws;
 	u8 num_vports;
-	u8 rmt_start;
-	u8 num_ctxt;
 };
 
 struct hfi1_vnic_vport_info;
@@ -1419,6 +1406,7 @@ struct hfi1_devdata {
 	struct hfi1_vnic_data vnic;
 	/* Lock to protect IRQ SRC register access */
 	spinlock_t irq_src_lock;
+	int vnic_num_vports;
 	struct net_device *dummy_netdev;
 
 	/* Keeps track of IPoIB RSM rule users */
drivers/infiniband/hw/hfi1/init.c +3 -6
index 64279d0..5eed436 100644
@@ -69,6 +69,7 @@
 #include "affinity.h"
 #include "vnic.h"
 #include "exp_rcv.h"
+#include "netdev.h"
 
 #undef pr_fmt
 #define pr_fmt(fmt) DRIVER_NAME ": " fmt
@@ -1665,9 +1666,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	/* do the generic initialization */
 	initfail = hfi1_init(dd, 0);
 
-	/* setup vnic */
-	hfi1_vnic_setup(dd);
-
 	ret = hfi1_register_ib_device(dd);
 
 	/*
@@ -1706,7 +1704,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 			hfi1_device_remove(dd);
 		if (!ret)
 			hfi1_unregister_ib_device(dd);
-		hfi1_vnic_cleanup(dd);
 		postinit_cleanup(dd);
 		if (initfail)
 			ret = initfail;
@@ -1751,8 +1748,8 @@ static void remove_one(struct pci_dev *pdev)
 	/* unregister from IB core */
 	hfi1_unregister_ib_device(dd);
 
-	/* cleanup vnic */
-	hfi1_vnic_cleanup(dd);
+	/* free netdev data */
+	hfi1_netdev_free(dd);
 
 	/*
 	 * Disable the IB link, disable interrupts on the device,
drivers/infiniband/hw/hfi1/ipoib_rx.c +9 -1
index 606ac69..3afa754 100644
@@ -74,8 +74,15 @@ int hfi1_ipoib_rxq_init(struct net_device *netdev)
 {
 	struct hfi1_ipoib_dev_priv *ipoib_priv = hfi1_ipoib_priv(netdev);
 	struct hfi1_devdata *dd = ipoib_priv->dd;
+	int ret;
 
-	return hfi1_netdev_rx_init(dd);
+	ret = hfi1_netdev_rx_init(dd);
+	if (ret)
+		return ret;
+
+	hfi1_init_aip_rsm(dd);
+
+	return ret;
 }
 
 void hfi1_ipoib_rxq_deinit(struct net_device *netdev)
@@ -83,5 +90,6 @@ void hfi1_ipoib_rxq_deinit(struct net_device *netdev)
 	struct hfi1_ipoib_dev_priv *ipoib_priv = hfi1_ipoib_priv(netdev);
 	struct hfi1_devdata *dd = ipoib_priv->dd;
 
+	hfi1_deinit_aip_rsm(dd);
 	hfi1_netdev_rx_destroy(dd);
 }
drivers/infiniband/hw/hfi1/msix.c +7 -5
index 7559875e..d61ee85 100644
@@ -172,7 +172,8 @@ static int msix_request_rcd_irq_common(struct hfi1_ctxtdata *rcd,
 				       const char *name)
 {
 	int nr = msix_request_irq(rcd->dd, rcd, handler, thread,
-				  IRQ_RCVCTXT, name);
+				  rcd->is_vnic ? IRQ_NETDEVCTXT : IRQ_RCVCTXT,
+				  name);
 	if (nr < 0)
 		return nr;
 
@@ -371,15 +372,16 @@ void msix_clean_up_interrupts(struct hfi1_devdata *dd)
 }
 
 /**
- * msix_vnic_syncrhonize_irq() - Vnic IRQ synchronize
+ * msix_netdev_syncrhonize_irq() - netdev IRQ synchronize
  * @dd: valid devdata
  */
-void msix_vnic_synchronize_irq(struct hfi1_devdata *dd)
+void msix_netdev_synchronize_irq(struct hfi1_devdata *dd)
 {
 	int i;
+	int ctxt_count = hfi1_netdev_ctxt_count(dd);
 
-	for (i = 0; i < dd->vnic.num_ctxt; i++) {
-		struct hfi1_ctxtdata *rcd = dd->vnic.ctxt[i];
+	for (i = 0; i < ctxt_count; i++) {
+		struct hfi1_ctxtdata *rcd = hfi1_netdev_get_ctxt(dd, i);
 		struct hfi1_msix_entry *me;
 
 		me = &dd->msix_info.msix_entries[rcd->msix_intr];
drivers/infiniband/hw/hfi1/msix.h +1 -1
index 42fab94..e63e944 100644
@@ -60,7 +60,7 @@
 void msix_free_irq(struct hfi1_devdata *dd, u8 msix_intr);
 
 /* Netdev interface */
-void msix_vnic_synchronize_irq(struct hfi1_devdata *dd);
+void msix_netdev_synchronize_irq(struct hfi1_devdata *dd);
 int msix_netdev_request_rcd_irq(struct hfi1_ctxtdata *rcd);
 
 #endif
drivers/infiniband/hw/hfi1/netdev.h +19 -0
index edb936f..947543a 100644
@@ -82,6 +82,25 @@ struct hfi1_ctxtdata *hfi1_netdev_get_ctxt(struct hfi1_devdata *dd, int ctxt)
 	return priv->rxq[ctxt].rcd;
 }
 
+static inline
+int hfi1_netdev_get_free_rmt_idx(struct hfi1_devdata *dd)
+{
+	struct hfi1_netdev_priv *priv = hfi1_netdev_priv(dd->dummy_netdev);
+
+	return priv->rmt_start;
+}
+
+static inline
+void hfi1_netdev_set_free_rmt_idx(struct hfi1_devdata *dd, int rmt_idx)
+{
+	struct hfi1_netdev_priv *priv = hfi1_netdev_priv(dd->dummy_netdev);
+
+	priv->rmt_start = rmt_idx;
+}
+
+u32 hfi1_num_netdev_contexts(struct hfi1_devdata *dd, u32 available_contexts,
+			     struct cpumask *cpu_mask);
+
 void hfi1_netdev_enable_queues(struct hfi1_devdata *dd);
 void hfi1_netdev_disable_queues(struct hfi1_devdata *dd);
 int hfi1_netdev_rx_init(struct hfi1_devdata *dd);
drivers/infiniband/hw/hfi1/netdev_rx.c +45 -1

58 lines not on this page · raw message

drivers/infiniband/hw/hfi1/vnic.h +2 -3
index 5ae7815..66150a13 100644
@@ -1,7 +1,7 @@
 #ifndef _HFI1_VNIC_H
 #define _HFI1_VNIC_H
 /*
- * Copyright(c) 2017 Intel Corporation.
+ * Copyright(c) 2017 - 2020 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.
@@ -69,6 +69,7 @@
 #define HFI1_VNIC_SC_SHIFT      4
 
 #define HFI1_VNIC_MAX_QUEUE 16
+#define HFI1_NUM_VNIC_CTXT 8
 
 /**
  * struct hfi1_vnic_sdma - VNIC per Tx ring SDMA information
@@ -104,7 +105,6 @@ struct hfi1_vnic_rx_queue {
 	struct hfi1_vnic_vport_info *vinfo;
 	struct net_device           *netdev;
 	struct napi_struct           napi;
-	struct sk_buff_head          skbq;
 };
 
 /**
@@ -146,7 +146,6 @@ struct hfi1_vnic_vport_info {
 
 /* vnic hfi1 internal functions */
 void hfi1_vnic_setup(struct hfi1_devdata *dd);
-void hfi1_vnic_cleanup(struct hfi1_devdata *dd);
 int hfi1_vnic_txreq_init(struct hfi1_devdata *dd);
 void hfi1_vnic_txreq_deinit(struct hfi1_devdata *dd);
 
drivers/infiniband/hw/hfi1/vnic_main.c +53 -259

453 lines not on this page · raw message

#24 lore raw

PATCH v3 13/16 IB/{hfi1, ipoib, rdma}: Broadcast ping sent packets which exceeded mtu size

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

When in connected mode ipoib sent broadcast pings which exceeded the mtu size for broadcast addresses.

Add an mtu attribute to the rdma_netdev structure which ipoib sets to its mcast mtu size.

The RDMA netdev uses this value to determine if the skb length is too long for the mtu specified and if it is, drops the packet and logs an error about the errant packet.

---
 drivers/infiniband/ulp/ipoib/ipoib_main.c      |    2 ++
 drivers/infiniband/ulp/ipoib/ipoib_multicast.c |    1 +
 drivers/infiniband/ulp/ipoib/ipoib_vlan.c      |    3 +++
 3 files changed, 6 insertions(+)
drivers/infiniband/ulp/ipoib/ipoib_main.c +2 -0
index 22216f1..a6c4322 100644
@@ -1896,6 +1896,7 @@ static int ipoib_ndo_init(struct net_device *ndev)
 {
 	struct ipoib_dev_priv *priv = ipoib_priv(ndev);
 	int rc;
+	struct rdma_netdev *rn = netdev_priv(ndev);
 
 	if (priv->parent) {
 		ipoib_child_init(ndev);
@@ -1908,6 +1909,7 @@ static int ipoib_ndo_init(struct net_device *ndev)
 	/* MTU will be reset when mcast join happens */
 	ndev->mtu = IPOIB_UD_MTU(priv->max_ib_mtu);
 	priv->mcast_mtu = priv->admin_mtu = ndev->mtu;
+	rn->mtu = priv->mcast_mtu;
 	ndev->max_mtu = IPOIB_CM_MTU;
 
 	ndev->neigh_priv_len = sizeof(struct ipoib_neigh);
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +1 -0
index 7166ee9b..3d5f6b8 100644
@@ -246,6 +246,7 @@ static int ipoib_mcast_join_finish(struct ipoib_mcast *mcast,
 		if (priv->mcast_mtu == priv->admin_mtu)
 			priv->admin_mtu = IPOIB_UD_MTU(mtu);
 		priv->mcast_mtu = IPOIB_UD_MTU(mtu);
+		rn->mtu = priv->mcast_mtu;
 
 		priv->qkey = be32_to_cpu(priv->broadcast->mcmember.qkey);
 		spin_unlock_irq(&priv->lock);
drivers/infiniband/ulp/ipoib/ipoib_vlan.c +3 -0
index 8ac8e18..3086560 100644
@@ -97,6 +97,7 @@ int __ipoib_vlan_add(struct ipoib_dev_priv *ppriv, struct ipoib_dev_priv *priv,
 {
 	struct net_device *ndev = priv->dev;
 	int result;
+	struct rdma_netdev *rn = netdev_priv(ndev);
 
 	ASSERT_RTNL();
 
@@ -117,6 +118,8 @@ int __ipoib_vlan_add(struct ipoib_dev_priv *ppriv, struct ipoib_dev_priv *priv,
 		goto out_early;
 	}
 
+	rn->mtu = priv->mcast_mtu;
+
 	priv->parent = ppriv->dev;
 	priv->pkey = pkey;
 	priv->child_type = type;
#25 lore raw

PATCH v3 14/16 IB/hfi1: Add packet histogram trace event

In reply to Dennis Dalessandro

From: Grzegorz Andrejczuk <[email protected]>

Add a simple trace event taking context number and building simple histogram to print packets distribution between contexts.

---
 drivers/infiniband/hw/hfi1/driver.c      |    1 +
 drivers/infiniband/hw/hfi1/trace.c       |   32 ++++++++++++++++++++++++++++++
 drivers/infiniband/hw/hfi1/trace_ctxts.h |   11 +++++++++-
 3 files changed, 43 insertions(+), 1 deletion(-)
drivers/infiniband/hw/hfi1/driver.c +1 -0
index 60ff6de..a40701a 100644
@@ -1706,6 +1706,7 @@ static void hfi1_ipoib_ib_rcv(struct hfi1_packet *packet)
 		goto drop_no_nd;
 
 	trace_input_ibhdr(rcd->dd, packet, !!(rhf_dc_info(packet->rhf)));
+	trace_ctxt_rsm_hist(rcd->ctxt);
 
 	/* handle congestion notifications */
 	do_work = hfi1_may_ecn(packet);
drivers/infiniband/hw/hfi1/trace.c +32 -0
index c8a9988..b219ea9 100644
@@ -520,6 +520,38 @@ u16 hfi1_trace_get_tid_idx(u32 ent)
 	return EXP_TID_GET(ent, IDX);
 }
 
+struct hfi1_ctxt_hist {
+	atomic_t count;
+	atomic_t data[255];
+};
+
+struct hfi1_ctxt_hist hist = {
+	.count = ATOMIC_INIT(0)
+};
+
+const char *hfi1_trace_print_rsm_hist(struct trace_seq *p, unsigned int ctxt)
+{
+	int i, len = ARRAY_SIZE(hist.data);
+	const char *ret = trace_seq_buffer_ptr(p);
+	unsigned long packet_count = atomic_fetch_inc(&hist.count);
+
+	trace_seq_printf(p, "packet[%lu]", packet_count);
+	for (i = 0; i < len; ++i) {
+		unsigned long val;
+		atomic_t *count = &hist.data[i];
+
+		if (ctxt == i)
+			val = atomic_fetch_inc(count);
+		else
+			val = atomic_read(count);
+
+		if (val)
+			trace_seq_printf(p, "(%d:%lu)", i, val);
+	}
+	trace_seq_putc(p, 0);
+	return ret;
+}
+
 __hfi1_trace_fn(AFFINITY);
 __hfi1_trace_fn(PKT);
 __hfi1_trace_fn(PROC);
drivers/infiniband/hw/hfi1/trace_ctxts.h +10 -1
index b5fc5c6..d8c168d 100644
@@ -1,5 +1,5 @@
 /*
-* Copyright(c) 2015, 2016 Intel Corporation.
+* Copyright(c) 2015 - 2020 Intel Corporation.
 *
 * This file is provided under a dual BSD/GPLv2 license.  When using or
 * redistributing this file, you may do so under either license.
@@ -138,6 +138,15 @@
 		      )
 );
 
+const char *hfi1_trace_print_rsm_hist(struct trace_seq *p, unsigned int ctxt);
+TRACE_EVENT(ctxt_rsm_hist,
+	    TP_PROTO(unsigned int ctxt),
+	    TP_ARGS(ctxt),
+	    TP_STRUCT__entry(__field(unsigned int, ctxt)),
+	    TP_fast_assign(__entry->ctxt = ctxt;),
+	    TP_printk("%s", hfi1_trace_print_rsm_hist(p, __entry->ctxt))
+);
+
 #endif /* __HFI1_TRACE_CTXTS_H */
 
 #undef TRACE_INCLUDE_PATH
#26 lore raw

PATCH v3 15/16 IB/ipoib: Add capability to switch between datagram and connected mode

In reply to Dennis Dalessandro

From: Gary Leshner <[email protected]>

This is the prerequisite modification to the ipoib ulp to allow a rdma netdev to obtain the default ndo ops for init/uninit/open/close.

This is accomplished by setting the netdev ops field within the callback function passed to the netdev allocation routine which in turn was passed into the rdma netdev allocation routine.

This allows the rdma netdev to call back into the ulp to create the resources required for connected mode operation.

Additionally as the ulp is not re-entrant, when switching modes, the number of real tx queues is set to 1 for the connected mode.

For datagram mode the number of real tx queues is set to the actual number of tx queues specified at the netdev's allocation.

For the internal ulp netdev the number of tx queues defaults to 1.

It is up to the rdma netdev to specify the actual number it can support.

When the driver does not support a rdma netdev for acceleration, (-ENOTSUPPORTED return code or the verbs function for allocation is NULL) the ipoib ulp functions are unaffected by using the internal netdev allocated by the ipoib ulp.

---
 drivers/infiniband/ulp/ipoib/ipoib_main.c |   18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_main.c +10 -8
index a6c4322..d12e5c9 100644
@@ -526,6 +526,7 @@ int ipoib_set_mode(struct net_device *dev, const char *buf)
 			   "will cause multicast packet drops\n");
 		netdev_update_features(dev);
 		dev_set_mtu(dev, ipoib_cm_max_mtu(dev));
+		netif_set_real_num_tx_queues(dev, 1);
 		rtnl_unlock();
 		priv->tx_wr.wr.send_flags &= ~IB_SEND_IP_CSUM;
 
@@ -537,6 +538,7 @@ int ipoib_set_mode(struct net_device *dev, const char *buf)
 		clear_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags);
 		netdev_update_features(dev);
 		dev_set_mtu(dev, min(priv->mcast_mtu, dev->mtu));
+		netif_set_real_num_tx_queues(dev, dev->num_tx_queues);
 		rtnl_unlock();
 		ipoib_flush_paths(dev);
 		return (!rtnl_trylock()) ? -EBUSY : 0;
@@ -2071,9 +2073,17 @@ static int ipoib_get_vf_stats(struct net_device *dev, int vf,
 	.ndo_do_ioctl		 = ipoib_ioctl,
 };
 
+static const struct net_device_ops ipoib_netdev_default_pf = {
+	.ndo_init		 = ipoib_dev_init_default,
+	.ndo_uninit		 = ipoib_dev_uninit_default,
+	.ndo_open		 = ipoib_ib_dev_open_default,
+	.ndo_stop		 = ipoib_ib_dev_stop_default,
+};
+
 void ipoib_setup_common(struct net_device *dev)
 {
 	dev->header_ops		 = &ipoib_header_ops;
+	dev->netdev_ops          = &ipoib_netdev_default_pf;
 
 	ipoib_set_ethtool_ops(dev);
 
@@ -2123,13 +2133,6 @@ static void ipoib_build_priv(struct net_device *dev)
 	INIT_DELAYED_WORK(&priv->neigh_reap_task, ipoib_reap_neigh);
 }
 
-static const struct net_device_ops ipoib_netdev_default_pf = {
-	.ndo_init		 = ipoib_dev_init_default,
-	.ndo_uninit		 = ipoib_dev_uninit_default,
-	.ndo_open		 = ipoib_ib_dev_open_default,
-	.ndo_stop		 = ipoib_ib_dev_stop_default,
-};
-
 static struct net_device *ipoib_alloc_netdev(struct ib_device *hca, u8 port,
 					     const char *name)
 {
@@ -2167,7 +2170,6 @@ int ipoib_intf_init(struct ib_device *hca, u8 port, const char *name,
 		if (rc != -EOPNOTSUPP)
 			goto out;
 
-		dev->netdev_ops = &ipoib_netdev_default_pf;
 		rn->send = ipoib_send;
 		rn->attach_mcast = ipoib_mcast_attach;
 		rn->detach_mcast = ipoib_mcast_detach;
#27 lore raw

PATCH v3 16/16 IB/hfi1: Enable the transmit side of the datagram ipoib netdev

In reply to Dennis Dalessandro

From: Piotr Stankiewicz <[email protected]>

This patch hooks the transmit side of the datagram netdev with ipoib by setting the rdma_netdev_get_params function for the hfi1 ib_device_ops structue. It also enables the receiving side by adding the AIP capability into the default capabilities.

---
 drivers/infiniband/hw/hfi1/common.h |    1 +
 drivers/infiniband/hw/hfi1/verbs.c  |    2 ++
 2 files changed, 3 insertions(+)
drivers/infiniband/hw/hfi1/common.h +1 -0
index 6062545..ff423e54 100644
@@ -160,6 +160,7 @@
 				 HFI1_CAP_PKEY_CHECK |			\
 				 HFI1_CAP_MULTI_PKT_EGR |		\
 				 HFI1_CAP_EXTENDED_PSN |		\
+				 HFI1_CAP_AIP |				\
 				 ((HFI1_CAP_HDRSUPP |			\
 				   HFI1_CAP_MULTI_PKT_EGR |		\
 				   HFI1_CAP_STATIC_RATE_CTRL |		\
drivers/infiniband/hw/hfi1/verbs.c +2 -0
index 19d5d00..43ddced 100644
@@ -66,6 +66,7 @@
 #include "vnic.h"
 #include "fault.h"
 #include "affinity.h"
+#include "ipoib.h"
 
 static unsigned int hfi1_lkey_table_size = 16;
 module_param_named(lkey_table_size, hfi1_lkey_table_size, uint,
@@ -1795,6 +1796,7 @@ static int get_hw_stats(struct ib_device *ibdev, struct rdma_hw_stats *stats,
 	.modify_device = modify_device,
 	/* keep process mad in the driver */
 	.process_mad = hfi1_process_mad,
+	.rdma_netdev_get_params = hfi1_ipoib_rn_get_params,
 };
 
 /**
#28 lore raw
In reply to Dennis Dalessandro
Quoted 40 lines · Dennis Dalessandro

This patch series is an accelerated ipoib using the rdma netdev mechanism already present in ipoib. A new device capability bit, IB_DEVICE_RDMA_NETDEV_OPA, triggers ipoib to create a datagram QP using the IB_QP_CREATE_NETDEV_USE.

The highlights include:
- Sharing send and receive resources with VNIC
- Allows for switching between connected mode and datagram mode
- Increases the maximum datagram MTU for opa devices to 10k

The same spreading capability exploited by VNIC is used here to vary the receive context that receives the packet.

The patches are fully bisectable and stepwise implement the capability.

changes since v2
*Rebased ontop of latest rdma/for-next

Changes since v1
*Fix incorrect parameter to xa_find() in patch 9 *Address Erez comments and try to hide opa from ipoib in patch 7 *Fix some typos in RB lines

Gary Leshner (6):
IB/hfi1: Add functions to transmit datagram ipoib packets IB/hfi1: Add the transmit side of a datagram ipoib RDMA netdev IB/hfi1: Remove module parameter for KDETH qpns IB/{rdmavt,hfi1}: Implement creation of accelerated UD QPs IB/{hfi1,ipoib,rdma}: Broadcast ping sent packets which exceeded mtu size IB/ipoib: Add capability to switch between datagram and connected mode

Grzegorz Andrejczuk (6):
IB/hfi1: RSM rules for AIP
IB/hfi1: Rename num_vnic_contexts as num_netdev_contexts IB/hfi1: Add interrupt handler functions for accelerated ipoib IB/hfi1: Add rx functions for dummy netdev
IB/hfi1: Activate the dummy netdev
IB/hfi1: Add packet histogram trace event

Quoted 7 lines

Kaike Wan (3):
IB/hfi1: Add accelerated IP capability bit IB/ipoib: Increase ipoib Datagram mode MTU's upper limit IB/hfi1: Add functions to receive accelerated ipoib packets

Piotr Stankiewicz (1):
IB/hfi1: Enable the transmit side of the datagram ipoib netdev

Applied to for-next

Thanks,
Jason