Skip to contentSkip to conversation

IB/core, hfi1, rdmavt: patches for next 05/19/2017

PATCH In review Dennis Dalessandro 6 participants 20 messages May 26, 2017 – Jun 27, 2017 10 messages not in the archive

Subsystems (3)
InfiniBand HFI1 driver Rdmavt - RDMA verbs software
lore
Share & tools
lore

Dennis Dalessandro

author
#0 lore raw

cover IB/core, hfi1, rdmavt: patches for next 05/19/2017

Hi Doug,

Another set of patches for 4.13. These apply on top of the last submitted
series: http://marc.info/?l=linux-rdma&m=149460607918479&w=2. I wanted to submit this last week but 0-day build service and my GitHub weren't seeing eye to eye. Things are working now and these have passed 0-day builds.

These are mostly fixes for the hfi1 driver, but there is a core change and one for rdmavt as well which adds the ability to send OPA cap_mask3 in a trap.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.13

---

Ira Weiny (1):
IB/hfi1: Fix up sdma_init function comment

Michael J. Ruhl (2):
IB/hfi1: Name function prototype parameters for affinity module IB/hfi1: Replace deprecated pci functions with new API

Michael N. Henry (1):
IB/core,rdmavt,hfi1: Send OPA cap_mask3 in trap

Sebastian Sanchez (5):
IB/hfi1: Don't remove RB entry when not needed.
IB/hfi1: Optimize cachelines for user SDMA request structure IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation IB/hfi1: Remove atomic SDMA_REQ_HAS_ERROR bit operation IB/hfi1: Reclassify type of messages printed for platform config logic

 drivers/infiniband/hw/hfi1/affinity.c  |   18 ++--
 drivers/infiniband/hw/hfi1/affinity.h  |   14 ++-
 drivers/infiniband/hw/hfi1/chip.c      |  105 ++++++++++-----------
 drivers/infiniband/hw/hfi1/hfi.h       |    6 -
 drivers/infiniband/hw/hfi1/mad.c       |    7 -
 drivers/infiniband/hw/hfi1/mad.h       |    2 
 drivers/infiniband/hw/hfi1/mmu_rb.c    |   14 ++-
 drivers/infiniband/hw/hfi1/mmu_rb.h    |    5 +
 drivers/infiniband/hw/hfi1/pcie.c      |   80 +++-------------
 drivers/infiniband/hw/hfi1/platform.c  |   20 ++--
 drivers/infiniband/hw/hfi1/sdma.c      |    6 -
 drivers/infiniband/hw/hfi1/user_sdma.c |  159 +++++++++++++++++---------------
 drivers/infiniband/hw/hfi1/verbs.c     |    7 +
 drivers/infiniband/sw/rdmavt/vt.c      |    9 +-
 include/rdma/ib_verbs.h                |    3 -
 include/rdma/rdma_vt.h                 |    1 
 16 files changed, 215 insertions(+), 241 deletions(-)

--
-Denny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170526123257.8158.23943.stgit-9QXIwq+3FY+1XWohqUldA0EOCMrvLtNR@public.gmane.org

author
#2 lore raw

PATCH 1/9 IB/hfi1: Don't remove RB entry when not needed.

From: Sebastian Sanchez <[email protected]>

An RB tree is used for the SDMA pinning cache. Cache entries are extracted and reinserted from the tree in case the address range for it changes. However, if the address range for the entry doesn't change, deleting the entry from the RB tree is not necessary.

This affects performance since the tree needs to be rebalanced for each insertion, and this happens in the hot path. Optimize RB search by not removing entries when it's not needed.

Reviewed-by Mike Marciniszyn <[email protected]>
Reviewed-by Mitko Haralanov <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/mmu_rb.c    |   14 ++++++++++----
 drivers/infiniband/hw/hfi1/mmu_rb.h    |    5 +++--
 drivers/infiniband/hw/hfi1/user_sdma.c |   23 ++++++++++++++++-------
 3 files changed, 29 insertions(+), 13 deletions(-)
drivers/infiniband/hw/hfi1/mmu_rb.c +10 -4
index ccbf52c..d41fd87 100644
@@ -217,21 +217,27 @@ int hfi1_mmu_rb_insert(struct mmu_rb_handler *handler,
 	return node;
 }
 
-struct mmu_rb_node *hfi1_mmu_rb_extract(struct mmu_rb_handler *handler,
-					unsigned long addr, unsigned long len)
+bool hfi1_mmu_rb_remove_unless_exact(struct mmu_rb_handler *handler,
+				     unsigned long addr, unsigned long len,
+				     struct mmu_rb_node **rb_node)
 {
 	struct mmu_rb_node *node;
 	unsigned long flags;
+	bool ret = false;
 
 	spin_lock_irqsave(&handler->lock, flags);
 	node = __mmu_rb_search(handler, addr, len);
 	if (node) {
+		if (node->addr == addr && node->len == len)
+			goto unlock;
 		__mmu_int_rb_remove(node, &handler->root);
 		list_del(&node->list); /* remove from LRU list */
+		ret = true;
 	}
+unlock:
 	spin_unlock_irqrestore(&handler->lock, flags);
-
-	return node;
+	*rb_node = node;
+	return ret;
 }
 
 void hfi1_mmu_rb_evict(struct mmu_rb_handler *handler, void *evict_arg)
drivers/infiniband/hw/hfi1/mmu_rb.h +3 -2
index 754f6eb..f04cec1 100644
@@ -81,7 +81,8 @@ int hfi1_mmu_rb_insert(struct mmu_rb_handler *handler,
 void hfi1_mmu_rb_evict(struct mmu_rb_handler *handler, void *evict_arg);
 void hfi1_mmu_rb_remove(struct mmu_rb_handler *handler,
 			struct mmu_rb_node *mnode);
-struct mmu_rb_node *hfi1_mmu_rb_extract(struct mmu_rb_handler *handler,
-					unsigned long addr, unsigned long len);
+bool hfi1_mmu_rb_remove_unless_exact(struct mmu_rb_handler *handler,
+				     unsigned long addr, unsigned long len,
+				     struct mmu_rb_node **rb_node);
 
 #endif /* _HFI1_MMU_RB_H */
drivers/infiniband/hw/hfi1/user_sdma.c +16 -7
index 16fd519..79450cf 100644
@@ -1165,14 +1165,23 @@ static int pin_vector_pages(struct user_sdma_request *req,
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct sdma_mmu_node *node = NULL;
 	struct mmu_rb_node *rb_node;
-
-	rb_node = hfi1_mmu_rb_extract(pq->handler,
-				      (unsigned long)iovec->iov.iov_base,
-				      iovec->iov.iov_len);
-	if (rb_node)
+	bool extracted;
+
+	extracted =
+		hfi1_mmu_rb_remove_unless_exact(pq->handler,
+						(unsigned long)
+						iovec->iov.iov_base,
+						iovec->iov.iov_len, &rb_node);
+	if (rb_node) {
 		node = container_of(rb_node, struct sdma_mmu_node, rb);
-	else
-		rb_node = NULL;
+		if (!extracted) {
+			atomic_inc(&node->refcount);
+			iovec->pages = node->pages;
+			iovec->npages = node->npages;
+			iovec->node = node;
+			return 0;
+		}
+	}
 
 	if (!node) {
 		node = kzalloc(sizeof(*node), GFP_KERNEL);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw

PATCH 2/9 IB/hfi1: Optimize cachelines for user SDMA request structure

In reply to a message not in the archive

From: Sebastian Sanchez <[email protected]>

The current user SDMA request structure layout has holes. The cachelines can be reduced to improve cacheline trading. Separate fields in the following categories: mostly read, writable and shared with interrupt.

Reviewed-by Mike Marciniszyn <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |  106 ++++++++++++++++++--------------
 1 files changed, 58 insertions(+), 48 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.c +58 -48
index 79450cf..92517ce 100644
@@ -117,6 +117,7 @@
 
 #define AHG_KDETH_INTR_SHIFT 12
 #define AHG_KDETH_SH_SHIFT   13
+#define AHG_KDETH_ARRAY_SIZE  9
 
 #define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
 #define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)

@@ -204,25 +205,42 @@ struct evict_data {
 };
 
 struct user_sdma_request {
-	struct sdma_req_info info;
-	struct hfi1_user_sdma_pkt_q *pq;
-	struct hfi1_user_sdma_comp_q *cq;
 	/* This is the original header from user space */
 	struct hfi1_pkt_header hdr;
+
+	/* Read mostly fields */
+	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
+	struct hfi1_user_sdma_comp_q *cq;
 	/*
 	 * Pointer to the SDMA engine for this request.
 	 * Since different request could be on different VLs,
 	 * each request will need it's own engine pointer.
 	 */
 	struct sdma_engine *sde;
-	s8 ahg_idx;
-	u32 ahg[9];
+	struct sdma_req_info info;
+	/* TID array values copied from the tid_iov vector */
+	u32 *tids;
+	/* total length of the data in the request */
+	u32 data_len;
+	/* number of elements copied to the tids array */
+	u16 n_tids;
 	/*
-	 * KDETH.Offset (Eager) field
-	 * We need to remember the initial value so the headers
-	 * can be updated properly.
+	 * We copy the iovs for this request (based on
+	 * info.iovcnt). These are only the data vectors
 	 */
-	u32 koffset;
+	u8 data_iovs;
+	s8 ahg_idx;
+
+	/* Writeable fields shared with interrupt */
+	u64 seqcomp ____cacheline_aligned_in_smp;
+	u64 seqsubmitted;
+	unsigned long flags;
+	/* status of the last txreq completed */
+	int status;
+
+	/* Send side fields */
+	struct list_head txps ____cacheline_aligned_in_smp;
+	u64 seqnum;
 	/*
 	 * KDETH.OFFSET (TID) field
 	 * The offset can cover multiple packets, depending on the

@@ -230,29 +248,19 @@ struct user_sdma_request {
 	 */
 	u32 tidoffset;
 	/*
-	 * We copy the iovs for this request (based on
-	 * info.iovcnt). These are only the data vectors
+	 * KDETH.Offset (Eager) field
+	 * We need to remember the initial value so the headers
+	 * can be updated properly.
 	 */
-	unsigned data_iovs;
-	/* total length of the data in the request */
-	u32 data_len;
+	u32 koffset;
+	u32 sent;
+	/* TID index copied from the tid_iov vector */
+	u16 tididx;
 	/* progress index moving along the iovs array */
-	unsigned iov_idx;
+	u8 iov_idx;
+
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
-	/* number of elements copied to the tids array */
-	u16 n_tids;
-	/* TID array values copied from the tid_iov vector */
-	u32 *tids;
-	u16 tididx;
-	u32 sent;
-	u64 seqnum;
-	u64 seqcomp;
-	u64 seqsubmitted;
-	struct list_head txps;
-	unsigned long flags;
-	/* status of the last txreq completed */
-	int status;
-};
+} ____cacheline_aligned_in_smp;
 
 /*
  * A single txreq could span up to 3 physical pages when the MTU

@@ -1034,11 +1042,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 							       datalen);
 				if (changes < 0)
 					goto free_tx;
-				sdma_txinit_ahg(&tx->txreq,
-						SDMA_TXREQ_F_USE_AHG,
-						datalen, req->ahg_idx, changes,
-						req->ahg, sizeof(req->hdr),
-						user_sdma_txreq_cb);
 			}
 		} else {
 			ret = sdma_txinit(&tx->txreq, 0, sizeof(req->hdr) +

@@ -1442,21 +1445,22 @@ static int set_txreq_header(struct user_sdma_request *req,
 }
 
 static int set_txreq_header_ahg(struct user_sdma_request *req,
-				struct user_sdma_txreq *tx, u32 len)
+				struct user_sdma_txreq *tx, u32 datalen)
 {
+	u32 ahg[AHG_KDETH_ARRAY_SIZE];
 	int diff = 0;
 	u8 omfactor; /* KDETH.OM */
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct hfi1_pkt_header *hdr = &req->hdr;
 	u16 pbclen = le16_to_cpu(hdr->pbc[0]);
-	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(len));
+	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(datalen));
 
 	if (PBC2LRH(pbclen) != lrhlen) {
 		/* PBC.PbcLengthDWs */
-		AHG_HEADER_SET(req->ahg, diff, 0, 0, 12,
+		AHG_HEADER_SET(ahg, diff, 0, 0, 12,
 			       cpu_to_le16(LRH2PBC(lrhlen)));
 		/* LRH.PktLen (we need the full 16 bits due to byte swap) */
-		AHG_HEADER_SET(req->ahg, diff, 3, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 3, 0, 16,
 			       cpu_to_be16(lrhlen >> 2));
 	}
 

@@ -1468,13 +1472,12 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 		(HFI1_CAP_IS_KSET(EXTENDED_PSN) ? 0x7fffffff : 0xffffff);
 	if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
 		val32 |= 1UL << 31;
-	AHG_HEADER_SET(req->ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
-	AHG_HEADER_SET(req->ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
+	AHG_HEADER_SET(ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
+	AHG_HEADER_SET(ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
 	/* KDETH.Offset */
-	AHG_HEADER_SET(req->ahg, diff, 15, 0, 16,
+	AHG_HEADER_SET(ahg, diff, 15, 0, 16,
 		       cpu_to_le16(req->koffset & 0xffff));
-	AHG_HEADER_SET(req->ahg, diff, 15, 16, 16,
-		       cpu_to_le16(req->koffset >> 16));
+	AHG_HEADER_SET(ahg, diff, 15, 16, 16, cpu_to_le16(req->koffset >> 16));
 	if (req_opcode(req->info.ctrl) == EXPECTED) {
 		__le16 val;
 

@@ -1492,9 +1495,8 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 			 * we have to check again.
 			 */
 			if (++req->tididx > req->n_tids - 1 ||
-			    !req->tids[req->tididx]) {
+			    !req->tids[req->tididx])
 				return -EINVAL;
-			}
 			tidval = req->tids[req->tididx];
 		}
 		omfactor = ((EXP_TID_GET(tidval, LEN) *

@@ -1502,7 +1504,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 				 KDETH_OM_MAX_SIZE) ? KDETH_OM_LARGE_SHIFT :
 				 KDETH_OM_SMALL_SHIFT;
 		/* KDETH.OM and KDETH.OFFSET (TID) */
-		AHG_HEADER_SET(req->ahg, diff, 7, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 7, 0, 16,
 			       ((!!(omfactor - KDETH_OM_SMALL_SHIFT)) << 15 |
 				((req->tidoffset >> omfactor)
 				 & 0x7fff)));

@@ -1522,12 +1524,20 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 					     AHG_KDETH_INTR_SHIFT));
 		}
 
-		AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
+		AHG_HEADER_SET(ahg, diff, 7, 16, 14, val);
 	}
+	if (diff < 0)
+		return diff;
 
 	trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
 					req->info.comp_idx, req->sde->this_idx,
-					req->ahg_idx, req->ahg, diff, tidval);
+					req->ahg_idx, ahg, diff, tidval);
+	sdma_txinit_ahg(&tx->txreq,
+			SDMA_TXREQ_F_USE_AHG,
+			datalen, req->ahg_idx, diff,
+			ahg, sizeof(req->hdr),
+			user_sdma_txreq_cb);
+
 	return diff;
 }

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170526123517.8158.80658.stgit-9QXIwq+3FY+1XWohqUldA0EOCMrvLtNR@public.gmane.org

#5 lore raw

Re: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

Quoted 105 lines · Dennis Dalessandro

From: Sebastian Sanchez <[email protected]>

The current user SDMA request structure layout has holes. The cachelines can be reduced to improve cacheline trading. Separate fields in the following categories: mostly read, writable and shared with interrupt.

Reviewed-by Mike Marciniszyn <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |  106 ++++++++++++++++++--------------
 1 files changed, 58 insertions(+), 48 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.c +36 -20
index 79450cf..92517ce 100644
@@ -117,6 +117,7 @@
 
 #define AHG_KDETH_INTR_SHIFT 12
 #define AHG_KDETH_SH_SHIFT   13
+#define AHG_KDETH_ARRAY_SIZE  9
 
 #define PBC2LRH(x) ((((x) & 0xfff) << 2) - 4)
 #define LRH2PBC(x) ((((x) >> 2) + 1) & 0xfff)

@@ -204,25 +205,42 @@ struct evict_data {
 };
 
 struct user_sdma_request {
-	struct sdma_req_info info;
-	struct hfi1_user_sdma_pkt_q *pq;
-	struct hfi1_user_sdma_comp_q *cq;
 	/* This is the original header from user space */
 	struct hfi1_pkt_header hdr;
+
+	/* Read mostly fields */
+	struct hfi1_user_sdma_pkt_q *pq ____cacheline_aligned_in_smp;
+	struct hfi1_user_sdma_comp_q *cq;
 	/*
 	 * Pointer to the SDMA engine for this request.
 	 * Since different request could be on different VLs,
 	 * each request will need it's own engine pointer.
 	 */
 	struct sdma_engine *sde;
-	s8 ahg_idx;
-	u32 ahg[9];
+	struct sdma_req_info info;
+	/* TID array values copied from the tid_iov vector */
+	u32 *tids;
+	/* total length of the data in the request */
+	u32 data_len;
+	/* number of elements copied to the tids array */
+	u16 n_tids;
 	/*
-	 * KDETH.Offset (Eager) field
-	 * We need to remember the initial value so the headers
-	 * can be updated properly.
+	 * We copy the iovs for this request (based on
+	 * info.iovcnt). These are only the data vectors
 	 */
-	u32 koffset;
+	u8 data_iovs;
+	s8 ahg_idx;
+
+	/* Writeable fields shared with interrupt */
+	u64 seqcomp ____cacheline_aligned_in_smp;
+	u64 seqsubmitted;
+	unsigned long flags;
+	/* status of the last txreq completed */
+	int status;
+
+	/* Send side fields */
+	struct list_head txps ____cacheline_aligned_in_smp;
+	u64 seqnum;
 	/*
 	 * KDETH.OFFSET (TID) field
 	 * The offset can cover multiple packets, depending on the

@@ -230,29 +248,19 @@ struct user_sdma_request {
 	 */
 	u32 tidoffset;
 	/*
-	 * We copy the iovs for this request (based on
-	 * info.iovcnt). These are only the data vectors
+	 * KDETH.Offset (Eager) field
+	 * We need to remember the initial value so the headers
+	 * can be updated properly.
 	 */
-	unsigned data_iovs;
-	/* total length of the data in the request */
-	u32 data_len;
+	u32 koffset;
+	u32 sent;
+	/* TID index copied from the tid_iov vector */
+	u16 tididx;
 	/* progress index moving along the iovs array */
-	unsigned iov_idx;
+	u8 iov_idx;
+
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
-	/* number of elements copied to the tids array */
-	u16 n_tids;
-	/* TID array values copied from the tid_iov vector */
-	u32 *tids;
-	u16 tididx;

drivers/infiniband/hw/hfi1/user_sdma.c struct user_sdma_request { · line 263

Quoted 10 lines
drivers/infiniband/hw/hfi1/user_sdma.c +1 -9
-	u32 sent;
-	u64 seqnum;
-	u64 seqcomp;
-	u64 seqsubmitted;
-	struct list_head txps;
-	unsigned long flags;
-	/* status of the last txreq completed */
-	int status;
-};
+} ____cacheline_aligned_in_smp;

Out of curiosity, do you have pahole data before vs. after to share?

Quoted 108 lines
drivers/infiniband/hw/hfi1/user_sdma.c +21 -20
 /*
  * A single txreq could span up to 3 physical pages when the MTU

@@ -1034,11 +1042,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 							       datalen);
 				if (changes < 0)
 					goto free_tx;
-				sdma_txinit_ahg(&tx->txreq,
-						SDMA_TXREQ_F_USE_AHG,
-						datalen, req->ahg_idx, changes,
-						req->ahg, sizeof(req->hdr),
-						user_sdma_txreq_cb);
 			}
 		} else {
 			ret = sdma_txinit(&tx->txreq, 0, sizeof(req->hdr) +

@@ -1442,21 +1445,22 @@ static int set_txreq_header(struct user_sdma_request *req,
 }
 
 static int set_txreq_header_ahg(struct user_sdma_request *req,
-				struct user_sdma_txreq *tx, u32 len)
+				struct user_sdma_txreq *tx, u32 datalen)
 {
+	u32 ahg[AHG_KDETH_ARRAY_SIZE];
 	int diff = 0;
 	u8 omfactor; /* KDETH.OM */
 	struct hfi1_user_sdma_pkt_q *pq = req->pq;
 	struct hfi1_pkt_header *hdr = &req->hdr;
 	u16 pbclen = le16_to_cpu(hdr->pbc[0]);
-	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(len));
+	u32 val32, tidval = 0, lrhlen = get_lrh_len(*hdr, pad_len(datalen));
 
 	if (PBC2LRH(pbclen) != lrhlen) {
 		/* PBC.PbcLengthDWs */
-		AHG_HEADER_SET(req->ahg, diff, 0, 0, 12,
+		AHG_HEADER_SET(ahg, diff, 0, 0, 12,
 			       cpu_to_le16(LRH2PBC(lrhlen)));
 		/* LRH.PktLen (we need the full 16 bits due to byte swap) */
-		AHG_HEADER_SET(req->ahg, diff, 3, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 3, 0, 16,
 			       cpu_to_be16(lrhlen >> 2));
 	}
 

@@ -1468,13 +1472,12 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 		(HFI1_CAP_IS_KSET(EXTENDED_PSN) ? 0x7fffffff : 0xffffff);
 	if (unlikely(tx->flags & TXREQ_FLAGS_REQ_ACK))
 		val32 |= 1UL << 31;
-	AHG_HEADER_SET(req->ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
-	AHG_HEADER_SET(req->ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
+	AHG_HEADER_SET(ahg, diff, 6, 0, 16, cpu_to_be16(val32 >> 16));
+	AHG_HEADER_SET(ahg, diff, 6, 16, 16, cpu_to_be16(val32 & 0xffff));
 	/* KDETH.Offset */
-	AHG_HEADER_SET(req->ahg, diff, 15, 0, 16,
+	AHG_HEADER_SET(ahg, diff, 15, 0, 16,
 		       cpu_to_le16(req->koffset & 0xffff));
-	AHG_HEADER_SET(req->ahg, diff, 15, 16, 16,
-		       cpu_to_le16(req->koffset >> 16));
+	AHG_HEADER_SET(ahg, diff, 15, 16, 16, cpu_to_le16(req->koffset >> 16));
 	if (req_opcode(req->info.ctrl) == EXPECTED) {
 		__le16 val;
 

@@ -1492,9 +1495,8 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 			 * we have to check again.
 			 */
 			if (++req->tididx > req->n_tids - 1 ||
-			    !req->tids[req->tididx]) {
+			    !req->tids[req->tididx])
 				return -EINVAL;
-			}
 			tidval = req->tids[req->tididx];
 		}
 		omfactor = ((EXP_TID_GET(tidval, LEN) *

@@ -1502,7 +1504,7 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 				 KDETH_OM_MAX_SIZE) ? KDETH_OM_LARGE_SHIFT :
 				 KDETH_OM_SMALL_SHIFT;
 		/* KDETH.OM and KDETH.OFFSET (TID) */
-		AHG_HEADER_SET(req->ahg, diff, 7, 0, 16,
+		AHG_HEADER_SET(ahg, diff, 7, 0, 16,
 			       ((!!(omfactor - KDETH_OM_SMALL_SHIFT)) << 15 |
 				((req->tidoffset >> omfactor)
 				 & 0x7fff)));

@@ -1522,12 +1524,20 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
 					     AHG_KDETH_INTR_SHIFT));
 		}
 
-		AHG_HEADER_SET(req->ahg, diff, 7, 16, 14, val);
+		AHG_HEADER_SET(ahg, diff, 7, 16, 14, val);
 	}
+	if (diff < 0)
+		return diff;
 
 	trace_hfi1_sdma_user_header_ahg(pq->dd, pq->ctxt, pq->subctxt,
 					req->info.comp_idx, req->sde->this_idx,
-					req->ahg_idx, req->ahg, diff, tidval);
+					req->ahg_idx, ahg, diff, tidval);
+	sdma_txinit_ahg(&tx->txreq,
+			SDMA_TXREQ_F_USE_AHG,
+			datalen, req->ahg_idx, diff,
+			ahg, sizeof(req->hdr),
+			user_sdma_txreq_cb);
+
 	return diff;
 }
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170529161657.GC17751-U/[email protected]

#7 lore raw

RE: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

Quoted 1 line

Out of curiosity, do you have pahole data before vs. after to share?

Yes, below is the pahole output for this patch.

Sebastian

---------

Before:

struct user_sdma_request {

	struct sdma_req_info       info;                 /*     0     8 */
	struct hfi1_user_sdma_pkt_q * pq;                /*     8     8 */
	struct hfi1_user_sdma_comp_q * cq;               /*    16     8 */
	struct hfi1_pkt_header     hdr;                  /*    24    64 */

/* --- cacheline 1 boundary (64 bytes) was 24 bytes ago --- */
struct sdma_engine * sde; /* 88 8 */
s8 ahg_idx; /* 96 1 */

/* XXX 3 bytes hole, try to pack */

u32 ahg[9]; /* 100 36 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
u32 koffset; /* 136 4 */
u32 tidoffset; /* 140 4 */
unsigned int data_iovs; /* 144 4 */
u32 data_len; /* 148 4 */
unsigned int iov_idx; /* 152 4 */

/* XXX 4 bytes hole, try to pack */

struct user_sdma_iovec iovs[8]; /* 160 512 */
/* --- cacheline 10 boundary (640 bytes) was 32 bytes ago --- */
u16 n_tids; /* 672 2 */

/* XXX 6 bytes hole, try to pack */

u32 * tids; /* 680 8 */
u16 tididx; /* 688 2 */

/* XXX 2 bytes hole, try to pack */

u32 sent; /* 692 4 */
u64 seqnum; /* 696 8 */
/* --- cacheline 11 boundary (704 bytes) --- */
u64 seqcomp; /* 704 8 */
u64 seqsubmitted; /* 712 8 */
struct list_head txps; /* 720 16 */
long unsigned int flags; /* 736 8 */
int status; /* 744 4 */

/* size: 752, cachelines: 12, members: 23 */ /* sum members: 733, holes: 4, sum holes: 15 */ /* padding: 4 */
/* last cacheline: 48 bytes */
};

After:

struct user_sdma_request {
struct hfi1_pkt_header hdr; /* 0 64 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct hfi1_user_sdma_pkt_q * pq; /* 64 8 */
struct hfi1_user_sdma_comp_q * cq; /* 72 8 */
struct sdma_engine * sde; /* 80 8 */
struct sdma_req_info info; /* 88 8 */
u32 * tids; /* 96 8 */
u32 data_len; /* 104 4 */
u16 n_tids; /* 108 2 */
u8 data_iovs; /* 110 1 */
s8 ahg_idx; /* 111 1 */

/* XXX 16 bytes hole, try to pack */

/* --- cacheline 2 boundary (128 bytes) --- */
u64 seqcomp; /* 128 8 */
u64 seqsubmitted; /* 136 8 */
long unsigned int flags; /* 144 8 */
int status; /* 152 4 */

/* XXX 36 bytes hole, try to pack */

/* --- cacheline 3 boundary (192 bytes) --- */
struct list_head txps; /* 192 16 */
u64 seqnum; /* 208 8 */
u32 tidoffset; /* 216 4 */
u32 koffset; /* 220 4 */
u32 sent; /* 224 4 */
u16 tididx; /* 228 2 */
u8 iov_idx; /* 230 1 */

/* XXX 1 byte hole, try to pack */

struct user_sdma_iovec iovs[8]; /* 232 512 */
/* --- cacheline 11 boundary (704 bytes) was 40 bytes ago --- */

/* size: 768, cachelines: 12, members: 22 */
/* sum members: 691, holes: 3, sum holes: 53 */
/* padding: 24 */
};
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 5CDA63463B33C94CA80846587415F0770408CAF7-8oqHQFITsIGkrb+BlOpmy7fspsVTdybXVpNB7YpNyf8@public.gmane.org

#9 lore raw

Re: [PATCH for-next 2/9] IB/hfi1: Optimize cachelines for user SDMA request structure

Quoted 3 lines · Sebastian
Quoted 1 line

Out of curiosity, do you have pahole data before vs. after to share?

Yes, below is the pahole output for this patch.

Thanks

Quoted 98 lines

Sebastian

---------

Before:

struct user_sdma_request {

	struct sdma_req_info       info;                 /*     0     8 */
	struct hfi1_user_sdma_pkt_q * pq;                /*     8     8 */
	struct hfi1_user_sdma_comp_q * cq;               /*    16     8 */
	struct hfi1_pkt_header     hdr;                  /*    24    64 */

/* --- cacheline 1 boundary (64 bytes) was 24 bytes ago --- */
struct sdma_engine * sde; /* 88 8 */
s8 ahg_idx; /* 96 1 */

/* XXX 3 bytes hole, try to pack */

u32 ahg[9]; /* 100 36 */
/* --- cacheline 2 boundary (128 bytes) was 8 bytes ago --- */
u32 koffset; /* 136 4 */
u32 tidoffset; /* 140 4 */
unsigned int data_iovs; /* 144 4 */
u32 data_len; /* 148 4 */
unsigned int iov_idx; /* 152 4 */

/* XXX 4 bytes hole, try to pack */

struct user_sdma_iovec iovs[8]; /* 160 512 */
/* --- cacheline 10 boundary (640 bytes) was 32 bytes ago --- */
u16 n_tids; /* 672 2 */

/* XXX 6 bytes hole, try to pack */

u32 * tids; /* 680 8 */
u16 tididx; /* 688 2 */

/* XXX 2 bytes hole, try to pack */

u32 sent; /* 692 4 */
u64 seqnum; /* 696 8 */
/* --- cacheline 11 boundary (704 bytes) --- */
u64 seqcomp; /* 704 8 */
u64 seqsubmitted; /* 712 8 */
struct list_head txps; /* 720 16 */
long unsigned int flags; /* 736 8 */
int status; /* 744 4 */

/* size: 752, cachelines: 12, members: 23 */ /* sum members: 733, holes: 4, sum holes: 15 */ /* padding: 4 */
/* last cacheline: 48 bytes */
};

After:

struct user_sdma_request {
struct hfi1_pkt_header hdr; /* 0 64 */
/* --- cacheline 1 boundary (64 bytes) --- */
struct hfi1_user_sdma_pkt_q * pq; /* 64 8 */
struct hfi1_user_sdma_comp_q * cq; /* 72 8 */
struct sdma_engine * sde; /* 80 8 */
struct sdma_req_info info; /* 88 8 */
u32 * tids; /* 96 8 */
u32 data_len; /* 104 4 */
u16 n_tids; /* 108 2 */
u8 data_iovs; /* 110 1 */
s8 ahg_idx; /* 111 1 */

/* XXX 16 bytes hole, try to pack */

/* --- cacheline 2 boundary (128 bytes) --- */
u64 seqcomp; /* 128 8 */
u64 seqsubmitted; /* 136 8 */
long unsigned int flags; /* 144 8 */
int status; /* 152 4 */

/* XXX 36 bytes hole, try to pack */

/* --- cacheline 3 boundary (192 bytes) --- */
struct list_head txps; /* 192 16 */
u64 seqnum; /* 208 8 */
u32 tidoffset; /* 216 4 */
u32 koffset; /* 220 4 */
u32 sent; /* 224 4 */
u16 tididx; /* 228 2 */
u8 iov_idx; /* 230 1 */

/* XXX 1 byte hole, try to pack */

struct user_sdma_iovec iovs[8]; /* 232 512 */
/* --- cacheline 11 boundary (704 bytes) was 40 bytes ago --- */

/* size: 768, cachelines: 12, members: 22 */ /* sum members: 691, holes: 3, sum holes: 53 */ /* padding: 24 */
};

author
#10 lore raw

PATCH 3/9 IB/hfi1: Name function prototype parameters for affinity module

In reply to a message not in the archive

From: Michael J. Ruhl <[email protected]>

To improve the readability of function prototypes, give the parameters names in the affinity module.

Reviewed-by Sebastian Sanchez <[email protected]>
Signed-off-by Michael J. Ruhl <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/affinity.h |   14 ++++++++------
 1 files changed, 8 insertions(+), 6 deletions(-)
drivers/infiniband/hw/hfi1/affinity.h +8 -6
index e78c7aa..2a1e374 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.

@@ -75,24 +75,26 @@ struct cpu_mask_set {
 /* Initialize non-HT cpu cores mask */
 void init_real_cpu_mask(void);
 /* Initialize driver affinity data */
-int hfi1_dev_affinity_init(struct hfi1_devdata *);
+int hfi1_dev_affinity_init(struct hfi1_devdata *dd);
 /*
  * Set IRQ affinity to a CPU. The function will determine the
  * CPU and set the affinity to it.
  */
-int hfi1_get_irq_affinity(struct hfi1_devdata *, struct hfi1_msix_entry *);
+int hfi1_get_irq_affinity(struct hfi1_devdata *dd,
+			  struct hfi1_msix_entry *msix);
 /*
  * Remove the IRQ's CPU affinity. This function also updates
  * any internal CPU tracking data
  */
-void hfi1_put_irq_affinity(struct hfi1_devdata *, struct hfi1_msix_entry *);
+void hfi1_put_irq_affinity(struct hfi1_devdata *dd,
+			   struct hfi1_msix_entry *msix);
 /*
  * Determine a CPU affinity for a user process, if the process does not
  * have an affinity set yet.
  */
-int hfi1_get_proc_affinity(int);
+int hfi1_get_proc_affinity(int node);
 /* Release a CPU used by a user process. */
-void hfi1_put_proc_affinity(int);
+void hfi1_put_proc_affinity(int cpu);
 
 struct hfi1_affinity_node {
 	int node;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#11 lore raw

PATCH 4/9 IB/hfi1: Replace deprecated pci functions with new API

In reply to a message not in the archive

From: Michael J. Ruhl <[email protected]>

pci_enable_msix_range() and pci_disable_msix() have been deprecated. Updating to the new pci_alloc_irq_vectors() interface.

Reviewed-by Sebastian Sanchez <[email protected]>
Signed-off-by Michael J. Ruhl <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/affinity.c |   18 +++---
 drivers/infiniband/hw/hfi1/chip.c     |  105 ++++++++++++++++-----------------
 drivers/infiniband/hw/hfi1/hfi.h      |    6 +-
 drivers/infiniband/hw/hfi1/pcie.c     |   80 +++++--------------------
 4 files changed, 78 insertions(+), 131 deletions(-)
drivers/infiniband/hw/hfi1/affinity.c +9 -9
index e2cd2cd..a97055d 100644
@@ -1,5 +1,5 @@
 /*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2017 Intel Corporation.
  *
  * This file is provided under a dual BSD/GPLv2 license.  When using or
  * redistributing this file, you may do so under either license.

@@ -335,10 +335,10 @@ static void hfi1_update_sdma_affinity(struct hfi1_msix_entry *msix, int cpu)
 	sde->cpu = cpu;
 	cpumask_clear(&msix->mask);
 	cpumask_set_cpu(cpu, &msix->mask);
-	dd_dev_dbg(dd, "IRQ vector: %u, type %s engine %u -> cpu: %d\n",
-		   msix->msix.vector, irq_type_names[msix->type],
+	dd_dev_dbg(dd, "IRQ: %u, type %s engine %u -> cpu: %d\n",
+		   msix->irq, irq_type_names[msix->type],
 		   sde->this_idx, cpu);
-	irq_set_affinity_hint(msix->msix.vector, &msix->mask);
+	irq_set_affinity_hint(msix->irq, &msix->mask);
 
 	/*
 	 * Set the new cpu in the hfi1_affinity_node and clean

@@ -387,7 +387,7 @@ static void hfi1_setup_sdma_notifier(struct hfi1_msix_entry *msix)
 {
 	struct irq_affinity_notify *notify = &msix->notify;
 
-	notify->irq = msix->msix.vector;
+	notify->irq = msix->irq;
 	notify->notify = hfi1_irq_notifier_notify;
 	notify->release = hfi1_irq_notifier_release;
 

@@ -472,10 +472,10 @@ static int get_irq_affinity(struct hfi1_devdata *dd,
 	}
 
 	cpumask_set_cpu(cpu, &msix->mask);
-	dd_dev_info(dd, "IRQ vector: %u, type %s %s -> cpu: %d\n",
-		    msix->msix.vector, irq_type_names[msix->type],
+	dd_dev_info(dd, "IRQ: %u, type %s %s -> cpu: %d\n",
+		    msix->irq, irq_type_names[msix->type],
 		    extra, cpu);
-	irq_set_affinity_hint(msix->msix.vector, &msix->mask);
+	irq_set_affinity_hint(msix->irq, &msix->mask);
 
 	if (msix->type == IRQ_SDMA) {
 		sde->cpu = cpu;

@@ -533,7 +533,7 @@ void hfi1_put_irq_affinity(struct hfi1_devdata *dd,
 		}
 	}
 
-	irq_set_affinity_hint(msix->msix.vector, NULL);
+	irq_set_affinity_hint(msix->irq, NULL);
 	cpumask_clear(&msix->mask);
 	mutex_unlock(&node_affinity.lock);
 }
drivers/infiniband/hw/hfi1/chip.c +51 -54

185 lines not on this page · raw message

drivers/infiniband/hw/hfi1/hfi.h +2 -4
index cc69b51..e1cd468 100644
@@ -521,7 +521,7 @@ static inline void incr_cntr32(u32 *cntr)
 #define MAX_NAME_SIZE 64
 struct hfi1_msix_entry {
 	enum irq_type type;
-	struct msix_entry msix;
+	int irq;
 	void *arg;
 	char name[MAX_NAME_SIZE];
 	cpumask_t mask;

@@ -1839,9 +1839,7 @@ int hfi1_create_port_files(struct ib_device *ibdev, u8 port_num,
 void hfi1_pcie_ddcleanup(struct hfi1_devdata *);
 void hfi1_pcie_flr(struct hfi1_devdata *dd);
 int pcie_speeds(struct hfi1_devdata *dd);
-void request_msix(struct hfi1_devdata *dd, u32 *nent,
-		  struct hfi1_msix_entry *entry);
-void hfi1_enable_intx(struct pci_dev *pdev);
+int request_msix(struct hfi1_devdata *dd, u32 msireq);
 void restore_pci_variables(struct hfi1_devdata *dd);
 int do_pcie_gen3_transition(struct hfi1_devdata *dd);
 int parse_platform_config(struct hfi1_devdata *dd);
drivers/infiniband/hw/hfi1/pcie.c +16 -64

104 lines not on this page · raw message

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#12 lore raw

PATCH 5/9 IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

In reply to a message not in the archive

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

Reviewed-by Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by Michael N. Henry <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/mad.c   |    7 ++-----
 drivers/infiniband/hw/hfi1/mad.h   |    2 +-
 drivers/infiniband/hw/hfi1/verbs.c |    7 ++++++-
 drivers/infiniband/sw/rdmavt/vt.c  |    9 +++++++--
 include/rdma/ib_verbs.h            |    3 ++-
 include/rdma/rdma_vt.h             |    1 +
 6 files changed, 19 insertions(+), 10 deletions(-)
drivers/infiniband/hw/hfi1/mad.c +2 -5
index 5977673..70831ad 100644
@@ -260,6 +260,7 @@ void hfi1_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num)
 	data.issuer_lid = cpu_to_be32(lid);
 	data.ntc_144.lid = data.issuer_lid;
 	data.ntc_144.new_cap_mask = cpu_to_be32(ibp->rvp.port_cap_flags);
+	data.ntc_144.cap_mask3 = cpu_to_be16(ibp->rvp.port_cap3_flags);
 
 	send_trap(ibp, &data, sizeof(data));
 }

@@ -704,11 +705,7 @@ static int __subn_get_opa_portinfo(struct opa_smp *smp, u32 am, u8 *data,
 	buffer_units |= (dd->vl15_init << 11) & OPA_PI_MASK_BUF_UNIT_VL15_INIT;
 	pi->buffer_units = cpu_to_be32(buffer_units);
 
-	pi->opa_cap_mask = cpu_to_be16(OPA_CAP_MASK3_IsSharedSpaceSupported |
-				       OPA_CAP_MASK3_IsEthOnFabricSupported);
-	/* Driver does not support mcast/collective configuration */
-	pi->opa_cap_mask &=
-		cpu_to_be16(~OPA_CAP_MASK3_IsAddrRangeConfigSupported);
+	pi->opa_cap_mask = cpu_to_be16(ibp->rvp.port_cap3_flags);
 	pi->collectivemask_multicastmask = ((HFI1_COLLECTIVE_NR & 0x7)
 					    << 3 | (HFI1_MCAST_NR & 0x7));
drivers/infiniband/hw/hfi1/mad.h +1 -1
index 5aa3fd1..a4e2506 100644
@@ -115,7 +115,7 @@ struct opa_mad_notice_attr {
 			__be32	lid;		/* LID where change occurred */
 			__be32	new_cap_mask;	/* new capability mask */
 			__be16	reserved2;
-			__be16	cap_mask;
+			__be16	cap_mask3;
 			__be16	change_flags;	/* low 4 bits only */
 		} __packed ntc_144;
drivers/infiniband/hw/hfi1/verbs.c +6 -1
index af54d3f..2881798 100644
@@ -1537,9 +1537,14 @@ static void init_ibport(struct hfi1_pportdata *ppd)
 	/* Set the prefix to the default value (see ch. 4.1.1) */
 	ibp->rvp.gid_prefix = IB_DEFAULT_GID_PREFIX;
 	ibp->rvp.sm_lid = 0;
-	/* Below should only set bits defined in OPA PortInfo.CapabilityMask */
+	/*
+	 * Below should only set bits defined in OPA PortInfo.CapabilityMask
+	 * and PortInfo.CapabilityMask3
+	 */
 	ibp->rvp.port_cap_flags = IB_PORT_AUTO_MIGR_SUP |
 		IB_PORT_CAP_MASK_NOTICE_SUP;
+	ibp->rvp.port_cap3_flags = OPA_CAP_MASK3_IsSharedSpaceSupported |
+				   OPA_CAP_MASK3_IsEthOnFabricSupported;
 	ibp->rvp.pma_counter_select[0] = IB_PMA_PORT_XMIT_DATA;
 	ibp->rvp.pma_counter_select[1] = IB_PMA_PORT_RCV_DATA;
 	ibp->rvp.pma_counter_select[2] = IB_PMA_PORT_XMIT_PKTS;
drivers/infiniband/sw/rdmavt/vt.c +7 -2
index 0d7c6bb..64bdd44 100644
@@ -202,8 +202,13 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 port_num,
 		return -EINVAL;
 
 	rvp = rdi->ports[port_index];
-	rvp->port_cap_flags |= props->set_port_cap_mask;
-	rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	if (port_modify_mask & IB_PORT_OPA_MASK_CHG) {
+		rvp->port_cap3_flags |= props->set_port_cap_mask;
+		rvp->port_cap3_flags &= ~props->clr_port_cap_mask;
+	} else {
+		rvp->port_cap_flags |= props->set_port_cap_mask;
+		rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	}
 
 	if (props->set_port_cap_mask || props->clr_port_cap_mask)
 		rdi->driver_f.cap_mask_chg(rdi, port_num);
include/rdma/ib_verbs.h +2 -1
index 785e927..6c4f6da 100644
@@ -576,7 +576,8 @@ struct ib_device_modify {
 enum ib_port_modify_flags {
 	IB_PORT_SHUTDOWN		= 1,
 	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)
 };
 
 struct ib_port_modify {
include/rdma/rdma_vt.h +1 -0
index d0b9f91..0f18ffd 100644
@@ -75,6 +75,7 @@ struct rvt_ibport {
 	__be64 mkey;
 	u64 tid;
 	u32 port_cap_flags;
+	u16 port_cap3_flags;
 	u32 pma_sample_start;
 	u32 pma_sample_interval;
 	__be16 pma_counter_select[5];

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170526123536.8158.237.stgit-9QXIwq+3FY+1XWohqUldA0EOCMrvLtNR@public.gmane.org

#14 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 100 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

Reviewed-by Niranjana Vishwanathapura <niranjana.vishwanathapura-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by Michael N. Henry <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/mad.c   |    7 ++-----
 drivers/infiniband/hw/hfi1/mad.h   |    2 +-
 drivers/infiniband/hw/hfi1/verbs.c |    7 ++++++-
 drivers/infiniband/sw/rdmavt/vt.c  |    9 +++++++--
 include/rdma/ib_verbs.h            |    3 ++-
 include/rdma/rdma_vt.h             |    1 +
 6 files changed, 19 insertions(+), 10 deletions(-)
drivers/infiniband/hw/hfi1/mad.c +2 -5
index 5977673..70831ad 100644
@@ -260,6 +260,7 @@ void hfi1_cap_mask_chg(struct rvt_dev_info *rdi, u8 port_num)
 	data.issuer_lid = cpu_to_be32(lid);
 	data.ntc_144.lid = data.issuer_lid;
 	data.ntc_144.new_cap_mask = cpu_to_be32(ibp->rvp.port_cap_flags);
+	data.ntc_144.cap_mask3 = cpu_to_be16(ibp->rvp.port_cap3_flags);
 
 	send_trap(ibp, &data, sizeof(data));
 }

@@ -704,11 +705,7 @@ static int __subn_get_opa_portinfo(struct opa_smp *smp, u32 am, u8 *data,
 	buffer_units |= (dd->vl15_init << 11) & OPA_PI_MASK_BUF_UNIT_VL15_INIT;
 	pi->buffer_units = cpu_to_be32(buffer_units);
 
-	pi->opa_cap_mask = cpu_to_be16(OPA_CAP_MASK3_IsSharedSpaceSupported |
-				       OPA_CAP_MASK3_IsEthOnFabricSupported);
-	/* Driver does not support mcast/collective configuration */
-	pi->opa_cap_mask &=
-		cpu_to_be16(~OPA_CAP_MASK3_IsAddrRangeConfigSupported);
+	pi->opa_cap_mask = cpu_to_be16(ibp->rvp.port_cap3_flags);
 	pi->collectivemask_multicastmask = ((HFI1_COLLECTIVE_NR & 0x7)
 					    << 3 | (HFI1_MCAST_NR & 0x7));
drivers/infiniband/hw/hfi1/mad.h +1 -1
index 5aa3fd1..a4e2506 100644
@@ -115,7 +115,7 @@ struct opa_mad_notice_attr {
 			__be32	lid;		/* LID where change occurred */
 			__be32	new_cap_mask;	/* new capability mask */
 			__be16	reserved2;
-			__be16	cap_mask;
+			__be16	cap_mask3;
 			__be16	change_flags;	/* low 4 bits only */
 		} __packed ntc_144;
drivers/infiniband/hw/hfi1/verbs.c +6 -1
index af54d3f..2881798 100644
@@ -1537,9 +1537,14 @@ static void init_ibport(struct hfi1_pportdata *ppd)
 	/* Set the prefix to the default value (see ch. 4.1.1) */
 	ibp->rvp.gid_prefix = IB_DEFAULT_GID_PREFIX;
 	ibp->rvp.sm_lid = 0;
-	/* Below should only set bits defined in OPA PortInfo.CapabilityMask */
+	/*
+	 * Below should only set bits defined in OPA PortInfo.CapabilityMask
+	 * and PortInfo.CapabilityMask3
+	 */
 	ibp->rvp.port_cap_flags = IB_PORT_AUTO_MIGR_SUP |
 		IB_PORT_CAP_MASK_NOTICE_SUP;
+	ibp->rvp.port_cap3_flags = OPA_CAP_MASK3_IsSharedSpaceSupported |
+				   OPA_CAP_MASK3_IsEthOnFabricSupported;
 	ibp->rvp.pma_counter_select[0] = IB_PMA_PORT_XMIT_DATA;
 	ibp->rvp.pma_counter_select[1] = IB_PMA_PORT_RCV_DATA;
 	ibp->rvp.pma_counter_select[2] = IB_PMA_PORT_XMIT_PKTS;
drivers/infiniband/sw/rdmavt/vt.c +7 -2
index 0d7c6bb..64bdd44 100644
@@ -202,8 +202,13 @@ static int rvt_modify_port(struct ib_device *ibdev, u8 port_num,
 		return -EINVAL;
 
 	rvp = rdi->ports[port_index];
-	rvp->port_cap_flags |= props->set_port_cap_mask;
-	rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	if (port_modify_mask & IB_PORT_OPA_MASK_CHG) {
+		rvp->port_cap3_flags |= props->set_port_cap_mask;
+		rvp->port_cap3_flags &= ~props->clr_port_cap_mask;
+	} else {
+		rvp->port_cap_flags |= props->set_port_cap_mask;
+		rvp->port_cap_flags &= ~props->clr_port_cap_mask;
+	}
 
 	if (props->set_port_cap_mask || props->clr_port_cap_mask)
 		rdi->driver_f.cap_mask_chg(rdi, port_num);
include/rdma/ib_verbs.h +0 -0
index 785e927..6c4f6da 100644

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 6 lines
include/rdma/ib_verbs.h +2 -1
@@ -576,7 +576,8 @@ struct ib_device_modify {
 enum ib_port_modify_flags {
 	IB_PORT_SHUTDOWN		= 1,
 	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

Quoted 20 lines
include/rdma/ib_verbs.h +0 -0
 };
 
 struct ib_port_modify {
include/rdma/rdma_vt.h +1 -1
index d0b9f91..0f18ffd 100644
@@ -75,6 +75,7 @@ struct rvt_ibport {
 	__be64 mkey;
 	u64 tid;
 	u32 port_cap_flags;
+	u16 port_cap3_flags;
 	u32 pma_sample_start;
 	u32 pma_sample_interval;
 	__be16 pma_counter_select[5];
 
--

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170529161407.GB17751-U/[email protected]

#16 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 6 lines · Leon Romanovsky
Quoted 5 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

Quoted 13 lines

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 11 lines
include/rdma/ib_verbs.h +2 -1
index 785e927..6c4f6da 100644
@@ -576,7 +576,8 @@ struct ib_device_modify {
  enum ib_port_modify_flags {
  	IB_PORT_SHUTDOWN		= 1,
  	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

This flag can be set as the port_modify_mask when calling ib_modify_port(..)

Thanks,

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 2fc7a7d6-9ba3-2365-45e3-946f56827dd9-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#18 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 20 lines · Michael N
Quoted 6 lines · Leon Romanovsky
Quoted 5 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

Quoted 13 lines

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 11 lines
include/rdma/ib_verbs.h +2 -1
index 785e927..6c4f6da 100644
@@ -576,7 +576,8 @@ struct ib_device_modify {
  enum ib_port_modify_flags {
  	IB_PORT_SHUTDOWN		= 1,
  	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

Quoted 1 line

This flag can be set as the port_modify_mask when calling ib_modify_port(..)

My question is more general. I'm as a representative of these "IB clients" would like to know when and why should I use this flag.

Thanks

Quoted 3 lines

-Mike

A message that is not in the archive (sent off-list or not yet received): 20170531043235.GH5406-U/[email protected]

#20 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Michael N
Quoted 19 lines · Leon Romanovsky
Quoted 10 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

include/rdma/ib_verbs.h +0 -0
index 785e927..6c4f6da 100644

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 6 lines
include/rdma/ib_verbs.h +2 -1
@@ -576,7 +576,8 @@ struct ib_device_modify {
  enum ib_port_modify_flags {
  	IB_PORT_SHUTDOWN		= 1,
  	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

Quoted 5 lines
Quoted 1 line

This flag can be set as the port_modify_mask when calling ib_modify_port(..)

My question is more general. I'm as a representative of these "IB clients" would like to know when and why should I use this flag.

Hi,
We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as below.

pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported; ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);

This patch support such usecase and we want to get comments early (if any) for that, hence this patch.

Niranjana

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20170531175821.GA29853-wPcXA7LoDC+1XWohqUldA0EOCMrvLtNR@public.gmane.org

#22 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 32 lines · Niranjana
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Michael N
Quoted 19 lines · Leon Romanovsky
Quoted 10 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

include/rdma/ib_verbs.h +0 -0
index 785e927..6c4f6da 100644

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 6 lines
include/rdma/ib_verbs.h +2 -1
@@ -576,7 +576,8 @@ struct ib_device_modify {
  enum ib_port_modify_flags {
  	IB_PORT_SHUTDOWN		= 1,
  	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

Quoted 5 lines
Quoted 1 line

This flag can be set as the port_modify_mask when calling ib_modify_port(..)

My question is more general. I'm as a representative of these "IB clients" would like to know when and why should I use this flag.

Hi,
We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as below.

Quoted 5 lines

pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported; ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);

This patch support such usecase and we want to get comments early (if any) for that, hence this patch.

The comment is pretty simple, please don't post patches which are not backed up by actual usage.

Right now, we have IB_PORT_SHUTDOWN, IB_PORT_INIT_TYPE and IB_PORT_RESET_QKEY_CNTR are not written in any place. We don't want to see another one.

And semantic of usage are more than welcomed.

Thanks

Quoted 3 lines

Niranjana

A message that is not in the archive (sent off-list or not yet received): 20170601055542.GQ5406-U/[email protected]

#24 lore raw

Re: [PATCH for-next 5/9] IB/core, rdmavt, hfi1: Send OPA cap_mask3 in trap

Quoted 39 lines · Leon Romanovsky
Quoted 32 lines · Niranjana
Quoted 21 lines · Leon Romanovsky
Quoted 20 lines · Michael N
Quoted 19 lines · Leon Romanovsky
Quoted 10 lines · Dennis Dalessandro

From: Michael N. Henry <[email protected]>

Provide the ability for IB clients to modify the OPA specific capability mask and include this mask in the subsequent trap data.

include/rdma/ib_verbs.h +0 -0
index 785e927..6c4f6da 100644

include/rdma/ib_verbs.h struct ib_device_modify { · line 580

Quoted 6 lines
include/rdma/ib_verbs.h +2 -1
@@ -576,7 +576,8 @@ struct ib_device_modify {
  enum ib_port_modify_flags {
  	IB_PORT_SHUTDOWN		= 1,
  	IB_PORT_INIT_TYPE		= (1<<2),
-	IB_PORT_RESET_QKEY_CNTR		= (1<<3)
+	IB_PORT_RESET_QKEY_CNTR		= (1<<3),
+	IB_PORT_OPA_MASK_CHG		= (1<<4)

How will this flag be set?

Quoted 5 lines
Quoted 1 line

This flag can be set as the port_modify_mask when calling ib_modify_port(..)

My question is more general. I'm as a representative of these "IB clients" would like to know when and why should I use this flag.

Hi,
We have an OPA VNIC use case (patch coming up) to set/clear OPA capmask as below.

Quoted 5 lines

pm.set/clear_port_cap_mask = OPA_CAP_MASK3_IsEthOnFabricSupported; ib_modify_port(hca, port, IB_PORT_OPA_MASK_CHG, &pm);

This patch support such usecase and we want to get comments early (if any) for that, hence this patch.

Quoted 9 lines

The comment is pretty simple, please don't post patches which are not backed up by actual usage.

Right now, we have IB_PORT_SHUTDOWN, IB_PORT_INIT_TYPE and IB_PORT_RESET_QKEY_CNTR are not written in any place. We don't want to see another one.

And semantic of usage are more than welcomed.

Revised patch along with OPA-VNIC usage is posted below
https://patchwork.kernel.org/patch/9761375/

Niranjana

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#25 lore raw

PATCH 6/9 IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation

In reply to a message not in the archive

From: Sebastian Sanchez <[email protected]>

The atomic SDMA_REQ_SEND_DONE bit is set by the process-level code, and then the same process-level code uses the bit to test that all packets have been submitted incurring a costly atomic read.

Use a bool type with a READ_ONCE/WRITE_ONCE pairing for this bit, and use the same condition that is used to set the bit to test that all packets have been submitted.

Reviewed-by Mike Marciniszyn <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   14 +++++++-------
 1 files changed, 7 insertions(+), 7 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.c +7 -7
index 92517ce..6fb70f0 100644
@@ -154,10 +154,8 @@
 #define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
 
 /* SDMA request flag bits */
-#define SDMA_REQ_FOR_THREAD 1
-#define SDMA_REQ_SEND_DONE  2
-#define SDMA_REQ_HAS_ERROR  3
-#define SDMA_REQ_DONE_ERROR 4
+#define SDMA_REQ_HAS_ERROR  1
+#define SDMA_REQ_DONE_ERROR 2
 
 #define SDMA_PKT_Q_INACTIVE BIT(0)
 #define SDMA_PKT_Q_ACTIVE   BIT(1)

@@ -258,6 +256,7 @@ struct user_sdma_request {
 	u16 tididx;
 	/* progress index moving along the iovs array */
 	u8 iov_idx;
+	u8 done;
 
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
 } ____cacheline_aligned_in_smp;

@@ -628,6 +627,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	req->seqsubmitted = 0;
 	req->flags = 0;
 	req->tids = NULL;
+	req->done = 0;
 	INIT_LIST_HEAD(&req->txps);
 
 	memcpy(&req->info, &info, sizeof(info));

@@ -809,7 +809,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	 * request have been submitted to the SDMA engine. However, it
 	 * will not wait for send completions.
 	 */
-	while (!test_bit(SDMA_REQ_SEND_DONE, &req->flags)) {
+	while (req->seqsubmitted != req->info.npkts) {
 		ret = user_sdma_send_pkts(req, pcount);
 		if (ret < 0) {
 			if (ret != -EBUSY) {

@@ -1118,7 +1118,7 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 	ret = sdma_send_txlist(req->sde, &pq->busy, &req->txps, &count);
 	req->seqsubmitted += count;
 	if (req->seqsubmitted == req->info.npkts) {
-		set_bit(SDMA_REQ_SEND_DONE, &req->flags);
+		WRITE_ONCE(req->done, 1);
 		/*
 		 * The txreq has already been submitted to the HW queue
 		 * so we can free the AHG entry now. Corruption will not

@@ -1585,7 +1585,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 		if (status != SDMA_TXREQ_S_OK)
 			req->status = status;
 		if (req->seqcomp == (ACCESS_ONCE(req->seqsubmitted) - 1) &&
-		    (test_bit(SDMA_REQ_SEND_DONE, &req->flags) ||
+		    (READ_ONCE(req->done) ||
 		     test_bit(SDMA_REQ_DONE_ERROR, &req->flags))) {
 			user_sdma_free_request(req, false);
 			pq_update(pq);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#26 lore raw

PATCH 7/9 IB/hfi1: Remove atomic SDMA_REQ_HAS_ERROR bit operation

In reply to a message not in the archive

From: Sebastian Sanchez <[email protected]>

Atomic bit tests are used to single errors and the completion of request submissions. These operations don't need to be atomic and show to be expensive on the profile.

Replace each atomic bit operation with a bool type and a READ_ONCE/WRITE_ONCE pairing.

Reviewed-by Mike Marciniszyn <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/user_sdma.c |   22 +++++++---------------
 1 files changed, 7 insertions(+), 15 deletions(-)
drivers/infiniband/hw/hfi1/user_sdma.c +7 -15
index 6fb70f0..fcadbb9 100644
@@ -153,10 +153,6 @@
 #define TXREQ_FLAGS_REQ_ACK   BIT(0)      /* Set the ACK bit in the header */
 #define TXREQ_FLAGS_REQ_DISABLE_SH BIT(1) /* Disable header suppression */
 
-/* SDMA request flag bits */
-#define SDMA_REQ_HAS_ERROR  1
-#define SDMA_REQ_DONE_ERROR 2
-
 #define SDMA_PKT_Q_INACTIVE BIT(0)
 #define SDMA_PKT_Q_ACTIVE   BIT(1)
 #define SDMA_PKT_Q_DEFERRED BIT(2)

@@ -232,7 +228,6 @@ struct user_sdma_request {
 	/* Writeable fields shared with interrupt */
 	u64 seqcomp ____cacheline_aligned_in_smp;
 	u64 seqsubmitted;
-	unsigned long flags;
 	/* status of the last txreq completed */
 	int status;
 

@@ -257,6 +252,7 @@ struct user_sdma_request {
 	/* progress index moving along the iovs array */
 	u8 iov_idx;
 	u8 done;
+	u8 has_error;
 
 	struct user_sdma_iovec iovs[MAX_VECTORS_PER_REQ];
 } ____cacheline_aligned_in_smp;

@@ -625,9 +621,9 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 	req->seqnum = 0;
 	req->seqcomp = 0;
 	req->seqsubmitted = 0;
-	req->flags = 0;
 	req->tids = NULL;
 	req->done = 0;
+	req->has_error = 0;
 	INIT_LIST_HEAD(&req->txps);
 
 	memcpy(&req->info, &info, sizeof(info));

@@ -814,7 +810,7 @@ int hfi1_user_sdma_process_request(struct hfi1_filedata *fd,
 		if (ret < 0) {
 			if (ret != -EBUSY) {
 				req->status = ret;
-				set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+				WRITE_ONCE(req->has_error, 1);
 				if (ACCESS_ONCE(req->seqcomp) ==
 				    req->seqsubmitted - 1)
 					goto free_req;

@@ -916,10 +912,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 	pq = req->pq;
 
 	/* If tx completion has reported an error, we are done. */
-	if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
-		set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+	if (READ_ONCE(req->has_error))
 		return -EFAULT;
-	}
 
 	/*
 	 * Check if we might have sent the entire request already

@@ -942,10 +936,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
 		 * with errors. If so, we are not going to process any
 		 * more packets from this request.
 		 */
-		if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
-			set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+		if (READ_ONCE(req->has_error))
 			return -EFAULT;
-		}
 
 		tx = kmem_cache_alloc(pq->txreq_cache, GFP_KERNEL);
 		if (!tx)

@@ -1566,7 +1558,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 	if (status != SDMA_TXREQ_S_OK) {
 		SDMA_DBG(req, "SDMA completion with error %d",
 			 status);
-		set_bit(SDMA_REQ_HAS_ERROR, &req->flags);
+		WRITE_ONCE(req->has_error, 1);
 	}
 
 	req->seqcomp = tx->seqnum;

@@ -1586,7 +1578,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
 			req->status = status;
 		if (req->seqcomp == (ACCESS_ONCE(req->seqsubmitted) - 1) &&
 		    (READ_ONCE(req->done) ||
-		     test_bit(SDMA_REQ_DONE_ERROR, &req->flags))) {
+		     READ_ONCE(req->has_error))) {
 			user_sdma_free_request(req, false);
 			pq_update(pq);
 			set_comp_state(pq, cq, idx, ERROR, req->status);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#27 lore raw

PATCH 8/9 IB/hfi1: Reclassify type of messages printed for platform config logic

In reply to a message not in the archive

From: Sebastian Sanchez <[email protected]>

Reclassify messages printed out to /var/log/messages into warnings and errors to facilitate debugging in the future for issues related to the platform config logic.

Reviewed-by Mike Marciniszyn <[email protected]>
Signed-off-by Sebastian Sanchez <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/platform.c |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)
drivers/infiniband/hw/hfi1/platform.c +10 -10
index 838fe84..cbda9b1 100644
@@ -242,7 +242,7 @@ static int qual_power(struct hfi1_pportdata *ppd)
 
 	if (ppd->offline_disabled_reason ==
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_POWER_POLICY)) {
-		dd_dev_info(
+		dd_dev_err(
 			ppd->dd,
 			"%s: Port disabled due to system power restrictions\n",
 			__func__);

@@ -268,7 +268,7 @@ static int qual_bitrate(struct hfi1_pportdata *ppd)
 
 	if (ppd->offline_disabled_reason ==
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_LINKSPEED_POLICY)) {
-		dd_dev_info(
+		dd_dev_err(
 			ppd->dd,
 			"%s: Cable failed bitrate check, disabling port\n",
 			__func__);

@@ -709,15 +709,15 @@ static void apply_tunings(
 		ret = load_8051_config(ppd->dd, DC_HOST_COMM_SETTINGS,
 				       GENERAL_CONFIG, config_data);
 		if (ret != HCMD_SUCCESS)
-			dd_dev_info(ppd->dd,
-				    "%s: Failed set ext device config params\n",
-				    __func__);
+			dd_dev_err(ppd->dd,
+				   "%s: Failed set ext device config params\n",
+				   __func__);
 	}
 
 	if (tx_preset_index == OPA_INVALID_INDEX) {
 		if (ppd->port_type == PORT_TYPE_QSFP && limiting_active)
-			dd_dev_info(ppd->dd, "%s: Invalid Tx preset index\n",
-				    __func__);
+			dd_dev_err(ppd->dd, "%s: Invalid Tx preset index\n",
+				   __func__);
 		return;
 	}
 

@@ -900,7 +900,7 @@ static int tune_qsfp(struct hfi1_pportdata *ppd,
 	case 0xD: /* fallthrough */
 	case 0xF:
 	default:
-		dd_dev_info(ppd->dd, "%s: Unknown/unsupported cable\n",
+		dd_dev_warn(ppd->dd, "%s: Unknown/unsupported cable\n",
 			    __func__);
 		break;
 	}

@@ -942,7 +942,7 @@ void tune_serdes(struct hfi1_pportdata *ppd)
 	case PORT_TYPE_DISCONNECTED:
 		ppd->offline_disabled_reason =
 			HFI1_ODR_MASK(OPA_LINKDOWN_REASON_DISCONNECTED);
-		dd_dev_info(dd, "%s: Port disconnected, disabling port\n",
+		dd_dev_warn(dd, "%s: Port disconnected, disabling port\n",
 			    __func__);
 		goto bail;
 	case PORT_TYPE_FIXED:

@@ -1027,7 +1027,7 @@ void tune_serdes(struct hfi1_pportdata *ppd)
 		}
 		break;
 	default:
-		dd_dev_info(ppd->dd, "%s: Unknown port type\n", __func__);
+		dd_dev_warn(ppd->dd, "%s: Unknown port type\n", __func__);
 		ppd->port_type = PORT_TYPE_UNKNOWN;
 		tuning_method = OPA_UNKNOWN_TUNING;
 		total_atten = 0;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#28 lore raw

PATCH 9/9 IB/hfi1: Fix up sdma_init function comment

In reply to a message not in the archive

From: Ira Weiny <[email protected]>

sdma_init does not take a number of sdma engine parameters, rather it initializes all of the sdma engines.

Signed-off-by Ira Weiny <[email protected]>
Signed-off-by Dennis Dalessandro <[email protected]>
---
 drivers/infiniband/hw/hfi1/sdma.c |    6 ++----
 1 files changed, 2 insertions(+), 4 deletions(-)
drivers/infiniband/hw/hfi1/sdma.c +2 -4
index bfd0d51..d82ff57 100644
@@ -1340,10 +1340,8 @@ static void sdma_clean(struct hfi1_devdata *dd, size_t num_engines)
  * @dd: hfi1_devdata
  * @port: port number (currently only zero)
  *
- * sdma_init initializes the specified number of engines.
- *
- * The code initializes each sde, its csrs.  Interrupts
- * are not required to be enabled.
+ * Initializes each sde and its csrs.
+ * Interrupts are not required to be enabled.
  *
  * Returns:
  * 0 - success, -errno on failure

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#29 lore raw
In reply to a message not in the archive
Quoted 11 lines · Dennis Dalessandro

Hi Doug,

Another set of patches for 4.13. These apply on top of the last submitted
series: http://marc.info/?l=linux-rdma&m=149460607918479&w=2. I wanted to
submit this last week but 0-day build service and my GitHub weren't seeing
eye to eye. Things are working now and these have passed 0-day builds.

Quoted 7 lines

These are mostly fixes for the hfi1 driver, but there is a core change and one
for rdmavt as well which adds the ability to send OPA cap_mask3 in a trap.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.13

Hi Denny,

Series, with updated patch 5/9 from Niranjana, applied.

Signature · 8 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html