Skip to contentSkip to conversation

ipath driver updates for 2.6.17-rc4

PATCH In review Bryan O'Sullivan 8 participants 87 messages May 12, 2006 – May 16, 2006 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ipath driver updates for 2.6.17-rc4

Hi, Roland -

Here is a series of patches to bring the ipath driver up to date. I believe you may already have two of them (but I've included them just in case), but the others should all be new.

They apply on top of Linus's current -git.

Cheers,

	<b
#1 lore raw

PATCH 1/1 ipath - fix spinlock recursion bug

The local loopback path for RC can lock the rkey table lock without blocking interrupts. The receive interrupt path can then call ipath_rkey_ok() and deadlock. Since the lock only protects a 64 bit read, the lock isn't needed.

drivers/infiniband/hw/ipath/ipath_keys.c +0 -6
diff -r 89f7c69a68bf -r 9b9f24aab350 drivers/infiniband/hw/ipath/ipath_keys.c
@@ -136,9 +136,7 @@ int ipath_lkey_ok(struct ipath_lkey_tabl
 		ret = 1;
 		goto bail;
 	}
-	spin_lock(&rkt->lock);
 	mr = rkt->table[(sge->lkey >> (32 - ib_ipath_lkey_table_size))];
-	spin_unlock(&rkt->lock);
 	if (unlikely(mr == NULL || mr->lkey != sge->lkey)) {
 		ret = 0;
 		goto bail;
@@ -184,8 +182,6 @@ bail:
  * @acc: access flags
  *
  * Return 1 if successful, otherwise 0.
- *
- * The QP r_rq.lock should be held.
  */
 int ipath_rkey_ok(struct ipath_ibdev *dev, struct ipath_sge_state *ss,
 		  u32 len, u64 vaddr, u32 rkey, int acc)
@@ -196,9 +192,7 @@ int ipath_rkey_ok(struct ipath_ibdev *de
 	size_t off;
 	int ret;
 
-	spin_lock(&rkt->lock);
 	mr = rkt->table[(rkey >> (32 - ib_ipath_lkey_table_size))];
-	spin_unlock(&rkt->lock);
 	if (unlikely(mr == NULL || mr->lkey != rkey)) {
 		ret = 0;
 		goto bail;
#2 lore raw

Re: [PATCH 1 of 53] ipath - fix spinlock recursion bug

Quoted 4 lines · Bryan O'Sullivan

The local loopback path for RC can lock the rkey table lock without blocking interrupts. The receive interrupt path can then call ipath_rkey_ok() and deadlock. Since the lock only protects a 64 bit read, the lock isn't needed.

Uhhh, a 64 bit read is not atomic on all architectures. Certainly not i386.

Might want to verify safety of this.

#3 lore raw

PATCH 1/1 ipath - cap number of PDs that can be allocated

In reply to Bryan O'Sullivan

Put an arbitrary cap on the maximum number of PDs that can be allocated for a device. This is arbitrary because the number we support is constrained only by system memory and what kmalloc can give us. Nevertheless, if we don't have a limit, some third-party OpenIB stress tests fail. The limit can be changed on the fly using a module parameter.

drivers/infiniband/hw/ipath/ipath_verbs.c +24 -1
diff -r 5d5e1e641b16 -r 300f0aa6f034 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -54,6 +54,11 @@ unsigned int ib_ipath_debug;	/* debug ma
 unsigned int ib_ipath_debug;	/* debug mask */
 module_param_named(debug, ib_ipath_debug, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(debug, "Verbs debug mask");
+
+static unsigned int ib_ipath_max_pds = 0xFFFF;
+module_param_named(max_pds, ib_ipath_max_pds, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_pds,
+		 "Maximum number of protection domains to support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -589,7 +594,7 @@ static int ipath_query_device(struct ib_
 	props->max_cq = 0xffff;
 	props->max_cqe = 0xffff;
 	props->max_mr = dev->lk_table.max;
-	props->max_pd = 0xffff;
+	props->max_pd = ib_ipath_max_pds;
 	props->max_qp_rd_atom = 1;
 	props->max_qp_init_rd_atom = 1;
 	/* props->max_res_rd_atom */
@@ -743,8 +748,23 @@ static struct ib_pd *ipath_alloc_pd(stru
 				    struct ib_ucontext *context,
 				    struct ib_udata *udata)
 {
+	struct ipath_ibdev *dev = to_idev(ibdev);
 	struct ipath_pd *pd;
 	struct ib_pd *ret;
+
+	/*
+	 * This is actually totally arbitrary.	Some correctness tests
+	 * assume there's a maximum number of PDs that can be allocated.
+	 * We don't actually have this limit, but we fail the test if
+	 * we allow allocations of more than we report for this value.
+	 */
+
+	if (dev->n_pds_allocated == ib_ipath_max_pds) {
+		ret = ERR_PTR(-ENOMEM);
+		goto bail;
+	}
+
+	dev->n_pds_allocated++;
 
 	pd = kmalloc(sizeof *pd, GFP_KERNEL);
 	if (!pd) {
@@ -764,6 +784,9 @@ static int ipath_dealloc_pd(struct ib_pd
 static int ipath_dealloc_pd(struct ib_pd *ibpd)
 {
 	struct ipath_pd *pd = to_ipd(ibpd);
+	struct ipath_ibdev *dev = to_idev(ibpd->device);
+
+	dev->n_pds_allocated--;
 
 	kfree(pd);
 
drivers/infiniband/hw/ipath/ipath_verbs.h +1 -0
diff -r 5d5e1e641b16 -r 300f0aa6f034 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -431,6 +431,7 @@ struct ipath_ibdev {
 	__be64 sys_image_guid;	/* in network order */
 	__be64 gid_prefix;	/* in network order */
 	__be64 mkey;
+	u32 n_pds_allocated;	/* number of PDs allocated for device */
 	u64 ipath_sword;	/* total dwords sent (sample result) */
 	u64 ipath_rword;	/* total dwords received (sample result) */
 	u64 ipath_spkts;	/* total packets sent (sample result) */
#4 lore raw

Re: [PATCH 4 of 53] ipath - cap number of PDs that can be allocated

Quoted 5 lines

Put an arbitrary cap on the maximum number of PDs that can be allocated for a device. This is arbitrary because the number we support is constrained only by system memory and what kmalloc can give us. Nevertheless, if we don't have a limit, some third-party OpenIB stress tests fail. The limit can be changed on the fly using a module parameter.

Would it make more sense to fix the stress test?

- R.

#5 lore raw

Re: [PATCH 4 of 53] ipath - cap number of PDs that can be allocated

Quoted 1 line · Roland Dreier

Would it make more sense to fix the stress test?

I don't think so. Without some kind of limits, it is simple for an unprivileged user process to cause the kernel to allocate huge wads of memory and thereby DoS or accidentally OOM the machine.

The test in question should probably be fixed, but this is a much more fundamental problem. I don't have any specific opinions on what should be done about it, other than "something".

	<b
#6 lore raw

PATCH 1/1 ipath - report max MR and QP sizes based on table sizes

In reply to Bryan O'Sullivan

Report max MR based on the lkey table size.
Report max QP based on the QP table size.

drivers/infiniband/hw/ipath/ipath_verbs.c +2 -2
diff -r 3ab7a7b10bf2 -r 5d5e1e641b16 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -583,12 +583,12 @@ static int ipath_query_device(struct ib_
 	props->sys_image_guid = dev->sys_image_guid;
 
 	props->max_mr_size = ~0ull;
-	props->max_qp = 0xffff;
+	props->max_qp = dev->qp_table.max;
 	props->max_qp_wr = 0xffff;
 	props->max_sge = 255;
 	props->max_cq = 0xffff;
 	props->max_cqe = 0xffff;
-	props->max_mr = 0xffff;
+	props->max_mr = dev->lk_table.max;
 	props->max_pd = 0xffff;
 	props->max_qp_rd_atom = 1;
 	props->max_qp_init_rd_atom = 1;
#7 lore raw

PATCH 1/1 ipath - make max mcast sizes configurable

In reply to Bryan O'Sullivan

Make the max IB mcast sizes configurable.

drivers/infiniband/hw/ipath/ipath_verbs.c +31 -5
diff -r c5f3731224bb -r df954e47ff67 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -81,6 +81,32 @@ unsigned int ib_ipath_max_sges = 0xFF;
 unsigned int ib_ipath_max_sges = 0xFF;
 module_param_named(max_sges, ib_ipath_max_sges, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_sges, "Maximum number of SGEs to support");
+
+unsigned int ib_ipath_max_mcast_grps = 16384;
+module_param_named(max_mcast_grps, ib_ipath_max_mcast_grps, uint,
+		   S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_mcast_grps,
+		 "Maximum number of multicast groups to support");
+
+unsigned int ib_ipath_max_mcast_qp_attached = 16;
+module_param_named(max_mcast_qp_attached, ib_ipath_max_mcast_qp_attached,
+		   uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_mcast_qp_attached,
+		 "Maximum number of attached QPs to support");
+
+unsigned int ib_ipath_max_srqs = 1024;
+module_param_named(max_srqs, ib_ipath_max_srqs, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_srqs, "Maximum number of SRQs to support");
+
+unsigned int ib_ipath_max_srq_sges = 128;
+module_param_named(max_srq_sges, ib_ipath_max_srq_sges,
+		   uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_srq_sges, "Maximum number of SRQ SGEs to support");
+
+unsigned int ib_ipath_max_srq_wrs = 0x1FFFF;
+module_param_named(max_srq_wrs, ib_ipath_max_srq_wrs,
+		   uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_srq_wrs, "Maximum number of SRQ WRs support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -621,14 +647,14 @@ static int ipath_query_device(struct ib_
 	props->max_qp_rd_atom = 1;
 	props->max_qp_init_rd_atom = 1;
 	/* props->max_res_rd_atom */
-	props->max_srq = 0xffff;
-	props->max_srq_wr = 0xffff;
-	props->max_srq_sge = 255;
+	props->max_srq = ib_ipath_max_srqs;
+	props->max_srq_wr = ib_ipath_max_srq_wrs;
+	props->max_srq_sge = ib_ipath_max_srq_sges;
 	/* props->local_ca_ack_delay */
 	props->atomic_cap = IB_ATOMIC_HCA;
 	props->max_pkeys = ipath_layer_get_npkeys(dev->dd);
-	props->max_mcast_grp = 0xffff;
-	props->max_mcast_qp_attach = 0xffff;
+	props->max_mcast_grp = ib_ipath_max_mcast_grps;
+	props->max_mcast_qp_attach = ib_ipath_max_mcast_qp_attached;
 	props->max_total_mcast_qp_attach = props->max_mcast_qp_attach *
 		props->max_mcast_grp;
 
drivers/infiniband/hw/ipath/ipath_verbs.h +12 -0
diff -r c5f3731224bb -r df954e47ff67 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -148,6 +148,7 @@ struct ipath_mcast {
 	struct list_head qp_list;
 	wait_queue_head_t wait;
 	atomic_t refcount;
+	int n_attached;
 };
 
 /* Memory region */
@@ -434,6 +435,7 @@ struct ipath_ibdev {
 	u32 n_pds_allocated;	/* number of PDs allocated for device */
 	u32 n_ahs_allocated;	/* number of AHs allocated for device */
 	u32 n_cqs_allocated;	/* number of CQs allocated for device */
+	u32 n_mcast_grps_allocated; /* number of mcast groups allocated */
 	u64 ipath_sword;	/* total dwords sent (sample result) */
 	u64 ipath_rword;	/* total dwords received (sample result) */
 	u64 ipath_spkts;	/* total packets sent (sample result) */
@@ -699,6 +701,16 @@ extern unsigned int ib_ipath_max_qp_wrs;
 
 extern unsigned int ib_ipath_max_sges;
 
+extern unsigned int ib_ipath_max_mcast_grps;
+
+extern unsigned int ib_ipath_max_mcast_qp_attached;
+
+extern unsigned int ib_ipath_max_srqs;
+
+extern unsigned int ib_ipath_max_srq_sges;
+
+extern unsigned int ib_ipath_max_srq_wrs;
+
 extern const u32 ib_ipath_rnr_table[];
 
 #endif				/* IPATH_VERBS_H */
drivers/infiniband/hw/ipath/ipath_verbs_mcast.c +31 -7
diff -r c5f3731224bb -r df954e47ff67 drivers/infiniband/hw/ipath/ipath_verbs_mcast.c
@@ -92,6 +92,7 @@ static struct ipath_mcast *ipath_mcast_a
 	INIT_LIST_HEAD(&mcast->qp_list);
 	init_waitqueue_head(&mcast->wait);
 	atomic_set(&mcast->refcount, 0);
+	mcast->n_attached = 0;
 
 bail:
 	return mcast;
@@ -157,7 +158,8 @@ bail:
  * the table but the QP was added.  Return ESRCH if the QP was already
  * attached and neither structure was added.
  */
-static int ipath_mcast_add(struct ipath_mcast *mcast,
+static int ipath_mcast_add(struct ipath_ibdev *dev,
+			   struct ipath_mcast *mcast,
 			   struct ipath_mcast_qp *mqp)
 {
 	struct rb_node **n = &mcast_tree.rb_node;
@@ -188,16 +190,28 @@ static int ipath_mcast_add(struct ipath_
 		/* Search the QP list to see if this is already there. */
 		list_for_each_entry_rcu(p, &tmcast->qp_list, list) {
 			if (p->qp == mqp->qp) {
-				spin_unlock_irqrestore(&mcast_lock, flags);
 				ret = ESRCH;
 				goto bail;
 			}
 		}
+		if (tmcast->n_attached == ib_ipath_max_mcast_qp_attached) {
+			ret = ENOMEM;
+			goto bail;
+		}
+
+		tmcast->n_attached++;
+
 		list_add_tail_rcu(&mqp->list, &tmcast->qp_list);
-		spin_unlock_irqrestore(&mcast_lock, flags);
 		ret = EEXIST;
 		goto bail;
 	}
+
+	if (dev->n_mcast_grps_allocated == ib_ipath_max_mcast_grps) {
+		ret = ENOMEM;
+		goto bail;
+	}
+
+	dev->n_mcast_grps_allocated++;
 
 	list_add_tail_rcu(&mqp->list, &mcast->qp_list);
 
@@ -205,17 +219,18 @@ static int ipath_mcast_add(struct ipath_
 	rb_link_node(&mcast->rb_node, pn, n);
 	rb_insert_color(&mcast->rb_node, &mcast_tree);
 
+	ret = 0;
+
+bail:
 	spin_unlock_irqrestore(&mcast_lock, flags);
 
-	ret = 0;
-
-bail:
 	return ret;
 }
 
 int ipath_multicast_attach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
 {
 	struct ipath_qp *qp = to_iqp(ibqp);
+	struct ipath_ibdev *dev = to_idev(ibqp->device);
 	struct ipath_mcast *mcast;
 	struct ipath_mcast_qp *mqp;
 	int ret;
@@ -235,7 +250,7 @@ int ipath_multicast_attach(struct ib_qp 
 		ret = -ENOMEM;
 		goto bail;
 	}
-	switch (ipath_mcast_add(mcast, mqp)) {
+	switch (ipath_mcast_add(dev, mcast, mqp)) {
 	case ESRCH:
 		/* Neither was used: can't attach the same QP twice. */
 		ipath_mcast_qp_free(mqp);
@@ -245,6 +260,12 @@ int ipath_multicast_attach(struct ib_qp 
 	case EEXIST:		/* The mcast wasn't used */
 		ipath_mcast_free(mcast);
 		break;
+	case ENOMEM:
+		/* Exceeded the maximum number of mcast groups. */
+		ipath_mcast_qp_free(mqp);
+		ipath_mcast_free(mcast);
+		ret = -ENOMEM;
+		goto bail;
 	default:
 		break;
 	}
@@ -258,6 +279,7 @@ int ipath_multicast_detach(struct ib_qp 
 int ipath_multicast_detach(struct ib_qp *ibqp, union ib_gid *gid, u16 lid)
 {
 	struct ipath_qp *qp = to_iqp(ibqp);
+	struct ipath_ibdev *dev = to_idev(ibqp->device);
 	struct ipath_mcast *mcast = NULL;
 	struct ipath_mcast_qp *p, *tmp;
 	struct rb_node *n;
@@ -296,6 +318,7 @@ int ipath_multicast_detach(struct ib_qp 
 		 * link until we are sure there are no list walkers.
 		 */
 		list_del_rcu(&p->list);
+		mcast->n_attached--;
 
 		/* If this was the last attached QP, remove the GID too. */
 		if (list_empty(&mcast->qp_list)) {
@@ -319,6 +342,7 @@ int ipath_multicast_detach(struct ib_qp 
 		atomic_dec(&mcast->refcount);
 		wait_event(mcast->wait, !atomic_read(&mcast->refcount));
 		ipath_mcast_free(mcast);
+		dev->n_mcast_grps_allocated--;
 	}
 
 	ret = 0;
#8 lore raw

PATCH 1/1 ipath - fix accounting of data packets with bad VLs

In reply to Bryan O'Sullivan

For better IB conformance.

drivers/infiniband/hw/ipath/ipath_layer.c +0 -2
diff -r 8e2d63833cf2 -r 551966b88d7c drivers/infiniband/hw/ipath/ipath_layer.c
@@ -1019,13 +1019,11 @@ int ipath_layer_get_counters(struct ipat
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_rxdroppktcnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_rcvovflcnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_portovflcnt) +
-		ipath_snap_cntr(dd, dd->ipath_cregs->cr_errrcvflowctrlcnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_err_rlencnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_invalidrlencnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_erricrccnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_errvcrccnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_errlpcrccnt) +
-		ipath_snap_cntr(dd, dd->ipath_cregs->cr_errlinkcnt) +
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_badformatcnt);
 	cntrs->port_rcv_remphys_errors =
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_rcvebpcnt);
drivers/infiniband/hw/ipath/ipath_mad.c +0 -24
diff -r 8e2d63833cf2 -r 551966b88d7c drivers/infiniband/hw/ipath/ipath_mad.c
@@ -1316,32 +1316,8 @@ int ipath_process_mad(struct ib_device *
 		      struct ib_wc *in_wc, struct ib_grh *in_grh,
 		      struct ib_mad *in_mad, struct ib_mad *out_mad)
 {
-	struct ipath_ibdev *dev = to_idev(ibdev);
 	int ret;
 
-	/*
-	 * Snapshot current HW counters to "clear" them.
-	 * This should be done when the driver is loaded except that for
-	 * some reason we get a zillion errors when brining up the link.
-	 */
-	if (dev->rcv_errors == 0) {
-		struct ipath_layer_counters cntrs;
-
-		ipath_layer_get_counters(to_idev(ibdev)->dd, &cntrs);
-		dev->rcv_errors++;
-		dev->n_symbol_error_counter = cntrs.symbol_error_counter;
-		dev->n_link_error_recovery_counter =
-			cntrs.link_error_recovery_counter;
-		dev->n_link_downed_counter = cntrs.link_downed_counter;
-		dev->n_port_rcv_errors = cntrs.port_rcv_errors + 1;
-		dev->n_port_rcv_remphys_errors =
-			cntrs.port_rcv_remphys_errors;
-		dev->n_port_xmit_discards = cntrs.port_xmit_discards;
-		dev->n_port_xmit_data = cntrs.port_xmit_data;
-		dev->n_port_rcv_data = cntrs.port_rcv_data;
-		dev->n_port_xmit_packets = cntrs.port_xmit_packets;
-		dev->n_port_rcv_packets = cntrs.port_rcv_packets;
-	}
 	switch (in_mad->mad_hdr.mgmt_class) {
 	case IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE:
 	case IB_MGMT_CLASS_SUBN_LID_ROUTED:
drivers/infiniband/hw/ipath/ipath_verbs.c +16 -0
diff -r 8e2d63833cf2 -r 551966b88d7c drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -981,6 +981,7 @@ static int ipath_verbs_register_sysfs(st
  */
 static void *ipath_register_ib_device(int unit, struct ipath_devdata *dd)
 {
+	struct ipath_layer_counters cntrs;
 	struct ipath_ibdev *idev;
 	struct ib_device *dev;
 	int ret;
@@ -1030,6 +1031,21 @@ static void *ipath_register_ib_device(in
 	idev->pma_counter_select[3] = IB_PMA_PORT_RCV_PKTS;
 	idev->pma_counter_select[5] = IB_PMA_PORT_XMIT_WAIT;
 	idev->link_width_enabled = 3;	/* 1x or 4x */
+
+	/* Snapshot current HW counters to "clear" them. */
+	ipath_layer_get_counters(dd, &cntrs);
+	idev->n_symbol_error_counter = cntrs.symbol_error_counter;
+	idev->n_link_error_recovery_counter =
+		cntrs.link_error_recovery_counter;
+	idev->n_link_downed_counter = cntrs.link_downed_counter;
+	idev->n_port_rcv_errors = cntrs.port_rcv_errors;
+	idev->n_port_rcv_remphys_errors =
+		cntrs.port_rcv_remphys_errors;
+	idev->n_port_xmit_discards = cntrs.port_xmit_discards;
+	idev->n_port_xmit_data = cntrs.port_xmit_data;
+	idev->n_port_rcv_data = cntrs.port_rcv_data;
+	idev->n_port_xmit_packets = cntrs.port_xmit_packets;
+	idev->n_port_rcv_packets = cntrs.port_rcv_packets;
 
 	/*
 	 * The system image GUID is supposed to be the same for all
#9 lore raw

PATCH 1/1 ipath - fix NULL dereference during cleanup

In reply to Bryan O'Sullivan

Fix NULL deref due to pcidev being clobbered before dd->ipath_f_cleanup() was called.

drivers/infiniband/hw/ipath/ipath_driver.c +11 -11
diff -r 4868daa7f215 -r b9fd1a46c910 drivers/infiniband/hw/ipath/ipath_driver.c
@@ -1897,19 +1897,19 @@ static void __exit infinipath_cleanup(vo
 			} else
 				ipath_dbg("irq is 0, not doing free_irq "
 					  "for unit %u\n", dd->ipath_unit);
+
+			/*
+			 * we check for NULL here, because it's outside
+			 * the kregbase check, and we need to call it
+			 * after the free_irq.  Thus it's possible that
+			 * the function pointers were never initialized.
+			 */
+			if (dd->ipath_f_cleanup)
+				/* clean up chip-specific stuff */
+				dd->ipath_f_cleanup(dd);
+
 			dd->pcidev = NULL;
 		}
-
-		/*
-		 * we check for NULL here, because it's outside the kregbase
-		 * check, and we need to call it after the free_irq.  Thus
-		 * it's possible that the function pointers were never
-		 * initialized.
-		 */
-		if (dd->ipath_f_cleanup)
-			/* clean up chip-specific stuff */
-			dd->ipath_f_cleanup(dd);
-
 		spin_lock_irqsave(&ipath_devs_lock, flags);
 	}
 
#10 lore raw

PATCH 1/1 ipath - count PE800 receive interrupts on user ports

In reply to Bryan O'Sullivan

Fixed so it works on the PE-800. It had not previously been updated to match PE-800 receive interrupt differences from HT-400.

drivers/infiniband/hw/ipath/ipath_file_ops.c +4 -0
diff -r e9306861dc6a -r 5b565c24d62a drivers/infiniband/hw/ipath/ipath_file_ops.c
@@ -1172,6 +1172,10 @@ static unsigned int ipath_poll(struct fi
 
 	if (tail == head) {
 		set_bit(IPATH_PORT_WAITING_RCV, &pd->port_flag);
+		if(dd->ipath_rhdrhead_intr_off) /* arm rcv interrupt */
+			(void)ipath_write_ureg(dd, ur_rcvhdrhead,
+					       dd->ipath_rhdrhead_intr_off
+					       | head, pd->port_port);
 		poll_wait(fp, &pd->port_wait, pt);
 
 		if (test_bit(IPATH_PORT_WAITING_RCV, &pd->port_flag)) {
#11 lore raw

PATCH 1/1 ipath - clean up some comments

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_kernel.h +2 -7
diff -r b9fd1a46c910 -r 5ddaf7c07cdf drivers/infiniband/hw/ipath/ipath_kernel.h
@@ -720,13 +720,8 @@ u64 ipath_read_kreg64_port(const struct 
  * @port: port number
  *
  * Return the contents of a register that is virtualized to be per port.
- * Prints a debug message and returns -1 on errors (not distinguishable from
- * valid contents at runtime; we may add a separate error variable at some
- * point).
- *
- * This is normally not used by the kernel, but may be for debugging, and
- * has a different implementation than user mode, which is why it's not in
- * _common.h.
+ * Returns -1 on errors (not distinguishable from valid contents at
+ * runtime; we may add a separate error variable at some point).
  */
 static inline u32 ipath_read_ureg32(const struct ipath_devdata *dd,
 				    ipath_ureg regno, int port)
#12 lore raw

PATCH 1/1 ipath - fix memory leak when create of QP fails

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_init_chip.c +1 -0
diff -r 28d938eb0463 -r b41e576e5202 drivers/infiniband/hw/ipath/ipath_init_chip.c
@@ -114,6 +114,7 @@ static int create_port0_egr(struct ipath
 				      "eager TID %u\n", e);
 			while (e != 0)
 				dev_kfree_skb(skbs[--e]);
+			vfree(skbs);
 			ret = -ENOMEM;
 			goto bail;
 		}
#13 lore raw

PATCH 1/1 ipath - allow diags on any unit

In reply to Bryan O'Sullivan

Previously, we hardwired all diags to unit 0.

drivers/infiniband/hw/ipath/ipath_diag.c +40 -68
diff -r 7634b2f0fc40 -r 28d938eb0463 drivers/infiniband/hw/ipath/ipath_diag.c
@@ -66,18 +66,20 @@ static struct file_operations diag_file_
 	.release = ipath_diag_release
 };
 
-static struct cdev *diag_cdev;
-static struct class_device *diag_class_dev;
-
-int ipath_diag_init(void)
-{
-	return ipath_cdev_init(IPATH_DIAG_MINOR, "ipath_diag",
-			       &diag_file_ops, &diag_cdev, &diag_class_dev);
-}
-
-void ipath_diag_cleanup(void)
-{
-	ipath_cdev_cleanup(&diag_cdev, &diag_class_dev);
+int ipath_diag_add(struct ipath_devdata *dd)
+{
+	char name[16];
+
+	snprintf(name, sizeof(name), "ipath_diag%d", dd->ipath_unit);
+
+	return ipath_cdev_init(IPATH_DIAG_MINOR_BASE + dd->ipath_unit, name,
+			       &diag_file_ops, &dd->diag_cdev,
+			       &dd->diag_class_dev);
+}
+
+void ipath_diag_remove(struct ipath_devdata *dd)
+{
+	ipath_cdev_cleanup(&dd->diag_cdev, &dd->diag_class_dev);
 }
 
 /**
@@ -101,8 +103,7 @@ static int ipath_read_umem64(struct ipat
 	int ret;
 
 	/* not very efficient, but it works for now */
-	if (reg_addr < dd->ipath_kregbase ||
-	    reg_end > dd->ipath_kregend) {
+	if (reg_addr < dd->ipath_kregbase || reg_end > dd->ipath_kregend) {
 		ret = -EINVAL;
 		goto bail;
 	}
@@ -139,8 +140,7 @@ static int ipath_write_umem64(struct ipa
 	int ret;
 
 	/* not very efficient, but it works for now */
-	if (reg_addr < dd->ipath_kregbase ||
-	    reg_end > dd->ipath_kregend) {
+	if (reg_addr < dd->ipath_kregbase || reg_end > dd->ipath_kregend) {
 		ret = -EINVAL;
 		goto bail;
 	}
@@ -240,59 +240,45 @@ bail:
 
 static int ipath_diag_open(struct inode *in, struct file *fp)
 {
+	int unit = iminor(in) - IPATH_DIAG_MINOR_BASE;
 	struct ipath_devdata *dd;
-	int unit = 0; /* XXX this is bogus */
-	unsigned long flags;
-	int ret;
-
-	dd = ipath_lookup(unit);
+	int ret;
 
 	mutex_lock(&ipath_mutex);
-	spin_lock_irqsave(&ipath_devs_lock, flags);
 
 	if (ipath_diag_inuse) {
 		ret = -EBUSY;
 		goto bail;
 	}
 
-	list_for_each_entry(dd, &ipath_dev_list, ipath_list) {
-		/*
-		 * we need at least one infinipath device to be present
-		 * (don't use INITTED, because we want to be able to open
-		 * even if device is in freeze mode, which cleared INITTED).
-		 * There is a small amount of risk to this, which is why we
-		 * also verify kregbase is set.
-		 */
-
-		if (!(dd->ipath_flags & IPATH_PRESENT) ||
-		    !dd->ipath_kregbase)
-			continue;
-
-		ipath_diag_inuse = 1;
-		diag_set_link = 0;
-		ret = 0;
-		goto bail;
-	}
-
-	ret = -ENODEV;
-
-bail:
-	spin_unlock_irqrestore(&ipath_devs_lock, flags);
+	dd = ipath_lookup(unit);
+
+	if (dd == NULL || !(dd->ipath_flags & IPATH_PRESENT) ||
+	    !dd->ipath_kregbase) {
+		ret = -ENODEV;
+		goto bail;
+	}
+
+	fp->private_data = dd;
+	ipath_diag_inuse = 1;
+	diag_set_link = 0;
+	ret = 0;
 
 	/* Only expose a way to reset the device if we
 	   make it into diag mode. */
-	if (ret == 0)
-		ipath_expose_reset(&dd->pcidev->dev);
-
+	ipath_expose_reset(&dd->pcidev->dev);
+
+bail:
 	mutex_unlock(&ipath_mutex);
 
 	return ret;
 }
 
-static int ipath_diag_release(struct inode *i, struct file *f)
+static int ipath_diag_release(struct inode *in, struct file *fp)
 {
 	mutex_lock(&ipath_mutex);
 	ipath_diag_inuse = 0;
+	fp->private_data = NULL;
 	mutex_unlock(&ipath_mutex);
 	return 0;
 }
@@ -300,16 +286,9 @@ static ssize_t ipath_diag_read(struct fi
 static ssize_t ipath_diag_read(struct file *fp, char __user *data,
 			       size_t count, loff_t *off)
 {
-	int unit = 0; /* XXX provide for reads on other units some day */
-	struct ipath_devdata *dd;
+	struct ipath_devdata *dd = fp->private_data;
 	void __iomem *kreg_base;
 	ssize_t ret;
-
-	dd = ipath_lookup(unit);
-	if (!dd) {
-		ret = -ENODEV;
-		goto bail;
-	}
 
 	kreg_base = dd->ipath_kregbase;
 
@@ -329,23 +308,16 @@ static ssize_t ipath_diag_read(struct fi
 		ret = count;
 	}
 
-bail:
 	return ret;
 }
 
 static ssize_t ipath_diag_write(struct file *fp, const char __user *data,
 				size_t count, loff_t *off)
 {
-	int unit = 0; /* XXX this is bogus */
-	struct ipath_devdata *dd;
+	struct ipath_devdata *dd = fp->private_data;
 	void __iomem *kreg_base;
 	ssize_t ret;
 
-	dd = ipath_lookup(unit);
-	if (!dd) {
-		ret = -ENODEV;
-		goto bail;
-	}
 	kreg_base = dd->ipath_kregbase;
 
 	if (count == 0)
@@ -364,6 +336,6 @@ static ssize_t ipath_diag_write(struct f
 		ret = count;
 	}
 
-bail:
-	return ret;
-}
+	return ret;
+}
+
drivers/infiniband/hw/ipath/ipath_driver.c +4 -2
diff -r 7634b2f0fc40 -r 28d938eb0463 drivers/infiniband/hw/ipath/ipath_driver.c
@@ -488,6 +488,7 @@ static int __devinit ipath_init_one(stru
 	ipath_device_create_group(&pdev->dev, dd);
 	ipathfs_add_device(dd);
 	ipath_user_add(dd);
+	ipath_diag_add(dd);
 	ipath_layer_add(dd);
 
 	goto bail;
@@ -517,8 +518,9 @@ static void __devexit ipath_remove_one(s
 		return;
 
 	dd = pci_get_drvdata(pdev);
-	ipath_layer_del(dd);
-	ipath_user_del(dd);
+	ipath_layer_remove(dd);
+	ipath_diag_remove(dd);
+	ipath_user_remove(dd);
 	ipathfs_remove_device(dd);
 	ipath_device_remove_group(&pdev->dev, dd);
 	ipath_cdbg(VERBOSE, "Releasing pci memory regions, dd %p, "
drivers/infiniband/hw/ipath/ipath_file_ops.c +11 -20
diff -r 7634b2f0fc40 -r 28d938eb0463 drivers/infiniband/hw/ipath/ipath_file_ops.c
@@ -1390,16 +1390,16 @@ done:
 
 static int ipath_open(struct inode *in, struct file *fp)
 {
-	int ret, minor;
+	int ret, user_minor;
 
 	mutex_lock(&ipath_mutex);
 
-	minor = iminor(in);
+	user_minor = iminor(in) - IPATH_USER_MINOR_BASE;
 	ipath_cdbg(VERBOSE, "open on dev %lx (minor %d)\n",
-		   (long)in->i_rdev, minor);
-
-	if (minor)
-		ret = find_free_port(minor - 1, fp);
+		   (long)in->i_rdev, user_minor);
+
+	if (user_minor)
+		ret = find_free_port(user_minor - 1, fp);
 	else
 		ret = find_best_unit(fp);
 
@@ -1799,19 +1799,13 @@ int ipath_user_add(struct ipath_devdata 
 				      "error %d\n", -ret);
 			goto bail;
 		}
-		ret = ipath_diag_init();
-		if (ret < 0) {
-			ipath_dev_err(dd, "Unable to set up diag support: "
-				      "error %d\n", -ret);
-			goto bail_sma;
-		}
 
 		ret = init_cdev(0, "ipath", &ipath_file_ops, &wildcard_cdev,
 				&wildcard_class_dev);
 		if (ret < 0) {
 			ipath_dev_err(dd, "Could not create wildcard "
 				      "minor: error %d\n", -ret);
-			goto bail_diag;
+			goto bail_sma;
 		}
 
 		atomic_set(&user_setup, 1);
@@ -1820,31 +1814,28 @@ int ipath_user_add(struct ipath_devdata 
 	snprintf(name, sizeof(name), "ipath%d", dd->ipath_unit);
 
 	ret = init_cdev(dd->ipath_unit + 1, name, &ipath_file_ops,
-			&dd->cdev, &dd->class_dev);
+			&dd->user_cdev, &dd->user_class_dev);
 	if (ret < 0)
 		ipath_dev_err(dd, "Could not create user minor %d, %s\n",
 			      dd->ipath_unit + 1, name);
 
 	goto bail;
 
-bail_diag:
-	ipath_diag_cleanup();
 bail_sma:
 	user_cleanup();
 bail:
 	return ret;
 }
 
-void ipath_user_del(struct ipath_devdata *dd)
-{
-	cleanup_cdev(&dd->cdev, &dd->class_dev);
+void ipath_user_remove(struct ipath_devdata *dd)
+{
+	cleanup_cdev(&dd->user_cdev, &dd->user_class_dev);
 
 	if (atomic_dec_return(&user_count) == 0) {
 		if (atomic_read(&user_setup) == 0)
 			goto bail;
 
 		cleanup_cdev(&wildcard_cdev, &wildcard_class_dev);
-		ipath_diag_cleanup();
 		user_cleanup();
 
 		atomic_set(&user_setup, 0);
drivers/infiniband/hw/ipath/ipath_kernel.h +11 -8
diff -r 7634b2f0fc40 -r 28d938eb0463 drivers/infiniband/hw/ipath/ipath_kernel.h
@@ -347,8 +347,10 @@ struct ipath_devdata {
 	char *ipath_freezemsg;
 	/* pci access data structure */
 	struct pci_dev *pcidev;
-	struct cdev *cdev;
-	struct class_device *class_dev;
+	struct cdev *user_cdev;
+	struct cdev *diag_cdev;
+	struct class_device *user_class_dev;
+	struct class_device *diag_class_dev;
 	/* timer used to prevent stats overflow, error throttling, etc. */
 	struct timer_list ipath_stats_timer;
 	/* check for stale messages in rcv queue */
@@ -531,7 +533,7 @@ extern int __ipath_verbs_rcv(struct ipat
 extern int __ipath_verbs_rcv(struct ipath_devdata *, void *, void *, u32);
 
 void ipath_layer_add(struct ipath_devdata *);
-void ipath_layer_del(struct ipath_devdata *);
+void ipath_layer_remove(struct ipath_devdata *);
 
 int ipath_init_chip(struct ipath_devdata *, int);
 int ipath_enable_wc(struct ipath_devdata *dd);
@@ -545,14 +547,14 @@ void ipath_cdev_cleanup(struct cdev **cd
 void ipath_cdev_cleanup(struct cdev **cdevp,
 			struct class_device **class_devp);
 
-int ipath_diag_init(void);
-void ipath_diag_cleanup(void);
+int ipath_diag_add(struct ipath_devdata *);
+void ipath_diag_remove(struct ipath_devdata *);
 void ipath_diag_bringup_link(struct ipath_devdata *);
 
 extern wait_queue_head_t ipath_sma_state_wait;
 
 int ipath_user_add(struct ipath_devdata *dd);
-void ipath_user_del(struct ipath_devdata *dd);
+void ipath_user_remove(struct ipath_devdata *dd);
 
 struct sk_buff *ipath_alloc_skb(struct ipath_devdata *dd, gfp_t);
 
@@ -831,9 +833,10 @@ extern struct mutex ipath_mutex;
 
 #define IPATH_DRV_NAME		"ipath_core"
 #define IPATH_MAJOR		233
+#define IPATH_USER_MINOR_BASE	0
 #define IPATH_SMA_MINOR		128
-#define IPATH_DIAG_MINOR	129
-#define IPATH_NMINORS		130
+#define IPATH_DIAG_MINOR_BASE	129
+#define IPATH_NMINORS		255
 
 #define ipath_dev_err(dd,fmt,...) \
 	do { \
drivers/infiniband/hw/ipath/ipath_layer.c +1 -1
diff -r 7634b2f0fc40 -r 28d938eb0463 drivers/infiniband/hw/ipath/ipath_layer.c
@@ -402,7 +402,7 @@ void ipath_layer_add(struct ipath_devdat
 	mutex_unlock(&ipath_layer_mutex);
 }
 
-void ipath_layer_del(struct ipath_devdata *dd)
+void ipath_layer_remove(struct ipath_devdata *dd)
 {
 	mutex_lock(&ipath_layer_mutex);
 
#14 lore raw

PATCH 1/1 ipath - QP should ignore receive queue size if SRQ specified

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_qp.c +16 -11
diff -r a1615956e57f -r 49b446b12f16 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -684,16 +684,22 @@ struct ib_qp *ipath_create_qp(struct ib_
 			ret = ERR_PTR(-ENOMEM);
 			goto bail;
 		}
-		qp->r_rq.size = init_attr->cap.max_recv_wr + 1;
-		sz = sizeof(struct ipath_sge) *
-			init_attr->cap.max_recv_sge +
-			sizeof(struct ipath_rwqe);
-		qp->r_rq.wq = vmalloc(qp->r_rq.size * sz);
-		if (!qp->r_rq.wq) {
-			kfree(qp);
-			vfree(swq);
-			ret = ERR_PTR(-ENOMEM);
-			goto bail;
+		if (init_attr->srq) {
+			qp->r_rq.size = 0;
+			qp->r_rq.max_sge = 0;
+			qp->r_rq.wq = NULL;
+		} else {
+			qp->r_rq.size = init_attr->cap.max_recv_wr + 1;
+			qp->r_rq.max_sge = init_attr->cap.max_recv_sge;
+			sz = (sizeof(struct ipath_sge) * qp->r_rq.max_sge) +
+				sizeof(struct ipath_rwqe);
+			qp->r_rq.wq = vmalloc(qp->r_rq.size * sz);
+			if (!qp->r_rq.wq) {
+				kfree(qp);
+				vfree(swq);
+				ret = ERR_PTR(-ENOMEM);
+				goto bail;
+			}
 		}
 
 		/*
@@ -712,7 +718,6 @@ struct ib_qp *ipath_create_qp(struct ib_
 		qp->s_wq = swq;
 		qp->s_size = init_attr->cap.max_send_wr + 1;
 		qp->s_max_sge = init_attr->cap.max_send_sge;
-		qp->r_rq.max_sge = init_attr->cap.max_recv_sge;
 		qp->s_flags = init_attr->sq_sig_type == IB_SIGNAL_REQ_WR ?
 			1 << IPATH_S_SIGNAL_REQ_WR : 0;
 		dev = to_idev(ibpd->device);
#15 lore raw

PATCH 1/1 ipath - reduce maximum table sizes

In reply to Bryan O'Sullivan

Decrease the number of WRs and SGEs we support from 131071/255 to 16383/60. This decreases our maximum memory usage per QP from ~1800MB down to about 40MB. This is still a lot, but it's better than 2GB.

drivers/infiniband/hw/ipath/ipath_verbs.c +2 -2
diff -r 40532fdc53f0 -r bd1de2e983db drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -73,12 +73,12 @@ module_param_named(max_cqs, ib_ipath_max
 module_param_named(max_cqs, ib_ipath_max_cqs, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_cqs, "Maximum number of completion queues to support");
 
-unsigned int ib_ipath_max_qp_wrs = 0x1FFFF;
+unsigned int ib_ipath_max_qp_wrs = 0x3FFF;
 module_param_named(max_qp_wrs, ib_ipath_max_qp_wrs, uint,
 		   S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_qp_wrs, "Maximum number of QP WRs to support");
 
-unsigned int ib_ipath_max_sges = 0xFF;
+unsigned int ib_ipath_max_sges = 0x60;
 module_param_named(max_sges, ib_ipath_max_sges, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_sges, "Maximum number of SGEs to support");
 
#16 lore raw

Re: [PATCH 50 of 53] ipath - reduce maximum table sizes

This is the third patch in the series that changes these -- how about making up your mind ;)

#17 lore raw

PATCH 1/1 ipath - add memory barrier when waiting for writes

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_eeprom.c +1 -0
diff -r fd9bdeea5b10 -r f8ebb8c1e436 drivers/infiniband/hw/ipath/ipath_eeprom.c
@@ -185,6 +185,7 @@ bail:
  */
 static void i2c_wait_for_writes(struct ipath_devdata *dd)
 {
+	mb();
 	(void)ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
 }
 
#18 lore raw

Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

drivers/infiniband/hw/ipath/ipath_eeprom.c bail: · line 190

Quoted 5 lines

static void i2c_wait_for_writes(struct ipath_devdata *dd) {

drivers/infiniband/hw/ipath/ipath_eeprom.c +1 -0
+	mb();
 	(void)ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
 }

This needs a comment explaining why it's needed. A memory barrier before a readl() looks very strange since readl() should be ordered anyway.

- R.

#19 lore raw

Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

Quoted 8 lines · Roland Dreier

drivers/infiniband/hw/ipath/ipath_eeprom.c bail: · line 190

Quoted 5 lines

static void i2c_wait_for_writes(struct ipath_devdata *dd) {

drivers/infiniband/hw/ipath/ipath_eeprom.c +1 -0
+	mb();
 	(void)ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
 }

This needs a comment explaining why it's needed. A memory barrier before a readl() looks very strange since readl() should be ordered anyway.

Yeah. It's actually working around what appears to be a gcc bug if the kernel is compiled with -Os. Ralph knows the details; he can give a more complete answer.

	<b
#20 lore raw

Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

Quoted 11 lines · Roland Dreier
Quoted 9 lines

drivers/infiniband/hw/ipath/ipath_eeprom.c bail: · line 190

Quoted 5 lines

static void i2c_wait_for_writes(struct ipath_devdata *dd) {

drivers/infiniband/hw/ipath/ipath_eeprom.c +1 -0
+	mb();
 	(void)ipath_read_kreg32(dd, dd->ipath_kregs->kr_scratch);
 }

This needs a comment explaining why it's needed. A memory barrier before a readl() looks very strange since readl() should be ordered anyway.

Quoted 5 lines

Yeah. It's actually working around what appears to be a gcc bug if the kernel is compiled with -Os. Ralph knows the details; he can give a more complete answer.

	<b

I don't have a lot to add to this other than I looked at the assembly code output for -Os and -O3 and both looked OK. I put the mb() in to be sure the writes were complete and I found this to work by experimentation.
Without it, the driver fails to read the EEPROM correctly.

#21 lore raw

Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

ralphc> I don't have a lot to add to this other than I looked at ralphc> the assembly code output for -Os and -O3 and both looked ralphc> OK. I put the mb() in to be sure the writes were complete ralphc> and I found this to work by experimentation. Without it, ralphc> the driver fails to read the EEPROM correctly.

Hmm, that doesn't give me a warm fuzzy feeling. Basically on x86-64 you're adding an unneeded mfence instruction to work around miscompilation?

Is i2c_wait_for_writes miscompiled without the mb() with -Os? What does the bad assembly look like?

- R.

#22 lore raw

Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

Quoted 6 lines

ralphc> I don't have a lot to add to this other than I looked at ralphc> the assembly code output for -Os and -O3 and both looked ralphc> OK. I put the mb() in to be sure the writes were complete ralphc> and I found this to work by experimentation. Without it, ralphc> the driver fails to read the EEPROM correctly.

Quoted 8 lines

Hmm, that doesn't give me a warm fuzzy feeling. Basically on x86-64 you're adding an unneeded mfence instruction to work around miscompilation?

Is i2c_wait_for_writes miscompiled without the mb() with -Os? What does the bad assembly look like?

- R.

We had a power failure here so I'm not able to reproduce the assembly code at the moment. What I remember from looking at the code is that the code for ipath_read_kreg32() was present in i2c_wait_for_writes() when compiled -Os so it didn't look like a compiler bug. I probably could put the mb() at the end of i2c_gpio_set() if that makes you more comfortable. The mb() is definitely needed though.

#23 lore raw

Re: [openib-general] Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

ralphc> We had a power failure here so I'm not able to reproduce ralphc> the assembly code at the moment. What I remember from ralphc> looking at the code is that the code for ralphc> ipath_read_kreg32() was present in i2c_wait_for_writes() ralphc> when compiled -Os so it didn't look like a compiler bug. ralphc> I probably could put the mb() at the end of i2c_gpio_set() ralphc> if that makes you more comfortable. The mb() is ralphc> definitely needed though.

Is it the mb()? Or is just a barrier() enough? In other words do you really need the mfence, or do you just need to stop the compiler from reordering things?

- R.

#24 lore raw

Re: [openib-general] Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

Quoted 9 lines

ralphc> We had a power failure here so I'm not able to reproduce ralphc> the assembly code at the moment. What I remember from ralphc> looking at the code is that the code for ralphc> ipath_read_kreg32() was present in i2c_wait_for_writes() ralphc> when compiled -Os so it didn't look like a compiler bug. ralphc> I probably could put the mb() at the end of i2c_gpio_set() ralphc> if that makes you more comfortable. The mb() is ralphc> definitely needed though.

Quoted 5 lines

Is it the mb()? Or is just a barrier() enough? In other words do you really need the mfence, or do you just need to stop the compiler from reordering things?

- R.

I didn't try calling barrier() so I don't know the answer. When power is restored, I can try it.
My guess is that it's a timing issue and not a code reordering issue.

#25 lore raw

Re: [openib-general] Re: [PATCH 53 of 53] ipath - add memory barrier when waiting for writes

ralphc> I didn't try calling barrier() so I don't know the answer. ralphc> When power is restored, I can try it. My guess is that ralphc> it's a timing issue and not a code reordering issue.

Hmm, then we really better understand what's going on, because otherwise you're just going to have trouble again if someone makes a CPU with a faster mfence instruction...

#26 lore raw

PATCH 1/1 ipath - purge sps_lid and sps_mlid arrays, and /sys entries

In reply to Bryan O'Sullivan

The two arrays only had space for 4 units, so didn't work for larger numbers of units. I thought I'd eliminated these before submitting the original driver patches.

Also fixed error return on ipath_sysfs_unit_write to not set an error code if the sysfs code reports consuming more chars than we wrote (since that can include the nul, and the user doesn't have to include the nul in the write).

Also changed from ipath_set_sps_lid() to ipath_set_lid(); the sps was a leftover piece of naming.

drivers/infiniband/hw/ipath/ipath_common.h +2 -5
diff -r 9b9f24aab350 -r 3ab7a7b10bf2 drivers/infiniband/hw/ipath/ipath_common.h
@@ -121,8 +121,7 @@ struct infinipath_stats {
 	__u64 sps_ports;
 	/* list of pkeys (other than default) accepted (0 means not set) */
 	__u16 sps_pkeys[4];
-	/* lids for up to 4 infinipaths, indexed by infinipath # */
-	__u16 sps_lid[4];
+	__u16 sps_unused16[4]; /* available; maintaining compatible layout */
 	/* number of user ports per chip (not IB ports) */
 	__u32 sps_nports;
 	/* not our interrupt, or already handled */
@@ -140,10 +139,8 @@ struct infinipath_stats {
 	 * packets if ipath not configured, sma/mad, etc.)
 	 */
 	__u64 sps_krdrops;
-	/* mlids for up to 4 infinipaths, indexed by infinipath # */
-	__u16 sps_mlid[4];
 	/* pad for future growth */
-	__u64 __sps_pad[45];
+	__u64 __sps_pad[46];
 };
 
 /*
drivers/infiniband/hw/ipath/ipath_init_chip.c +0 -2
diff -r 9b9f24aab350 -r 3ab7a7b10bf2 drivers/infiniband/hw/ipath/ipath_init_chip.c
@@ -836,8 +836,6 @@ int ipath_init_chip(struct ipath_devdata
 	/* clear any interrups up to this point (ints still not enabled) */
 	ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, -1LL);
 
-	ipath_stats.sps_lid[dd->ipath_unit] = dd->ipath_lid;
-
 	/*
 	 * Set up the port 0 (kernel) rcvhdr q and egr TIDs.  If doing
 	 * re-init, the simplest way to handle this is to free
drivers/infiniband/hw/ipath/ipath_layer.c +5 -6
diff -r 9b9f24aab350 -r 3ab7a7b10bf2 drivers/infiniband/hw/ipath/ipath_layer.c
@@ -299,9 +299,8 @@ bail:
 
 EXPORT_SYMBOL_GPL(ipath_layer_set_mtu);
 
-int ipath_set_sps_lid(struct ipath_devdata *dd, u32 arg, u8 lmc)
-{
-	ipath_stats.sps_lid[dd->ipath_unit] = arg;
+int ipath_set_lid(struct ipath_devdata *dd, u32 arg, u8 lmc)
+{
 	dd->ipath_lid = arg;
 	dd->ipath_lmc = lmc;
 
@@ -315,7 +314,7 @@ int ipath_set_sps_lid(struct ipath_devda
 	return 0;
 }
 
-EXPORT_SYMBOL_GPL(ipath_set_sps_lid);
+EXPORT_SYMBOL_GPL(ipath_set_lid);
 
 int ipath_layer_set_guid(struct ipath_devdata *dd, __be64 guid)
 {
@@ -616,9 +615,9 @@ int ipath_layer_open(struct ipath_devdat
 
 	if (*dd->ipath_statusp & IPATH_STATUS_IB_READY)
 		intval |= IPATH_LAYER_INT_IF_UP;
-	if (ipath_stats.sps_lid[dd->ipath_unit])
+	if (dd->ipath_lid)
 		intval |= IPATH_LAYER_INT_LID;
-	if (ipath_stats.sps_mlid[dd->ipath_unit])
+	if (dd->ipath_mlid)
 		intval |= IPATH_LAYER_INT_BCAST;
 	/*
 	 * do this on open, in case low level is already up and
drivers/infiniband/hw/ipath/ipath_layer.h +1 -1
diff -r 9b9f24aab350 -r 3ab7a7b10bf2 drivers/infiniband/hw/ipath/ipath_layer.h
@@ -126,7 +126,7 @@ u32 ipath_layer_get_cr_errpkey(struct ip
 u32 ipath_layer_get_cr_errpkey(struct ipath_devdata *dd);
 int ipath_layer_set_linkstate(struct ipath_devdata *dd, u8 state);
 int ipath_layer_set_mtu(struct ipath_devdata *, u16);
-int ipath_set_sps_lid(struct ipath_devdata *, u32, u8);
+int ipath_set_lid(struct ipath_devdata *, u32, u8);
 int ipath_layer_send_hdr(struct ipath_devdata *dd,
 			 struct ether_header *hdr);
 int ipath_verbs_send(struct ipath_devdata *dd, u32 hdrwords,
drivers/infiniband/hw/ipath/ipath_mad.c +1 -1
diff -r 9b9f24aab350 -r 3ab7a7b10bf2 drivers/infiniband/hw/ipath/ipath_mad.c
@@ -341,7 +341,7 @@ static int recv_subn_set_portinfo(struct
 		/* Must be a valid unicast LID address. */
 		if (lid == 0 || lid >= IPS_MULTICAST_LID_BASE)
 			goto err;
-		ipath_set_sps_lid(dev->dd, lid, pip->mkeyprot_resv_lmc & 7);
+		ipath_set_lid(dev->dd, lid, pip->mkeyprot_resv_lmc & 7);
 		event.event = IB_EVENT_LID_CHANGE;
 		ib_dispatch_event(&event);
 	}
drivers/infiniband/hw/ipath/ipath_sysfs.c +3 -101

143 lines not on this page · raw message

#27 lore raw

PATCH 1/1 ipath - fix reporting of vendor ID and a few other trivial bits

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_layer.c +27 -12
diff -r bd1de2e983db -r 5f665c503f0d drivers/infiniband/hw/ipath/ipath_layer.c
@@ -339,18 +339,26 @@ u32 ipath_layer_get_nguid(struct ipath_d
 
 EXPORT_SYMBOL_GPL(ipath_layer_get_nguid);
 
-int ipath_layer_query_device(struct ipath_devdata *dd, u32 * vendor,
-			     u32 * boardrev, u32 * majrev, u32 * minrev)
-{
-	*vendor = dd->ipath_vendorid;
-	*boardrev = dd->ipath_boardrev;
-	*majrev = dd->ipath_majrev;
-	*minrev = dd->ipath_minrev;
-
-	return 0;
-}
-
-EXPORT_SYMBOL_GPL(ipath_layer_query_device);
+u32 ipath_layer_get_majrev(struct ipath_devdata *dd)
+{
+	return dd->ipath_majrev;
+}
+
+EXPORT_SYMBOL_GPL(ipath_layer_get_majrev);
+
+u32 ipath_layer_get_minrev(struct ipath_devdata *dd)
+{
+	return dd->ipath_minrev;
+}
+
+EXPORT_SYMBOL_GPL(ipath_layer_get_minrev);
+
+u32 ipath_layer_get_pcirev(struct ipath_devdata *dd)
+{
+	return dd->ipath_pcirev;
+}
+
+EXPORT_SYMBOL_GPL(ipath_layer_get_pcirev);
 
 u32 ipath_layer_get_flags(struct ipath_devdata *dd)
 {
@@ -372,6 +380,13 @@ u16 ipath_layer_get_deviceid(struct ipat
 }
 
 EXPORT_SYMBOL_GPL(ipath_layer_get_deviceid);
+
+u32 ipath_layer_get_vendorid(struct ipath_devdata *dd)
+{
+	return dd->ipath_vendorid;
+}
+
+EXPORT_SYMBOL_GPL(ipath_layer_get_vendorid);
 
 u64 ipath_layer_get_lastibcstat(struct ipath_devdata *dd)
 {
drivers/infiniband/hw/ipath/ipath_layer.h +4 -2
diff -r bd1de2e983db -r 5f665c503f0d drivers/infiniband/hw/ipath/ipath_layer.h
@@ -145,11 +145,13 @@ int ipath_layer_set_guid(struct ipath_de
 int ipath_layer_set_guid(struct ipath_devdata *, __be64 guid);
 __be64 ipath_layer_get_guid(struct ipath_devdata *);
 u32 ipath_layer_get_nguid(struct ipath_devdata *);
-int ipath_layer_query_device(struct ipath_devdata *, u32 * vendor,
-			     u32 * boardrev, u32 * majrev, u32 * minrev);
+u32 ipath_layer_get_majrev(struct ipath_devdata *);
+u32 ipath_layer_get_minrev(struct ipath_devdata *);
+u32 ipath_layer_get_pcirev(struct ipath_devdata *);
 u32 ipath_layer_get_flags(struct ipath_devdata *dd);
 struct device *ipath_layer_get_device(struct ipath_devdata *dd);
 u16 ipath_layer_get_deviceid(struct ipath_devdata *dd);
+u32 ipath_layer_get_vendorid(struct ipath_devdata *);
 u64 ipath_layer_get_lastibcstat(struct ipath_devdata *dd);
 u32 ipath_layer_get_ibmtu(struct ipath_devdata *dd);
 int ipath_layer_enable_timer(struct ipath_devdata *dd);
drivers/infiniband/hw/ipath/ipath_mad.c +4 -2
diff -r bd1de2e983db -r 5f665c503f0d drivers/infiniband/hw/ipath/ipath_mad.c
@@ -84,7 +84,7 @@ static int recv_subn_get_nodeinfo(struct
 {
 	struct nodeinfo *nip = (struct nodeinfo *)&smp->data;
 	struct ipath_devdata *dd = to_idev(ibdev)->dd;
-	u32 vendor, boardid, majrev, minrev;
+	u32 vendor, majrev, minrev;
 
 	if (smp->attr_mod)
 		smp->status |= IB_SMP_INVALID_FIELD;
@@ -104,9 +104,11 @@ static int recv_subn_get_nodeinfo(struct
 	nip->port_guid = nip->sys_guid;
 	nip->partition_cap = cpu_to_be16(ipath_layer_get_npkeys(dd));
 	nip->device_id = cpu_to_be16(ipath_layer_get_deviceid(dd));
-	ipath_layer_query_device(dd, &vendor, &boardid, &majrev, &minrev);
+	majrev = ipath_layer_get_majrev(dd);
+	minrev = ipath_layer_get_minrev(dd);
 	nip->revision = cpu_to_be32((majrev << 16) | minrev);
 	nip->local_port_num = port;
+	vendor = ipath_layer_get_vendorid(dd);
 	nip->vendor_id[0] = 0;
 	nip->vendor_id[1] = vendor >> 8;
 	nip->vendor_id[2] = vendor;
drivers/infiniband/hw/ipath/ipath_verbs.c +7 -13
diff -r bd1de2e983db -r 5f665c503f0d drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -620,18 +620,15 @@ static int ipath_query_device(struct ib_
 			      struct ib_device_attr *props)
 {
 	struct ipath_ibdev *dev = to_idev(ibdev);
-	u32 vendor, boardrev, majrev, minrev;
 
 	memset(props, 0, sizeof(*props));
 
 	props->device_cap_flags = IB_DEVICE_BAD_PKEY_CNTR |
 		IB_DEVICE_BAD_QKEY_CNTR | IB_DEVICE_SHUTDOWN_PORT |
 		IB_DEVICE_SYS_IMAGE_GUID;
-	ipath_layer_query_device(dev->dd, &vendor, &boardrev,
-				 &majrev, &minrev);
-	props->vendor_id = vendor;
-	props->vendor_part_id = boardrev;
-	props->hw_ver = boardrev << 16 | majrev << 8 | minrev;
+	props->vendor_id = ipath_layer_get_vendorid(dev->dd);
+	props->vendor_part_id = ipath_layer_get_deviceid(dev->dd);
+	props->hw_ver = ipath_layer_get_pcirev(dev->dd);
 
 	props->sys_image_guid = dev->sys_image_guid;
 
@@ -1220,11 +1217,8 @@ static ssize_t show_rev(struct class_dev
 {
 	struct ipath_ibdev *dev =
 		container_of(cdev, struct ipath_ibdev, ibdev.class_dev);
-	int vendor, boardrev, majrev, minrev;
-
-	ipath_layer_query_device(dev->dd, &vendor, &boardrev,
-				 &majrev, &minrev);
-	return sprintf(buf, "%d.%d\n", majrev, minrev);
+
+	return sprintf(buf, "%x\n", ipath_layer_get_pcirev(dev->dd));
 }
 
 static ssize_t show_hca(struct class_device *cdev, char *buf)
@@ -1253,7 +1247,7 @@ static ssize_t show_stats(struct class_d
 	len = sprintf(buf,
 		      "RC resends  %d\n"
 		      "RC no QACK  %d\n"
-		      "RC ACKs     %d\n"
+		      "RC ACKs	   %d\n"
 		      "RC SEQ NAKs %d\n"
 		      "RC RDMA seq %d\n"
 		      "RC RNR NAKs %d\n"
@@ -1263,7 +1257,7 @@ static ssize_t show_stats(struct class_d
 		      "piobuf wait %d\n"
 		      "no piobuf   %d\n"
 		      "PKT drops   %d\n"
-		      "WQE errs    %d\n",
+		      "WQE errs	   %d\n",
 		      dev->n_rc_resends, dev->n_rc_qacks, dev->n_rc_acks,
 		      dev->n_seq_naks, dev->n_rdma_seq, dev->n_rnr_naks,
 		      dev->n_other_naks, dev->n_timeouts,
#28 lore raw

PATCH 1/1 ipath - forbid creation of AHs with illegal ports

In reply to Bryan O'Sullivan

Don't allow an AH to be created with an illegal port.

drivers/infiniband/hw/ipath/ipath_verbs.c +6 -0
diff -r 300f0aa6f034 -r db56c0ab6a64 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -810,6 +810,12 @@ static struct ib_ah *ipath_create_ah(str
 	if (ah_attr->dlid >= IPS_MULTICAST_LID_BASE &&
 	    ah_attr->dlid != IPS_PERMISSIVE_LID &&
 	    !(ah_attr->ah_flags & IB_AH_GRH)) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
+	if (ah_attr->port_num != 1 ||
+	    ah_attr->port_num > pd->device->phys_port_cnt) {
 		ret = ERR_PTR(-EINVAL);
 		goto bail;
 	}
#29 lore raw

PATCH 1/1 ipath - register as IB device owner

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -0
diff -r 5f665c503f0d -r fd9bdeea5b10 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -1060,6 +1060,7 @@ static void *ipath_register_ib_device(in
 	idev->dd = dd;
 
 	strlcpy(dev->name, "ipath%d", IB_DEVICE_NAME_MAX);
+	dev->owner = THIS_MODULE;
 	dev->node_guid = ipath_layer_get_guid(dd);
 	dev->uverbs_abi_ver = IPATH_UVERBS_ABI_VERSION;
 	dev->uverbs_cmd_mask =
#30 lore raw

PATCH 1/1 ipath - cap number of CQEs

In reply to Bryan O'Sullivan

Cap the number of CQEs. Not a real limitation for us, but expected by the verbs code.

drivers/infiniband/hw/ipath/ipath_cq.c +5 -0
diff -r e823378bd19c -r 1d3e85454b53 drivers/infiniband/hw/ipath/ipath_cq.c
@@ -161,6 +161,11 @@ struct ib_cq *ipath_create_cq(struct ib_
 	struct ib_wc *wc;
 	struct ib_cq *ret;
 
+	if (entries > ib_ipath_max_cqe) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
 	/*
 	 * Need to use vmalloc() if we want to support large #s of
 	 * entries.
drivers/infiniband/hw/ipath/ipath_verbs.c +6 -1
diff -r e823378bd19c -r 1d3e85454b53 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -64,6 +64,11 @@ module_param_named(max_ahs, ib_ipath_max
 module_param_named(max_ahs, ib_ipath_max_ahs, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_ahs,
 		 "Maximum number of address handles to support");
+
+unsigned int ib_ipath_max_cqe = 0xFFFF;
+module_param_named(max_cqe, ib_ipath_max_cqe, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_cqe,
+		 "Maximum number of completion queue entries to support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -598,7 +603,7 @@ static int ipath_query_device(struct ib_
 	props->max_sge = 255;
 	props->max_cq = 0xffff;
 	props->max_ah = ib_ipath_max_ahs;
-	props->max_cqe = 0xffff;
+	props->max_cqe = ib_ipath_max_cqe;
 	props->max_mr = dev->lk_table.max;
 	props->max_pd = ib_ipath_max_pds;
 	props->max_qp_rd_atom = 1;
drivers/infiniband/hw/ipath/ipath_verbs.h +2 -0
diff -r e823378bd19c -r 1d3e85454b53 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -690,6 +690,8 @@ extern const int ib_ipath_state_ops[];
 
 extern unsigned int ib_ipath_lkey_table_size;
 
+extern unsigned int ib_ipath_max_cqe;
+
 extern const u32 ib_ipath_rnr_table[];
 
 #endif				/* IPATH_VERBS_H */
#31 lore raw

PATCH 1/1 ipath - don't modify QP if changes fail

In reply to Bryan O'Sullivan

Make sure modify_qp won't modify the QP if any of the changes failed.

drivers/infiniband/hw/ipath/ipath_qp.c +17 -15
diff -r 2fea0d127a41 -r cc6d7f2537b2 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -427,6 +427,7 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 int ipath_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
 		    int attr_mask)
 {
+	struct ipath_ibdev *dev = to_idev(ibqp->device);
 	struct ipath_qp *qp = to_iqp(ibqp);
 	enum ib_qp_state cur_state, new_state;
 	unsigned long flags;
@@ -443,6 +444,19 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 				attr_mask))
 		goto inval;
 
+	if (attr_mask & IB_QP_AV)
+		if (attr->ah_attr.dlid == 0 ||
+		    attr->ah_attr.dlid >= IPS_MULTICAST_LID_BASE)
+			goto inval;
+
+	if (attr_mask & IB_QP_PKEY_INDEX)
+		if (attr->pkey_index >= ipath_layer_get_npkeys(dev->dd))
+			goto inval;
+
+	if (attr_mask & IB_QP_MIN_RNR_TIMER)
+		if (attr->min_rnr_timer > 31)
+			goto inval;
+
 	switch (new_state) {
 	case IB_QPS_RESET:
 		ipath_reset_qp(qp);
@@ -457,13 +471,8 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 
 	}
 
-	if (attr_mask & IB_QP_PKEY_INDEX) {
-		struct ipath_ibdev *dev = to_idev(ibqp->device);
-
-		if (attr->pkey_index >= ipath_layer_get_npkeys(dev->dd))
-			goto inval;
+	if (attr_mask & IB_QP_PKEY_INDEX)
 		qp->s_pkey_index = attr->pkey_index;
-	}
 
 	if (attr_mask & IB_QP_DEST_QPN)
 		qp->remote_qpn = attr->dest_qp_num;
@@ -479,12 +488,8 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 	if (attr_mask & IB_QP_ACCESS_FLAGS)
 		qp->qp_access_flags = attr->qp_access_flags;
 
-	if (attr_mask & IB_QP_AV) {
-		if (attr->ah_attr.dlid == 0 ||
-		    attr->ah_attr.dlid >= IPS_MULTICAST_LID_BASE)
-			goto inval;
+	if (attr_mask & IB_QP_AV)
 		qp->remote_ah_attr = attr->ah_attr;
-	}
 
 	if (attr_mask & IB_QP_PATH_MTU)
 		qp->path_mtu = attr->path_mtu;
@@ -499,11 +504,8 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 		qp->s_rnr_retry_cnt = qp->s_rnr_retry;
 	}
 
-	if (attr_mask & IB_QP_MIN_RNR_TIMER) {
-		if (attr->min_rnr_timer > 31)
-			goto inval;
+	if (attr_mask & IB_QP_MIN_RNR_TIMER)
 		qp->s_min_rnr_timer = attr->min_rnr_timer;
-	}
 
 	if (attr_mask & IB_QP_QKEY)
 		qp->qkey = attr->qkey;
#32 lore raw

PATCH 1/1 ipath - forbid creation of AH with DLID of 0

In reply to Bryan O'Sullivan

Don't allow an AH to be created with a DLID of 0.

drivers/infiniband/hw/ipath/ipath_verbs.c +5 -0
diff -r db56c0ab6a64 -r def81ab50644 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -810,6 +810,11 @@ static struct ib_ah *ipath_create_ah(str
 	if (ah_attr->dlid >= IPS_MULTICAST_LID_BASE &&
 	    ah_attr->dlid != IPS_PERMISSIVE_LID &&
 	    !(ah_attr->ah_flags & IB_AH_GRH)) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
+	if (ah_attr->dlid == 0) {
 		ret = ERR_PTR(-EINVAL);
 		goto bail;
 	}
#33 lore raw

PATCH 1/1 ipath - limit number of SGEs and WRs per QP

In reply to Bryan O'Sullivan

We can't create more than a certain number of SGEs or WRs per QP.

drivers/infiniband/hw/ipath/ipath_cq.c +1 -1
diff -r ab2b013f1f95 -r 02a05b853d20 drivers/infiniband/hw/ipath/ipath_cq.c
@@ -162,7 +162,7 @@ struct ib_cq *ipath_create_cq(struct ib_
 	struct ib_wc *wc;
 	struct ib_cq *ret;
 
-	if (entries > ib_ipath_max_cqe) {
+	if (entries > ib_ipath_max_cqes) {
 		ret = ERR_PTR(-EINVAL);
 		goto bail;
 	}
drivers/infiniband/hw/ipath/ipath_qp.c +4 -2
diff -r ab2b013f1f95 -r 02a05b853d20 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -663,8 +663,10 @@ struct ib_qp *ipath_create_qp(struct ib_
 	size_t sz;
 	struct ib_qp *ret;
 
-	if (init_attr->cap.max_send_sge > 255 ||
-	    init_attr->cap.max_recv_sge > 255) {
+	if (init_attr->cap.max_send_sge > ib_ipath_max_sges ||
+	    init_attr->cap.max_recv_sge > ib_ipath_max_sges ||
+	    init_attr->cap.max_send_wr > ib_ipath_max_qp_wrs ||
+	    init_attr->cap.max_recv_wr > ib_ipath_max_qp_wrs) {
 		ret = ERR_PTR(-ENOMEM);
 		goto bail;
 	}
drivers/infiniband/hw/ipath/ipath_verbs.c +18 -11
diff -r ab2b013f1f95 -r 02a05b853d20 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -62,18 +62,25 @@ MODULE_PARM_DESC(max_pds,
 
 static unsigned int ib_ipath_max_ahs = 0xFFFF;
 module_param_named(max_ahs, ib_ipath_max_ahs, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_ahs,
-		 "Maximum number of address handles to support");
-
-unsigned int ib_ipath_max_cqe = 0xFFFF;
-module_param_named(max_cqe, ib_ipath_max_cqe, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_cqe,
+MODULE_PARM_DESC(max_ahs, "Maximum number of address handles to support");
+
+unsigned int ib_ipath_max_cqes = 0xFFFF;
+module_param_named(max_cqes, ib_ipath_max_cqes, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_cqes,
 		 "Maximum number of completion queue entries to support");
 
 unsigned int ib_ipath_max_cqs = 0xFFFF;
 module_param_named(max_cqs, ib_ipath_max_cqs, uint, S_IWUSR | S_IRUGO);
-MODULE_PARM_DESC(max_cqs,
-		 "Maximum number of completion queues to support");
+MODULE_PARM_DESC(max_cqs, "Maximum number of completion queues to support");
+
+unsigned int ib_ipath_max_qp_wrs = 255;
+module_param_named(max_qp_wrs, ib_ipath_max_qp_wrs, uint,
+		   S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_qp_wrs, "Maximum number of QP WRs to support");
+
+unsigned int ib_ipath_max_sges = 255;
+module_param_named(max_sges, ib_ipath_max_sges, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_sges, "Maximum number of SGEs to support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -604,11 +611,11 @@ static int ipath_query_device(struct ib_
 
 	props->max_mr_size = ~0ull;
 	props->max_qp = dev->qp_table.max;
-	props->max_qp_wr = 0xffff;
-	props->max_sge = 255;
+	props->max_qp_wr = ib_ipath_max_qp_wrs;
+	props->max_sge = ib_ipath_max_sges;
 	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
-	props->max_cqe = ib_ipath_max_cqe;
+	props->max_cqe = ib_ipath_max_cqes;
 	props->max_mr = dev->lk_table.max;
 	props->max_pd = ib_ipath_max_pds;
 	props->max_qp_rd_atom = 1;
drivers/infiniband/hw/ipath/ipath_verbs.h +5 -1
diff -r ab2b013f1f95 -r 02a05b853d20 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -691,10 +691,14 @@ extern const int ib_ipath_state_ops[];
 
 extern unsigned int ib_ipath_lkey_table_size;
 
-extern unsigned int ib_ipath_max_cqe;
+extern unsigned int ib_ipath_max_cqes;
 
 extern unsigned int ib_ipath_max_cqs;
 
+extern unsigned int ib_ipath_max_qp_wrs;
+
+extern unsigned int ib_ipath_max_sges;
+
 extern const u32 ib_ipath_rnr_table[];
 
 #endif				/* IPATH_VERBS_H */
#34 lore raw

PATCH 1/1 ipath - cap number of CQs

In reply to Bryan O'Sullivan

Cap the number of CQs that can be created. Not a real limitation for us, but the user verbs code expects a real number.

drivers/infiniband/hw/ipath/ipath_cq.c +10 -0
diff -r 1d3e85454b53 -r a89145f4846c drivers/infiniband/hw/ipath/ipath_cq.c
@@ -157,12 +157,18 @@ struct ib_cq *ipath_create_cq(struct ib_
 			      struct ib_ucontext *context,
 			      struct ib_udata *udata)
 {
+	struct ipath_ibdev *dev = to_idev(ibdev);
 	struct ipath_cq *cq;
 	struct ib_wc *wc;
 	struct ib_cq *ret;
 
 	if (entries > ib_ipath_max_cqe) {
 		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
+	if (dev->n_cqs_allocated == ib_ipath_max_cqs) {
+		ret = ERR_PTR(-ENOMEM);
 		goto bail;
 	}
 
@@ -201,6 +207,8 @@ struct ib_cq *ipath_create_cq(struct ib_
 
 	ret = &cq->ibcq;
 
+	dev->n_cqs_allocated++;
+
 bail:
 	return ret;
 }
@@ -215,9 +223,11 @@ bail:
  */
 int ipath_destroy_cq(struct ib_cq *ibcq)
 {
+	struct ipath_ibdev *dev = to_idev(ibcq->device);
 	struct ipath_cq *cq = to_icq(ibcq);
 
 	tasklet_kill(&cq->comptask);
+	dev->n_cqs_allocated--;
 	vfree(cq->queue);
 	kfree(cq);
 
drivers/infiniband/hw/ipath/ipath_verbs.c +6 -1
diff -r 1d3e85454b53 -r a89145f4846c drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -69,6 +69,11 @@ module_param_named(max_cqe, ib_ipath_max
 module_param_named(max_cqe, ib_ipath_max_cqe, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_cqe,
 		 "Maximum number of completion queue entries to support");
+
+unsigned int ib_ipath_max_cqs = 0xFFFF;
+module_param_named(max_cqs, ib_ipath_max_cqs, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_cqs,
+		 "Maximum number of completion queues to support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -601,7 +606,7 @@ static int ipath_query_device(struct ib_
 	props->max_qp = dev->qp_table.max;
 	props->max_qp_wr = 0xffff;
 	props->max_sge = 255;
-	props->max_cq = 0xffff;
+	props->max_cq = ib_ipath_max_cqs;
 	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = ib_ipath_max_cqe;
 	props->max_mr = dev->lk_table.max;
drivers/infiniband/hw/ipath/ipath_verbs.h +3 -0
diff -r 1d3e85454b53 -r a89145f4846c drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -433,6 +433,7 @@ struct ipath_ibdev {
 	__be64 mkey;
 	u32 n_pds_allocated;	/* number of PDs allocated for device */
 	u32 n_ahs_allocated;	/* number of AHs allocated for device */
+	u32 n_cqs_allocated;	/* number of CQs allocated for device */
 	u64 ipath_sword;	/* total dwords sent (sample result) */
 	u64 ipath_rword;	/* total dwords received (sample result) */
 	u64 ipath_spkts;	/* total packets sent (sample result) */
@@ -692,6 +693,8 @@ extern unsigned int ib_ipath_lkey_table_
 
 extern unsigned int ib_ipath_max_cqe;
 
+extern unsigned int ib_ipath_max_cqs;
+
 extern const u32 ib_ipath_rnr_table[];
 
 #endif				/* IPATH_VERBS_H */
#35 lore raw

PATCH 1/1 ipath - reduce overhead of receive interrupts

In reply to Bryan O'Sullivan

Somewhat reduce overhead on receive interrupts, and count the number of interrupts where that works (fastrcvint).

drivers/infiniband/hw/ipath/ipath_common.h +2 -2
diff -r cc6d7f2537b2 -r ab2b013f1f95 drivers/infiniband/hw/ipath/ipath_common.h
@@ -96,8 +96,8 @@ struct infinipath_stats {
 	__u64 sps_hwerrs;
 	/* number of times IB link changed state unexpectedly */
 	__u64 sps_iblink;
-	/* no longer used; left for compatibility */
-	__u64 sps_unused3;
+	/* kernel receive interrupts that didn't read intstat */
+	__u64 sps_fastrcvint;
 	/* number of kernel (port0) packets received */
 	__u64 sps_port0pkts;
 	/* number of "ethernet" packets sent by driver */
drivers/infiniband/hw/ipath/ipath_driver.c +1 -10
diff -r cc6d7f2537b2 -r ab2b013f1f95 drivers/infiniband/hw/ipath/ipath_driver.c
@@ -936,12 +936,7 @@ void ipath_kreceive(struct ipath_devdata
 	    (u32)le64_to_cpu(*dd->ipath_hdrqtailptr))
 		goto done;
 
-gotmore:
-	/*
-	 * read only once at start.  If in flood situation, this helps
-	 * performance slightly.  If more arrive while we are processing,
-	 * we'll come back here and do them
-	 */
+	/* read only once at start for performance */
 	hdrqtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
 
 	for (i = 0, l = dd->ipath_port0head; l != hdrqtail; i++) {
@@ -1070,10 +1065,6 @@ gotmore:
 
 	dd->ipath_port0head = l;
 
-	if (hdrqtail != (u32)le64_to_cpu(*dd->ipath_hdrqtailptr))
-		/* more arrived while we handled first batch */
-		goto gotmore;
-
 	if (pkttot > ipath_stats.sps_maxpkts_call)
 		ipath_stats.sps_maxpkts_call = pkttot;
 	ipath_stats.sps_port0pkts += pkttot;
drivers/infiniband/hw/ipath/ipath_intr.c +103 -61

236 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_stats.c +0 -1
diff -r cc6d7f2537b2 -r ab2b013f1f95 drivers/infiniband/hw/ipath/ipath_stats.c
@@ -185,7 +185,6 @@ static void ipath_qcheck(struct ipath_de
 				   dd->ipath_port0head,
 				   (unsigned long long)
 				   ipath_stats.sps_port0pkts);
-			ipath_kreceive(dd);
 		}
 		dd->ipath_lastport0rcv_cnt = ipath_stats.sps_port0pkts;
 	}
#36 lore raw

PATCH 1/1 ipath - fail properly if GID missing

In reply to Bryan O'Sullivan

Return -EINVAL if we can't find a multicast GID.

drivers/infiniband/hw/ipath/ipath_verbs_mcast.c +1 -1
diff -r 176d1f0c26a3 -r c5f3731224bb drivers/infiniband/hw/ipath/ipath_verbs_mcast.c
@@ -272,7 +272,7 @@ int ipath_multicast_detach(struct ib_qp 
 	while (1) {
 		if (n == NULL) {
 			spin_unlock_irqrestore(&mcast_lock, flags);
-			ret = 0;
+			ret = -EINVAL;
 			goto bail;
 		}
 
#37 lore raw

PATCH 1/1 ipath - enable GPIO interrupt on HT-460

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_eeprom.c +4 -5
diff -r b41e576e5202 -r 04c86dd11b27 drivers/infiniband/hw/ipath/ipath_eeprom.c
@@ -505,11 +505,10 @@ static u8 flash_csum(struct ipath_flash 
  * ipath_get_guid - get the GUID from the i2c device
  * @dd: the infinipath device
  *
- * When we add the multi-chip support, we will probably have to add
- * the ability to use the number of guids field, and get the guid from
- * the first chip's flash, to use for all of them.
- */
-void ipath_get_guid(struct ipath_devdata *dd)
+ * We have the capability to use the ipath_nguid field, and get
+ * the guid from the first chip's flash, to use for all of them.
+ */
+void ipath_get_eeprom_info(struct ipath_devdata *dd)
 {
 	void *buf;
 	struct ipath_flash *ifp;
drivers/infiniband/hw/ipath/ipath_ht400.c +18 -1
diff -r b41e576e5202 -r 04c86dd11b27 drivers/infiniband/hw/ipath/ipath_ht400.c
@@ -607,7 +607,12 @@ static int ipath_ht_boardname(struct ipa
 	case 4:		/* Ponderosa is one of the bringup boards */
 		n = "Ponderosa";
 		break;
-	case 5:		/* HT-460 original production board */
+	case 5:	
+		/*
+		 * HT-460 original production board; two production levels, with
+		 * different serial number ranges.   See ipath_ht_early_init() for
+		 * case where we enable IPATH_GPIO_INTR for later serial # range.
+		 */
 		n = "InfiniPath_HT-460";
 		break;
 	case 6:
@@ -1520,6 +1525,18 @@ static int ipath_ht_early_init(struct ip
 	 */
 	ipath_write_kreg(dd, dd->ipath_kregs->kr_sendctrl,
 			 INFINIPATH_S_ABORT);
+
+	ipath_get_eeprom_info(dd);
+	if(dd->ipath_boardrev == 5 && dd->ipath_serial[0] == '1' &&
+		dd->ipath_serial[1] == '2' && dd->ipath_serial[2] == '8') {
+		/*
+		 * Later production HT-460 has same changes as HT-465, so
+		 * can use GPIO interrupts.  They have serial #'s starting
+		 * with 128, rather than 112.
+		 */
+		dd->ipath_flags |= IPATH_GPIO_INTR;
+		dd->ipath_flags &= ~IPATH_POLL_RX_INTR;
+	}
 	return 0;
 }
 
drivers/infiniband/hw/ipath/ipath_init_chip.c +0 -1
diff -r b41e576e5202 -r 04c86dd11b27 drivers/infiniband/hw/ipath/ipath_init_chip.c
@@ -857,7 +857,6 @@ int ipath_init_chip(struct ipath_devdata
 
 done:
 	if (!ret) {
-		ipath_get_guid(dd);
 		*dd->ipath_statusp |= IPATH_STATUS_CHIP_PRESENT;
 		if (!dd->ipath_f_intrsetup(dd)) {
 			/* now we can enable all interrupts from the chip */
drivers/infiniband/hw/ipath/ipath_kernel.h +1 -1
diff -r b41e576e5202 -r 04c86dd11b27 drivers/infiniband/hw/ipath/ipath_kernel.h
@@ -646,7 +646,7 @@ void ipath_init_pe800_funcs(struct ipath
 void ipath_init_pe800_funcs(struct ipath_devdata *);
 /* init HT-400-specific func */
 void ipath_init_ht400_funcs(struct ipath_devdata *);
-void ipath_get_guid(struct ipath_devdata *);
+void ipath_get_eeprom_info(struct ipath_devdata *);
 u64 ipath_snap_cntr(struct ipath_devdata *, ipath_creg);
 
 /*
drivers/infiniband/hw/ipath/ipath_pe800.c +2 -0
diff -r b41e576e5202 -r 04c86dd11b27 drivers/infiniband/hw/ipath/ipath_pe800.c
@@ -1180,6 +1180,8 @@ static int ipath_pe_early_init(struct ip
 	 */
 	dd->ipath_rhdrhead_intr_off = 1ULL<<32;
 
+	ipath_get_eeprom_info(dd);
+
 	return 0;
 }
 
#38 lore raw

PATCH 1/1 ipath - some interrelated stability and cleanliness fixes

In reply to Bryan O'Sullivan

Made in-memory rcvhdrq tail update be in dma_alloc'ed memory, not random user or special kernel (needed for powerpc, also "just the right thing to do".

Some cleanups to make unexpected link transitions less likely to produce complaints about packet errors, and also to not leave SMA packets stuck and unable to go out.

Call dma_free_coherent without ipath_mutex held.

A few other random debug and comment cleanups.

Always init rcvhdrq head/tail registers to 0, to avoid race conditions (should have been that way some time ago).

drivers/infiniband/hw/ipath/ipath_common.h +4 -7
diff -r 09077b2f476f -r e29625bd9050 drivers/infiniband/hw/ipath/ipath_common.h
@@ -307,6 +307,9 @@ struct ipath_base_info {
 	__u32 spi_rcv_egrchunksize;
 	/* total size of mmap to cover full rcvegrbuffers */
 	__u32 spi_rcv_egrbuftotlen;
+	__u32 spi_filler_for_align;
+	/* address of readonly memory copy of the rcvhdrq tail register. */
+	__u64 spi_rcvhdr_tailaddr;
 } __attribute__ ((aligned(8)));
 
 
@@ -376,13 +379,7 @@ struct ipath_user_info {
 	 */
 	__u32 spu_rcvhdrsize;
 
-	/*
-	 * cache line aligned (64 byte) user address to
-	 * which the rcvhdrtail register will be written by infinipath
-	 * whenever it changes, so that no chip registers are read in
-	 * the performance path.
-	 */
-	__u64 spu_rcvhdraddr;
+	__u64 spu_unused; /* kept for compatible layout */
 
 	/*
 	 * address of struct base_info to write to
drivers/infiniband/hw/ipath/ipath_driver.c +116 -160

408 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_file_ops.c +114 -173

432 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_init_chip.c +12 -33

72 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_intr.c +54 -9

105 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_kernel.h +3 -14
diff -r 09077b2f476f -r e29625bd9050 drivers/infiniband/hw/ipath/ipath_kernel.h
@@ -61,9 +61,7 @@ struct ipath_portdata {
 	/* rcvhdrq base, needs mmap before useful */
 	void *port_rcvhdrq;
 	/* kernel virtual address where hdrqtail is updated */
-	u64 *port_rcvhdrtail_kvaddr;
-	/* page * used for uaddr */
-	struct page *port_rcvhdrtail_pagep;
+	volatile __le64 *port_rcvhdrtail_kvaddr;
 	/*
 	 * temp buffer for expected send setup, allocated at open, instead
 	 * of each setup call
@@ -78,11 +76,7 @@ struct ipath_portdata {
 	dma_addr_t port_rcvegr_phys;
 	/* mmap of hdrq, must fit in 44 bits */
 	dma_addr_t port_rcvhdrq_phys;
-	/*
-	 * the actual user address that we ipath_mlock'ed, so we can
-	 * ipath_munlock it at close
-	 */
-	unsigned long port_rcvhdrtail_uaddr;
+	dma_addr_t port_rcvhdrqtailaddr_phys;
 	/*
 	 * number of opens on this instance (0 or 1; ignoring forks, dup,
 	 * etc. for now)
@@ -167,7 +161,6 @@ struct ipath_devdata {
 	 * only written to by the chip, not the driver.
 	 */
 	volatile __le64 *ipath_hdrqtailptr;
-	dma_addr_t ipath_dma_addr;
 	/* ipath_cfgports pointers */
 	struct ipath_portdata **ipath_pd;
 	/* sk_buffs used by port 0 eager receive queue */
@@ -518,10 +511,6 @@ struct ipath_devdata {
 	u8 ipath_lmc;
 };
 
-extern volatile __le64 *ipath_port0_rcvhdrtail;
-extern dma_addr_t ipath_port0_rcvhdrtail_dma;
-
-#define IPATH_PORT0_RCVHDRTAIL_SIZE PAGE_SIZE
 
 extern struct list_head ipath_dev_list;
 extern spinlock_t ipath_devs_lock;
@@ -582,7 +571,7 @@ void ipath_disarm_piobufs(struct ipath_d
 			  unsigned cnt);
 
 int ipath_create_rcvhdrq(struct ipath_devdata *, struct ipath_portdata *);
-void ipath_free_pddata(struct ipath_devdata *, u32, int);
+void ipath_free_pddata(struct ipath_devdata *, struct ipath_portdata *);
 
 int ipath_parse_ushort(const char *str, unsigned short *valp);
 
#39 lore raw

Re: [PATCH 35 of 53] ipath - some interrelated stability and cleanliness fixes

This looks like a pastiche of several patches. Why can't it be split up into logical pieces?

Quoted 1 line

Call dma_free_coherent without ipath_mutex held.

Why? Doesn't freeing work with the mutex held?

- R.

#40 lore raw

PATCH 1/1 ipath - [TRIVIAL] typo fixes

In reply to Bryan O'Sullivan

A few typo fixes.

drivers/infiniband/hw/ipath/ipath_intr.c +1 -1
diff -r 1887e7b3e2a3 -r 8b882bb46a32 drivers/infiniband/hw/ipath/ipath_intr.c
@@ -753,7 +753,7 @@ irqreturn_t ipath_intr(int irq, void *da
 	}
 
 	/*
-	 * We try to avoid readint the interrupt status register, since
+	 * We try to avoid reading the interrupt status register, since
 	 * that's a PIO read, and stalls the processor for up to about
 	 * ~0.25 usec. The idea is that if we processed a port0 packet,
 	 * we blindly clear the  port 0 receive interrupt bits, and nothing
drivers/infiniband/hw/ipath/ipath_layer.c +1 -1
diff -r 1887e7b3e2a3 -r 8b882bb46a32 drivers/infiniband/hw/ipath/ipath_layer.c
@@ -882,7 +882,7 @@ static void copy_io(u32 __iomem *piobuf,
 /**
  * ipath_verbs_send - send a packet from the verbs layer
  * @dd: the infinipath device
- * @hdrwords: the number of works in the header
+ * @hdrwords: the number of words in the header
  * @hdr: the packet header
  * @len: the length of the packet in bytes
  * @ss: the SGE to send
#41 lore raw

PATCH 1/1 ipath - fix memory leak when creating a QP fails

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_qp.c +2 -0
diff -r 0aba84dce506 -r 7634b2f0fc40 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -680,6 +680,7 @@ struct ib_qp *ipath_create_qp(struct ib_
 	case IB_QPT_GSI:
 		qp = kmalloc(sizeof(*qp), GFP_KERNEL);
 		if (!qp) {
+			vfree(swq);
 			ret = ERR_PTR(-ENOMEM);
 			goto bail;
 		}
@@ -690,6 +691,7 @@ struct ib_qp *ipath_create_qp(struct ib_
 		qp->r_rq.wq = vmalloc(qp->r_rq.size * sz);
 		if (!qp->r_rq.wq) {
 			kfree(qp);
+			vfree(swq);
 			ret = ERR_PTR(-ENOMEM);
 			goto bail;
 		}
#42 lore raw

PATCH 1/1 ipath - count local link integrity errors

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_driver.c +27 -11
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_driver.c
@@ -446,6 +446,8 @@ static int __devinit ipath_init_one(stru
 	 * by ipath_setup_htconfig.
 	 */
 	dd->ipath_flags = 0;
+	dd->ipath_lli_counter = 0;
+	dd->ipath_lli_errors = 0;
 
 	if (dd->ipath_f_bus(dd, pdev))
 		ipath_dev_err(dd, "Failed to setup config space; "
@@ -927,6 +929,18 @@ void ipath_kreceive(struct ipath_devdata
 				   "tlen=%x opcode=%x egridx=%x: %s\n",
 				   eflags, l, etype, tlen, bthbytes[0],
 				   ips_get_index((__le32 *) rc), emsg);
+			/* Count local link integrity errors. */
+			if (eflags & (INFINIPATH_RHF_H_ICRCERR |
+				      INFINIPATH_RHF_H_VCRCERR)) {
+				u8 n = (dd->ipath_ibcctrl >>
+					INFINIPATH_IBCC_PHYERRTHRESHOLD_SHIFT) &
+					INFINIPATH_IBCC_PHYERRTHRESHOLD_MASK;
+
+				if (++dd->ipath_lli_counter > n) {
+					dd->ipath_lli_counter = 0;
+					dd->ipath_lli_errors++;
+				}
+			}
 		} else if (etype == RCVHQ_RCV_TYPE_NON_KD) {
 				int ret = __ipath_verbs_rcv(dd, rc + 1,
 							    ebuf, tlen);
@@ -934,6 +948,8 @@ void ipath_kreceive(struct ipath_devdata
 					ipath_cdbg(VERBOSE,
 						   "received IB packet, "
 						   "not SMA (QP=%x)\n", qp);
+				if (dd->ipath_lli_counter)
+					dd->ipath_lli_counter--;
 		} else if (etype == RCVHQ_RCV_TYPE_EAGER) {
 			if (qp == IPATH_KD_QP &&
 			    bthbytes[0] == ipath_layer_rcv_opcode &&
@@ -1864,19 +1880,19 @@ static void __exit infinipath_cleanup(vo
 			} else
 				ipath_dbg("irq is 0, not doing free_irq "
 					  "for unit %u\n", dd->ipath_unit);
+
+			/*
+			 * we check for NULL here, because it's outside
+			 * the kregbase check, and we need to call it
+			 * after the free_irq.  Thus it's possible that
+			 * the function pointers were never initialized.
+			 */
+			if (dd->ipath_f_cleanup)
+				/* clean up chip-specific stuff */
+				dd->ipath_f_cleanup(dd);
+
 			dd->pcidev = NULL;
 		}
-
-		/*
-		 * we check for NULL here, because it's outside the kregbase
-		 * check, and we need to call it after the free_irq.  Thus
-		 * it's possible that the function pointers were never
-		 * initialized.
-		 */
-		if (dd->ipath_f_cleanup)
-			/* clean up chip-specific stuff */
-			dd->ipath_f_cleanup(dd);
-
 		spin_lock_irqsave(&ipath_devs_lock, flags);
 	}
 
drivers/infiniband/hw/ipath/ipath_intr.c +1 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_intr.c
@@ -261,6 +261,7 @@ static void handle_e_ibstatuschanged(str
 				     | IPATH_LINKACTIVE |
 				     IPATH_LINKARMED);
 		*dd->ipath_statusp &= ~IPATH_STATUS_IB_READY;
+		dd->ipath_lli_counter = 0;
 		if (!noprint) {
 			if (((dd->ipath_lastibcstat >>
 			      INFINIPATH_IBCS_LINKSTATE_SHIFT) &
drivers/infiniband/hw/ipath/ipath_kernel.h +5 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_kernel.h
@@ -509,6 +509,11 @@ struct ipath_devdata {
 	u8 ipath_pci_cacheline;
 	/* LID mask control */
 	u8 ipath_lmc;
+
+	/* local link integrity counter */
+	u32 ipath_lli_counter;
+	/* local link integrity errors */
+	u32 ipath_lli_errors;
 };
 
 
drivers/infiniband/hw/ipath/ipath_layer.c +7 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_layer.c
@@ -1013,6 +1013,11 @@ int ipath_layer_get_counters(struct ipat
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_ibsymbolerrcnt);
 	cntrs->link_error_recovery_counter =
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_iblinkerrrecovcnt);
+	/*
+	 * The link downed counter counts when the other side downs the
+	 * connection.  We add in the number of times we downed the link
+	 * due to local link integrity errors to compensate.
+	 */
 	cntrs->link_downed_counter =
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_iblinkdowncnt);
 	cntrs->port_rcv_errors =
@@ -1037,6 +1042,8 @@ int ipath_layer_get_counters(struct ipat
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktsendcnt);
 	cntrs->port_rcv_packets =
 		ipath_snap_cntr(dd, dd->ipath_cregs->cr_pktrcvcnt);
+	cntrs->local_link_integrity_errors = dd->ipath_lli_errors;
+	cntrs->excessive_buffer_overrun_errors = 0; /* XXX */
 
 	ret = 0;
 
drivers/infiniband/hw/ipath/ipath_layer.h +2 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_layer.h
@@ -54,6 +54,8 @@ struct ipath_layer_counters {
 	u64 port_rcv_data;
 	u64 port_xmit_packets;
 	u64 port_rcv_packets;
+	u32 local_link_integrity_errors;
+	u32 excessive_buffer_overrun_errors;
 };
 
 /*
drivers/infiniband/hw/ipath/ipath_mad.c +20 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_mad.c
@@ -646,6 +646,8 @@ struct ib_pma_portcounters {
 #define IB_PMA_SEL_PORT_RCV_ERRORS		__constant_htons(0x0008)
 #define IB_PMA_SEL_PORT_RCV_REMPHYS_ERRORS	__constant_htons(0x0010)
 #define IB_PMA_SEL_PORT_XMIT_DISCARDS		__constant_htons(0x0040)
+#define IB_PMA_SEL_LOCAL_LINK_INTEGRITY_ERRORS	__constant_htons(0x0200)
+#define IB_PMA_SEL_EXCESSIVE_BUFFER_OVERRUNS	__constant_htons(0x0400)
 #define IB_PMA_SEL_PORT_VL15_DROPPED		__constant_htons(0x0800)
 #define IB_PMA_SEL_PORT_XMIT_DATA		__constant_htons(0x1000)
 #define IB_PMA_SEL_PORT_RCV_DATA		__constant_htons(0x2000)
@@ -893,6 +895,10 @@ static int recv_pma_get_portcounters(str
 	cntrs.port_rcv_data -= dev->n_port_rcv_data;
 	cntrs.port_xmit_packets -= dev->n_port_xmit_packets;
 	cntrs.port_rcv_packets -= dev->n_port_rcv_packets;
+	cntrs.local_link_integrity_errors -=
+		dev->z_local_link_integrity_errors;
+	cntrs.excessive_buffer_overrun_errors -=
+		dev->z_excessive_buffer_overrun_errors;
 
 	memset(pmp->data, 0, sizeof(pmp->data));
 
@@ -930,6 +936,12 @@ static int recv_pma_get_portcounters(str
 	else
 		p->port_xmit_discards =
 			cpu_to_be16((u16)cntrs.port_xmit_discards);
+	if (cntrs.local_link_integrity_errors > 0xFUL)
+		cntrs.local_link_integrity_errors = 0xFUL;
+	if (cntrs.excessive_buffer_overrun_errors > 0xFUL)
+		cntrs.excessive_buffer_overrun_errors = 0xFUL;
+	p->lli_ebor_errors = (cntrs.local_link_integrity_errors << 4) |
+		cntrs.excessive_buffer_overrun_errors;
 	if (dev->n_vl15_dropped > 0xFFFFUL)
 		p->vl15_dropped = __constant_cpu_to_be16(0xFFFF);
 	else
@@ -1028,6 +1040,14 @@ static int recv_pma_set_portcounters(str
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DISCARDS)
 		dev->n_port_xmit_discards = cntrs.port_xmit_discards;
 
+	if (p->counter_select & IB_PMA_SEL_LOCAL_LINK_INTEGRITY_ERRORS)
+		dev->z_local_link_integrity_errors =
+			cntrs.local_link_integrity_errors;
+
+	if (p->counter_select & IB_PMA_SEL_EXCESSIVE_BUFFER_OVERRUNS)
+		dev->z_excessive_buffer_overrun_errors =
+			cntrs.excessive_buffer_overrun_errors;
+
 	if (p->counter_select & IB_PMA_SEL_PORT_VL15_DROPPED)
 		dev->n_vl15_dropped = 0;
 
drivers/infiniband/hw/ipath/ipath_verbs.c +4 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -1046,6 +1046,10 @@ static void *ipath_register_ib_device(in
 	idev->n_port_rcv_data = cntrs.port_rcv_data;
 	idev->n_port_xmit_packets = cntrs.port_xmit_packets;
 	idev->n_port_rcv_packets = cntrs.port_rcv_packets;
+	idev->z_local_link_integrity_errors =
+		cntrs.local_link_integrity_errors;
+	idev->z_excessive_buffer_overrun_errors =
+		cntrs.excessive_buffer_overrun_errors;
 
 	/*
 	 * The system image GUID is supposed to be the same for all
drivers/infiniband/hw/ipath/ipath_verbs.h +2 -0
diff -r e29625bd9050 -r ec1934faf5d1 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -459,6 +459,8 @@ struct ipath_ibdev {
 	u64 n_port_xmit_packets;	/* starting count for PMA */
 	u64 n_port_rcv_packets;	/* starting count for PMA */
 	u32 n_pkey_violations;	/* starting count for PMA */
+	u32 z_local_link_integrity_errors;	/* starting count for PMA */
+	u32 z_excessive_buffer_overrun_errors;	/* starting count for PMA */
 	u32 n_rc_resends;
 	u32 n_rc_acks;
 	u32 n_rc_qacks;
#43 lore raw

PATCH 1/1 ipath - disable interrupts while holding spinlock in RWQE get

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_ruc.c +2 -1
diff -r 160a111381ae -r 83f1832c6015 drivers/infiniband/hw/ipath/ipath_ruc.c
@@ -171,12 +171,13 @@ int ipath_get_rwqe(struct ipath_qp *qp, 
 			n = rq->head - rq->tail;
 		if (n < srq->limit) {
 			srq->limit = 0;
-			spin_unlock(&rq->lock);
+			spin_unlock_irqrestore(&rq->lock, flags);
 			ev.device = qp->ibqp.device;
 			ev.element.srq = qp->ibqp.srq;
 			ev.event = IB_EVENT_SRQ_LIMIT_REACHED;
 			srq->ibsrq.event_handler(&ev,
 						 srq->ibsrq.srq_context);
+			spin_lock_irqsave(&rq->lock, flags);
 		}
 	}
 done:
#44 lore raw

Re: [PATCH 41 of 53] ipath - disable interrupts while holding spinlock in RWQE get

drivers/infiniband/hw/ipath/ipath_ruc.c int ipath_get_rwqe(struct ipath_qp *qp, · line 180

Quoted 12 lines
drivers/infiniband/hw/ipath/ipath_ruc.c +2 -1
@@ -171,12 +171,13 @@ int ipath_get_rwqe(struct ipath_qp *qp, 
 			n = rq->head - rq->tail;
 		if (n < srq->limit) {
 			srq->limit = 0;
-			spin_unlock(&rq->lock);
+			spin_unlock_irqrestore(&rq->lock, flags);
 			ev.device = qp->ibqp.device;
 			ev.element.srq = qp->ibqp.srq;
 			ev.event = IB_EVENT_SRQ_LIMIT_REACHED;
 			srq->ibsrq.event_handler(&ev,
 						 srq->ibsrq.srq_context);
+			spin_lock_irqsave(&rq->lock, flags);

ipath_get_rwqe() in the kernel now doesn't even have a flags variable. So this looks like a bug introduced earlier in this patch series. Please roll the fix up into the place where you added the bug.

- R.

#45 lore raw

PATCH 1/1 ipath - SRQ compliance checks

In reply to Bryan O'Sullivan

We were not rigorous enough in checking SRQs.

drivers/infiniband/hw/ipath/ipath_srq.c +38 -15
diff -r f8debae94d44 -r e9306861dc6a drivers/infiniband/hw/ipath/ipath_srq.c
@@ -125,11 +125,23 @@ struct ib_srq *ipath_create_srq(struct i
 				struct ib_srq_init_attr *srq_init_attr,
 				struct ib_udata *udata)
 {
+	struct ipath_ibdev *dev = to_idev(ibpd->device);
 	struct ipath_srq *srq;
 	u32 sz;
 	struct ib_srq *ret;
 
-	if (srq_init_attr->attr.max_sge < 1) {
+	if (dev->n_srqs_allocated == ib_ipath_max_srqs) {
+		ret = ERR_PTR(-ENOMEM);
+		goto bail;
+	}
+
+	if (srq_init_attr->attr.max_wr == 0) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
+	if ((srq_init_attr->attr.max_sge > ib_ipath_max_srq_sges) ||
+	    (srq_init_attr->attr.max_wr > ib_ipath_max_srq_wrs)) {
 		ret = ERR_PTR(-EINVAL);
 		goto bail;
 	}
@@ -164,6 +176,8 @@ struct ib_srq *ipath_create_srq(struct i
 
 	ret = &srq->ibsrq;
 
+	dev->n_srqs_allocated++;
+
 bail:
 	return ret;
 }
@@ -181,24 +195,26 @@ int ipath_modify_srq(struct ib_srq *ibsr
 	unsigned long flags;
 	int ret;
 
-	if (attr_mask & IB_SRQ_LIMIT) {
-		spin_lock_irqsave(&srq->rq.lock, flags);
-		srq->limit = attr->srq_limit;
-		spin_unlock_irqrestore(&srq->rq.lock, flags);
-	}
+	if (attr_mask & IB_SRQ_MAX_WR)
+		if ((attr->max_wr > ib_ipath_max_srq_wrs) ||
+		    (attr->max_sge > srq->rq.max_sge)) {
+			ret = -EINVAL;
+			goto bail;
+		}
+
+	if (attr_mask & IB_SRQ_LIMIT)
+		if (attr->srq_limit >= srq->rq.size) {
+			ret = -EINVAL;
+			goto bail;
+		}
+
 	if (attr_mask & IB_SRQ_MAX_WR) {
-		u32 size = attr->max_wr + 1;
 		struct ipath_rwqe *wq, *p;
-		u32 n;
-		u32 sz;
-
-		if (attr->max_sge < srq->rq.max_sge) {
-			ret = -EINVAL;
-			goto bail;
-		}
+		u32 sz, size, n;
 
 		sz = sizeof(struct ipath_rwqe) +
 			attr->max_sge * sizeof(struct ipath_sge);
+		size = attr->max_wr + 1;
 		wq = vmalloc(size * sz);
 		if (!wq) {
 			ret = -ENOMEM;
@@ -242,6 +258,11 @@ int ipath_modify_srq(struct ib_srq *ibsr
 		spin_unlock_irqrestore(&srq->rq.lock, flags);
 	}
 
+	if (attr_mask & IB_SRQ_LIMIT) {
+		spin_lock_irqsave(&srq->rq.lock, flags);
+		srq->limit = attr->srq_limit;
+		spin_unlock_irqrestore(&srq->rq.lock, flags);
+	}
 	ret = 0;
 
 bail:
@@ -265,7 +286,9 @@ int ipath_destroy_srq(struct ib_srq *ibs
 int ipath_destroy_srq(struct ib_srq *ibsrq)
 {
 	struct ipath_srq *srq = to_isrq(ibsrq);
-
+	struct ipath_ibdev *dev = to_idev(ibsrq->device);
+
+	dev->n_srqs_allocated--;
 	vfree(srq->rq.wq);
 	kfree(srq);
 
drivers/infiniband/hw/ipath/ipath_verbs.h +1 -0
diff -r f8debae94d44 -r e9306861dc6a drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -436,6 +436,7 @@ struct ipath_ibdev {
 	u32 n_pds_allocated;	/* number of PDs allocated for device */
 	u32 n_ahs_allocated;	/* number of AHs allocated for device */
 	u32 n_cqs_allocated;	/* number of CQs allocated for device */
+	u32 n_srqs_allocated;	/* number of SRQs allocated for device */
 	u32 n_mcast_grps_allocated; /* number of mcast groups allocated */
 	u64 ipath_sword;	/* total dwords sent (sample result) */
 	u64 ipath_rword;	/* total dwords received (sample result) */
#46 lore raw

PATCH 1/1 ipath - name zero counter offsets consistently

In reply to Bryan O'Sullivan

Name zero counter offsets consistently so it's clear they aren't counters.

drivers/infiniband/hw/ipath/ipath_mad.c +30 -30
diff -r ec1934faf5d1 -r f8debae94d44 drivers/infiniband/hw/ipath/ipath_mad.c
@@ -251,7 +251,7 @@ static int recv_subn_get_portinfo(struct
 	/* P_KeyViolations are counted by hardware. */
 	pip->pkey_violations =
 		cpu_to_be16((ipath_layer_get_cr_errpkey(dev->dd) -
-			     dev->n_pkey_violations) & 0xFFFF);
+			     dev->z_pkey_violations) & 0xFFFF);
 	pip->qkey_violations = cpu_to_be16(dev->qkey_violations);
 	/* Only the hardware GUID is supported for now */
 	pip->guid_cap = 1;
@@ -425,7 +425,7 @@ static int recv_subn_set_portinfo(struct
 	 * later.
 	 */
 	if (pip->pkey_violations == 0)
-		dev->n_pkey_violations =
+		dev->z_pkey_violations =
 			ipath_layer_get_cr_errpkey(dev->dd);
 
 	if (pip->qkey_violations == 0)
@@ -883,18 +883,18 @@ static int recv_pma_get_portcounters(str
 	ipath_layer_get_counters(dev->dd, &cntrs);
 
 	/* Adjust counters for any resets done. */
-	cntrs.symbol_error_counter -= dev->n_symbol_error_counter;
+	cntrs.symbol_error_counter -= dev->z_symbol_error_counter;
 	cntrs.link_error_recovery_counter -=
-		dev->n_link_error_recovery_counter;
-	cntrs.link_downed_counter -= dev->n_link_downed_counter;
+		dev->z_link_error_recovery_counter;
+	cntrs.link_downed_counter -= dev->z_link_downed_counter;
 	cntrs.port_rcv_errors += dev->rcv_errors;
-	cntrs.port_rcv_errors -= dev->n_port_rcv_errors;
-	cntrs.port_rcv_remphys_errors -= dev->n_port_rcv_remphys_errors;
-	cntrs.port_xmit_discards -= dev->n_port_xmit_discards;
-	cntrs.port_xmit_data -= dev->n_port_xmit_data;
-	cntrs.port_rcv_data -= dev->n_port_rcv_data;
-	cntrs.port_xmit_packets -= dev->n_port_xmit_packets;
-	cntrs.port_rcv_packets -= dev->n_port_rcv_packets;
+	cntrs.port_rcv_errors -= dev->z_port_rcv_errors;
+	cntrs.port_rcv_remphys_errors -= dev->z_port_rcv_remphys_errors;
+	cntrs.port_xmit_discards -= dev->z_port_xmit_discards;
+	cntrs.port_xmit_data -= dev->z_port_xmit_data;
+	cntrs.port_rcv_data -= dev->z_port_rcv_data;
+	cntrs.port_xmit_packets -= dev->z_port_xmit_packets;
+	cntrs.port_rcv_packets -= dev->z_port_rcv_packets;
 	cntrs.local_link_integrity_errors -=
 		dev->z_local_link_integrity_errors;
 	cntrs.excessive_buffer_overrun_errors -=
@@ -981,10 +981,10 @@ static int recv_pma_get_portcounters_ext
 				      &rpkts, &xwait);
 
 	/* Adjust counters for any resets done. */
-	swords -= dev->n_port_xmit_data;
-	rwords -= dev->n_port_rcv_data;
-	spkts -= dev->n_port_xmit_packets;
-	rpkts -= dev->n_port_rcv_packets;
+	swords -= dev->z_port_xmit_data;
+	rwords -= dev->z_port_rcv_data;
+	spkts -= dev->z_port_xmit_packets;
+	rpkts -= dev->z_port_rcv_packets;
 
 	memset(pmp->data, 0, sizeof(pmp->data));
 
@@ -1020,25 +1020,25 @@ static int recv_pma_set_portcounters(str
 	ipath_layer_get_counters(dev->dd, &cntrs);
 
 	if (p->counter_select & IB_PMA_SEL_SYMBOL_ERROR)
-		dev->n_symbol_error_counter = cntrs.symbol_error_counter;
+		dev->z_symbol_error_counter = cntrs.symbol_error_counter;
 
 	if (p->counter_select & IB_PMA_SEL_LINK_ERROR_RECOVERY)
-		dev->n_link_error_recovery_counter =
+		dev->z_link_error_recovery_counter =
 			cntrs.link_error_recovery_counter;
 
 	if (p->counter_select & IB_PMA_SEL_LINK_DOWNED)
-		dev->n_link_downed_counter = cntrs.link_downed_counter;
+		dev->z_link_downed_counter = cntrs.link_downed_counter;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_RCV_ERRORS)
-		dev->n_port_rcv_errors =
+		dev->z_port_rcv_errors =
 			cntrs.port_rcv_errors + dev->rcv_errors;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_RCV_REMPHYS_ERRORS)
-		dev->n_port_rcv_remphys_errors =
+		dev->z_port_rcv_remphys_errors =
 			cntrs.port_rcv_remphys_errors;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DISCARDS)
-		dev->n_port_xmit_discards = cntrs.port_xmit_discards;
+		dev->z_port_xmit_discards = cntrs.port_xmit_discards;
 
 	if (p->counter_select & IB_PMA_SEL_LOCAL_LINK_INTEGRITY_ERRORS)
 		dev->z_local_link_integrity_errors =
@@ -1052,16 +1052,16 @@ static int recv_pma_set_portcounters(str
 		dev->n_vl15_dropped = 0;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DATA)
-		dev->n_port_xmit_data = cntrs.port_xmit_data;
+		dev->z_port_xmit_data = cntrs.port_xmit_data;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_RCV_DATA)
-		dev->n_port_rcv_data = cntrs.port_rcv_data;
+		dev->z_port_rcv_data = cntrs.port_rcv_data;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_PACKETS)
-		dev->n_port_xmit_packets = cntrs.port_xmit_packets;
+		dev->z_port_xmit_packets = cntrs.port_xmit_packets;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_RCV_PACKETS)
-		dev->n_port_rcv_packets = cntrs.port_rcv_packets;
+		dev->z_port_rcv_packets = cntrs.port_rcv_packets;
 
 	return recv_pma_get_portcounters(pmp, ibdev, port);
 }
@@ -1078,16 +1078,16 @@ static int recv_pma_set_portcounters_ext
 				      &rpkts, &xwait);
 
 	if (p->counter_select & IB_PMA_SELX_PORT_XMIT_DATA)
-		dev->n_port_xmit_data = swords;
+		dev->z_port_xmit_data = swords;
 
 	if (p->counter_select & IB_PMA_SELX_PORT_RCV_DATA)
-		dev->n_port_rcv_data = rwords;
+		dev->z_port_rcv_data = rwords;
 
 	if (p->counter_select & IB_PMA_SELX_PORT_XMIT_PACKETS)
-		dev->n_port_xmit_packets = spkts;
+		dev->z_port_xmit_packets = spkts;
 
 	if (p->counter_select & IB_PMA_SELX_PORT_RCV_PACKETS)
-		dev->n_port_rcv_packets = rpkts;
+		dev->z_port_rcv_packets = rpkts;
 
 	if (p->counter_select & IB_PMA_SELX_PORT_UNI_XMIT_PACKETS)
 		dev->n_unicast_xmit = 0;
drivers/infiniband/hw/ipath/ipath_verbs.c +11 -11
diff -r ec1934faf5d1 -r f8debae94d44 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -700,7 +700,7 @@ static int ipath_query_port(struct ib_de
 	props->max_msg_sz = 4096;
 	props->pkey_tbl_len = ipath_layer_get_npkeys(dev->dd);
 	props->bad_pkey_cntr = ipath_layer_get_cr_errpkey(dev->dd) -
-		dev->n_pkey_violations;
+		dev->z_pkey_violations;
 	props->qkey_viol_cntr = dev->qkey_violations;
 	props->active_width = IB_WIDTH_4X;
 	/* See rate_show() */
@@ -1034,18 +1034,18 @@ static void *ipath_register_ib_device(in
 
 	/* Snapshot current HW counters to "clear" them. */
 	ipath_layer_get_counters(dd, &cntrs);
-	idev->n_symbol_error_counter = cntrs.symbol_error_counter;
-	idev->n_link_error_recovery_counter =
+	idev->z_symbol_error_counter = cntrs.symbol_error_counter;
+	idev->z_link_error_recovery_counter =
 		cntrs.link_error_recovery_counter;
-	idev->n_link_downed_counter = cntrs.link_downed_counter;
-	idev->n_port_rcv_errors = cntrs.port_rcv_errors;
-	idev->n_port_rcv_remphys_errors =
+	idev->z_link_downed_counter = cntrs.link_downed_counter;
+	idev->z_port_rcv_errors = cntrs.port_rcv_errors;
+	idev->z_port_rcv_remphys_errors =
 		cntrs.port_rcv_remphys_errors;
-	idev->n_port_xmit_discards = cntrs.port_xmit_discards;
-	idev->n_port_xmit_data = cntrs.port_xmit_data;
-	idev->n_port_rcv_data = cntrs.port_rcv_data;
-	idev->n_port_xmit_packets = cntrs.port_xmit_packets;
-	idev->n_port_rcv_packets = cntrs.port_rcv_packets;
+	idev->z_port_xmit_discards = cntrs.port_xmit_discards;
+	idev->z_port_xmit_data = cntrs.port_xmit_data;
+	idev->z_port_rcv_data = cntrs.port_rcv_data;
+	idev->z_port_xmit_packets = cntrs.port_xmit_packets;
+	idev->z_port_rcv_packets = cntrs.port_rcv_packets;
 	idev->z_local_link_integrity_errors =
 		cntrs.local_link_integrity_errors;
 	idev->z_excessive_buffer_overrun_errors =
drivers/infiniband/hw/ipath/ipath_verbs.h +11 -11
diff -r ec1934faf5d1 -r f8debae94d44 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -448,17 +448,17 @@ struct ipath_ibdev {
 	u64 n_unicast_rcv;	/* total unicast packets received */
 	u64 n_multicast_xmit;	/* total multicast packets sent */
 	u64 n_multicast_rcv;	/* total multicast packets received */
-	u64 n_symbol_error_counter;	/* starting count for PMA */
-	u64 n_link_error_recovery_counter;	/* starting count for PMA */
-	u64 n_link_downed_counter;	/* starting count for PMA */
-	u64 n_port_rcv_errors;	/* starting count for PMA */
-	u64 n_port_rcv_remphys_errors;	/* starting count for PMA */
-	u64 n_port_xmit_discards;	/* starting count for PMA */
-	u64 n_port_xmit_data;	/* starting count for PMA */
-	u64 n_port_rcv_data;	/* starting count for PMA */
-	u64 n_port_xmit_packets;	/* starting count for PMA */
-	u64 n_port_rcv_packets;	/* starting count for PMA */
-	u32 n_pkey_violations;	/* starting count for PMA */
+	u64 z_symbol_error_counter;		/* starting count for PMA */
+	u64 z_link_error_recovery_counter;	/* starting count for PMA */
+	u64 z_link_downed_counter;		/* starting count for PMA */
+	u64 z_port_rcv_errors;			/* starting count for PMA */
+	u64 z_port_rcv_remphys_errors;		/* starting count for PMA */
+	u64 z_port_xmit_discards;		/* starting count for PMA */
+	u64 z_port_xmit_data;			/* starting count for PMA */
+	u64 z_port_rcv_data;			/* starting count for PMA */
+	u64 z_port_xmit_packets;		/* starting count for PMA */
+	u64 z_port_rcv_packets;			/* starting count for PMA */
+	u32 z_pkey_violations;			/* starting count for PMA */
 	u32 z_local_link_integrity_errors;	/* starting count for PMA */
 	u32 z_excessive_buffer_overrun_errors;	/* starting count for PMA */
 	u32 n_rc_resends;
#47 lore raw

PATCH 1/1 ipath - fix problem with lost interrupts on HT-400

In reply to Bryan O'Sullivan

We can have a race clearing chip interrupt with another interrupt about to be delivered and can clear it before it is delivered on the GPIO workaround. By doing the extra check here for the in-memory tail register updating while we were doing earlier packets, we "almost" guarantee we have covered that case.

drivers/infiniband/hw/ipath/ipath_driver.c +24 -7
diff -r 04c86dd11b27 -r a1615956e57f drivers/infiniband/hw/ipath/ipath_driver.c
@@ -858,7 +858,7 @@ void ipath_kreceive(struct ipath_devdata
 	const u32 maxcnt = dd->ipath_rcvhdrcnt * rsize;	/* words */
 	u32 etail = -1, l, hdrqtail;
 	struct ips_message_header *hdr;
-	u32 eflags, i, etype, tlen, pkttot = 0, updegr=0;
+	u32 eflags, i, etype, tlen, pkttot = 0, updegr=0, reloop=0;
 	static u64 totcalls;	/* stats, may eventually remove */
 	char emsg[128];
 
@@ -873,12 +873,11 @@ void ipath_kreceive(struct ipath_devdata
 		goto bail;
 
 	l = dd->ipath_port0head;
-	if(l == (u32)le64_to_cpu(*dd->ipath_hdrqtailptr))
+	hdrqtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
+	if(l == hdrqtail)
 		goto done;
 
-	/* read only once at start for performance */
-	hdrqtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
-
+reloop:
 	for (i = 0; l != hdrqtail; i++) {
 		u32 qp;
 		u8 *bthbytes;
@@ -1013,16 +1012,34 @@ void ipath_kreceive(struct ipath_devdata
 		 */
 		if(l == hdrqtail || (i && !(i&0xf))) {
 			u64 lval;
-			if(l == hdrqtail) /* want interrupt only on last */
+			if(l == hdrqtail) {
+				/* PE-800 interrupt only on last */
 				lval = dd->ipath_rhdrhead_intr_off | l;
+			}
 			else
 				lval = l;
 			(void)ipath_write_ureg(dd, ur_rcvhdrhead, lval, 0);
 			if(updegr) {
-				(void)ipath_write_ureg(dd, ur_rcvegrindexhead,
+				ipath_write_ureg(dd, ur_rcvegrindexhead,
 						       etail, 0);
 				updegr = 0;
 			}
+		}
+	}
+	if(!dd->ipath_rhdrhead_intr_off && !reloop) {
+		/* HT-400 workaround; we can have a race clearing chip
+		 * interrupt with another interrupt about to be delivered,
+		 * and can clear it before it is delivered on the GPIO
+		 * workaround.  By doing the extra check here for the
+		 * in-memory tail register updating while we were doing
+		 * earlier packets, we "almost" guarantee we have covered
+		 * that case.
+		 */
+		u32 hqtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
+		if(hqtail != hdrqtail) {
+		    	hdrqtail = hqtail;
+			reloop = 1; /* loop 1 extra time at most */
+			goto reloop;
 		}
 	}
 
drivers/infiniband/hw/ipath/ipath_intr.c +17 -16
diff -r 04c86dd11b27 -r a1615956e57f drivers/infiniband/hw/ipath/ipath_intr.c
@@ -761,13 +761,14 @@ static void handle_urcv(struct ipath_dev
 }
 
 
+
 irqreturn_t ipath_intr(int irq, void *data, struct pt_regs *regs)
 {
 	struct ipath_devdata *dd = data;
 	u32 istat, chk0rcv = 0;
 	ipath_err_t estat = 0;
 	irqreturn_t ret;
-	u32 p0bits, oldhead;
+	u32 oldhead, curtail;
 	static unsigned unexpected = 0;
 	static const u32 port0rbits = (1U<<INFINIPATH_I_RCVAVAIL_SHIFT) |
 		 (1U<<INFINIPATH_I_RCVURG_SHIFT);
@@ -810,15 +811,16 @@ irqreturn_t ipath_intr(int irq, void *da
 	 * lose intr for later packets that arrive while we are processing.
 	 */
 	oldhead = dd->ipath_port0head;
-	if (oldhead != (u32)le64_to_cpu(*dd->ipath_hdrqtailptr)) {
+	curtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
+	if (oldhead != curtail) {
 		if(dd->ipath_flags & IPATH_GPIO_INTR) {
 			ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_clear,
 					 (u64) (1 << 2));
-			p0bits = port0rbits | INFINIPATH_I_GPIO;
+			istat = port0rbits | INFINIPATH_I_GPIO;
 		}
 		else
-			p0bits = port0rbits;
-		ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, p0bits);
+			istat = port0rbits;
+		ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, istat);
 		ipath_kreceive(dd);
 		if(oldhead != dd->ipath_port0head) {
 			ipath_stats.sps_fastrcvint++;
@@ -827,7 +829,6 @@ irqreturn_t ipath_intr(int irq, void *da
 	}
 
 	istat = ipath_read_kreg32(dd, dd->ipath_kregs->kr_intstatus);
-	p0bits = port0rbits;
 
 	if (unlikely(!istat)) {
 		ipath_stats.sps_nullintr++;
@@ -890,19 +891,19 @@ irqreturn_t ipath_intr(int irq, void *da
 		else {
 			/* Clear GPIO status bit 2 */
 			ipath_write_kreg(dd, dd->ipath_kregs->kr_gpio_clear,
-					 (u64) (1 << 2));
-			p0bits |= INFINIPATH_I_GPIO;
+					(u64) (1 << 2));
 			chk0rcv = 1;
 		}
 	}
-	chk0rcv |= istat & p0bits;
-
-	/*
-	 * clear the ones we will deal with on this round
-	 * We clear it early, mostly for receive interrupts, so we
-	 * know the chip will have seen this by the time we process
-	 * the queue, and will re-interrupt if necessary.  The processor
-	 * itself won't take the interrupt again until we return.
+	chk0rcv |= istat & port0rbits;
+
+	/*
+	 * Clear the interrupt bits we found set, unless they are receive
+	 * related, in which case we already cleared them above, and don't
+	 * want to clear them again, because we might lose an interrupt.
+	 * Clear it early, so we "know" know the chip will have seen this by
+	 * the time we process the queue, and will re-interrupt if necessary.
+	 * The processor itself won't take the interrupt again until we return.
 	 */
 	ipath_write_kreg(dd, dd->ipath_kregs->kr_intclear, istat);
 
#48 lore raw

PATCH 1/1 ipath - NULL-terminate pci_device_id table

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_driver.c +1 -0
diff -r 49b446b12f16 -r 40532fdc53f0 drivers/infiniband/hw/ipath/ipath_driver.c
@@ -120,6 +120,7 @@ static const struct pci_device_id ipath_
 		    PCI_DEVICE_ID_INFINIPATH_HT)},
 	{PCI_DEVICE(PCI_VENDOR_ID_PATHSCALE,
 		    PCI_DEVICE_ID_INFINIPATH_PE800)},
+	{0}
 };
 
 MODULE_DEVICE_TABLE(pci, ipath_pci_tbl);
#49 lore raw

PATCH 1/1 ipath - increment pointer properly when doing a diag read

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_diag.c +5 -4
diff -r 83f1832c6015 -r 0aba84dce506 drivers/infiniband/hw/ipath/ipath_diag.c
@@ -113,7 +113,7 @@ static int ipath_read_umem64(struct ipat
 			goto bail;
 		}
 		reg_addr++;
-		uaddr++;
+		uaddr += sizeof(u64);
 	}
 	ret = 0;
 bail:
@@ -153,7 +153,7 @@ static int ipath_write_umem64(struct ipa
 		writeq(data, reg_addr);
 
 		reg_addr++;
-		uaddr++;
+		uaddr += sizeof(u64);
 	}
 	ret = 0;
 bail:
@@ -191,7 +191,8 @@ static int ipath_read_umem32(struct ipat
 		}
 
 		reg_addr++;
-		uaddr++;
+		uaddr += sizeof(u32);
+
 	}
 	ret = 0;
 bail:
@@ -230,7 +231,7 @@ static int ipath_write_umem32(struct ipa
 		writel(data, reg_addr);
 
 		reg_addr++;
-		uaddr++;
+		uaddr += sizeof(u32);
 	}
 	ret = 0;
 bail:
#50 lore raw

PATCH 1/1 ipath - remember to drop spinlock

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_rc.c +3 -1
diff -r 5b565c24d62a -r 160a111381ae drivers/infiniband/hw/ipath/ipath_rc.c
@@ -1505,8 +1505,10 @@ void ipath_rc_rcv(struct ipath_ibdev *de
 			ok = ipath_rkey_ok(dev, &qp->s_rdma_sge,
 					   qp->s_rdma_len, vaddr, rkey,
 					   IB_ACCESS_REMOTE_READ);
-			if (unlikely(!ok))
+			if (unlikely(!ok)) {
+				spin_unlock_irq(&qp->s_lock);
 				goto nack_acc;
+			}
 			/*
 			 * Update the next expected PSN.  We add 1 later
 			 * below, so only add the remainder here.
#51 lore raw

PATCH 1/1 ipath - remove some duplicated lines of code

In reply to Bryan O'Sullivan

Cosmetic fixes.

drivers/infiniband/hw/ipath/ipath_ht400.c +0 -1
diff -r e468ad0bd83e -r 2b7918a7133e drivers/infiniband/hw/ipath/ipath_ht400.c
@@ -1555,7 +1555,6 @@ void ipath_init_ht400_funcs(struct ipath
 	dd->ipath_f_reset = ipath_setup_ht_reset;
 	dd->ipath_f_get_boardname = ipath_ht_boardname;
 	dd->ipath_f_init_hwerrors = ipath_ht_init_hwerrors;
-	dd->ipath_f_init_hwerrors = ipath_ht_init_hwerrors;
 	dd->ipath_f_early_init = ipath_ht_early_init;
 	dd->ipath_f_handle_hwerrors = ipath_ht_handle_hwerrors;
 	dd->ipath_f_quiet_serdes = ipath_ht_quiet_serdes;
drivers/infiniband/hw/ipath/ipath_qp.c +0 -3
diff -r e468ad0bd83e -r 2b7918a7133e drivers/infiniband/hw/ipath/ipath_qp.c
@@ -513,9 +513,6 @@ int ipath_modify_qp(struct ib_qp *ibqp, 
 	if (attr_mask & IB_QP_QKEY)
 		qp->qkey = attr->qkey;
 
-	if (attr_mask & IB_QP_PKEY_INDEX)
-		qp->s_pkey_index = attr->pkey_index;
-
 	qp->state = new_state;
 	spin_unlock_irqrestore(&qp->s_lock, flags);
 
#52 lore raw

PATCH 1/1 ipath - treat PE800 rev1 and rev2 as similar

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_pe800.c +1 -1
diff -r 2b7918a7133e -r 8e2d63833cf2 drivers/infiniband/hw/ipath/ipath_pe800.c
@@ -532,7 +532,7 @@ static int ipath_pe_boardname(struct ipa
 	if (n)
 		snprintf(name, namelen, "%s", n);
 
-	if (dd->ipath_majrev != 4 || dd->ipath_minrev != 1) {
+	if (dd->ipath_majrev != 4 || !dd->ipath_minrev || dd->ipath_minrev>2) {
 		ipath_dev_err(dd, "Unsupported PE-800 revision %u.%u!\n",
 			      dd->ipath_majrev, dd->ipath_minrev);
 		ret = 1;
#53 lore raw

PATCH 1/1 ipath - use phys_to_virt instead of bus_to_virt

In reply to Bryan O'Sullivan

I think Roland already has this patch.

drivers/infiniband/hw/ipath/ipath_keys.c +2 -2
diff -r 201654fe1962 -r 4e0a07d20868 drivers/infiniband/hw/ipath/ipath_keys.c
@@ -126,11 +126,11 @@ int ipath_lkey_ok(struct ipath_lkey_tabl
 	/*
 	 * We use LKEY == zero to mean a physical kmalloc() address.
 	 * This is a bit of a hack since we rely on dma_map_single()
-	 * being reversible by calling bus_to_virt().
+	 * being reversible by calling phys_to_virt().
 	 */
 	if (sge->lkey == 0) {
 		isge->mr = NULL;
-		isge->vaddr = bus_to_virt(sge->addr);
+		isge->vaddr = phys_to_virt(sge->addr);
 		isge->length = sge->length;
 		isge->sge_length = sge->length;
 		ret = 1;
#54 lore raw

Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

drivers/infiniband/hw/ipath/ipath_keys.c int ipath_lkey_ok(struct ipath_lkey_tabl · line 129

Quoted 5 lines

I think Roland already has this patch.

drivers/infiniband/hw/ipath/ipath_keys.c +1 -1
 	 * This is a bit of a hack since we rely on dma_map_single()
-	 * being reversible by calling bus_to_virt().
+	 * being reversible by calling phys_to_virt().

Actually I NAK'ed this patch. It compiles the same thing on x86_64 but makes the source code wrong -- dma_map_single() returns a bus address, not a physical address.

- R.

#55 lore raw

Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Quoted 3 lines · Roland Dreier

Actually I NAK'ed this patch. It compiles the same thing on x86_64 but makes the source code wrong -- dma_map_single() returns a bus address, not a physical address.

As Segher mentioned, bus_to_virt is unportable, so it's definitely the wrong thing to use.

I don't recall what you suggested instead, but I seem to recall that the discussion kind of went "oh, right, the layering is all broken".

Any ideas? Should this turn from a one-liner into a big-refactor-for-2.6.18 patch?

	<b
#56 lore raw

Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Bryan> As Segher mentioned, bus_to_virt is unportable, so it's Bryan> definitely the wrong thing to use.

Yes, but at least it says what you're trying to do. asm-powerpc's io.h has this for phys_to_virt:

* This function does not handle bus mappings for DMA transfers. In
* almost all conceivable cases a device driver should not be using
* this function

so replacing bus_to_virt with that is not a step forward.

Bryan> Any ideas? Should this turn from a one-liner into a Bryan> big-refactor-for-2.6.18 patch?

I don't think there's a quick way to fix this. What you really want to do is override the DMA mapping functions for your device so that you can keep track of the kernel mapping. powerpc can already do this (cf the ehca driver), and I think patches to do it on x86-64 are floating around as part of the "Calgary IOMMU" work.

- R.

#57 lore raw

Re: [openib-general] Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Quoted 6 lines · Roland Dreier

Bryan> Any ideas? Should this turn from a one-liner into a Bryan> big-refactor-for-2.6.18 patch?

I don't think there's a quick way to fix this. What you really want to do is override the DMA mapping functions for your device so that you can keep track of the kernel mapping.

Or figure out which openib.org interface has to change so the original virt addresses that were registered/handed to the ULP are passed down to the low level interface driver too. Seems like a more obvious way to fix the problem. Someone did suggest this already, right?

Quoted 2 lines

(cf the ehca driver), and I think patches to do it on x86-64 are floating around as part of the "Calgary IOMMU" work.

parisc has been using dma_ops for several years. I don't expect dma_ops to become part of generic code. DMA support is inherently arch specific. Because of that, I don't look forward to a low level device driver that is mucking with dma_ops.

hth,
grant

#58 lore raw

Re: [openib-general] Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Grant> Or figure out which openib.org interface has to change so Grant> the original virt addresses that were registered/handed to Grant> the ULP are passed down to the low level interface driver Grant> too. Seems like a more obvious way to fix the problem. Grant> Someone did suggest this already, right?

It's been suggested many times, but no one ever comes up with a way to handle the fact that RDMA means that addresses come from remote systems as well as being passed in through an API.

- R.

#59 lore raw

Re: [openib-general] Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Quoted 9 lines · Roland Dreier

Grant> Or figure out which openib.org interface has to change so Grant> the original virt addresses that were registered/handed to Grant> the ULP are passed down to the low level interface driver Grant> too. Seems like a more obvious way to fix the problem. Grant> Someone did suggest this already, right?

It's been suggested many times, but no one ever comes up with a way to handle the fact that RDMA means that addresses come from remote systems as well as being passed in through an API.

Aren't remote addresses handled differently than local ones? ULP has to map local addresses.
We can't map remote ones (remote host maps it).
The ULP must know the difference and can tell the lower level driver which is which.

Sorry, I hope my ignorance of RDMA isn't getting in the way again.

thanks,
grant

#60 lore raw

Re: [openib-general] Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

Grant> Aren't remote addresses handled differently than local Grant> ones? ULP has to map local addresses. We can't map remote Grant> ones (remote host maps it). The ULP must know the Grant> difference and can tell the lower level driver which is Grant> which.

The problem is that RDMA requests have to be handled by the low-level driver (or hardware) without any ULP involvement. So every device has to handle getting messages like "send me XXX bytes of data from address YYY in the memory region corresponding to R_Key ZZZ."

- R.

#61 lore raw

Re: [PATCH 21 of 53] ipath - use phys_to_virt instead of bus_to_virt

In reply to Bryan O'Sullivan
Quoted 8 lines · Bryan O'Sullivan
Quoted 3 lines · Roland Dreier

Actually I NAK'ed this patch. It compiles the same thing on x86_64 but makes the source code wrong -- dma_map_single() returns a bus address, not a physical address.

As Segher mentioned, bus_to_virt is unportable, so it's definitely the wrong thing to use.

phys_to_virt is as bad. please fix your code to do the right thing, that is to stop pretending to be able to map back from a bus to a virtual address. The only way to get at the virtual address from a bus one is to store it away at the time you call the dma mapping function.

#62 lore raw

PATCH 1/1 ipath - remove redundant register read

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_intr.c +0 -1
diff -r 47f1df66d097 -r 23519e578bf0 drivers/infiniband/hw/ipath/ipath_intr.c
@@ -780,7 +780,6 @@ irqreturn_t ipath_intr(int irq, void *da
 			ipath_stats.sps_fastrcvint++;
 			goto done;
 		}
-		istat = ipath_read_kreg32(dd, dd->ipath_kregs->kr_intstatus);
 	}
 
 	istat = ipath_read_kreg32(dd, dd->ipath_kregs->kr_intstatus);
#63 lore raw

PATCH 1/1 ipath - replace uses of LIST_POISON

In reply to Bryan O'Sullivan

Per Andrew's request.

drivers/infiniband/hw/ipath/ipath_qp.c +14 -16
diff -r df954e47ff67 -r 947e92f4b370 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -375,10 +375,10 @@ static void ipath_error_qp(struct ipath_
 
 	spin_lock(&dev->pending_lock);
 	/* XXX What if its already removed by the timeout code? */
-	if (qp->timerwait.next != LIST_POISON1)
-		list_del(&qp->timerwait);
-	if (qp->piowait.next != LIST_POISON1)
-		list_del(&qp->piowait);
+	if (!list_empty(&qp->timerwait))
+		list_del_init(&qp->timerwait);
+	if (!list_empty(&qp->piowait))
+		list_del_init(&qp->piowait);
 	spin_unlock(&dev->pending_lock);
 
 	wc.status = IB_WC_WR_FLUSH_ERR;
@@ -722,10 +722,8 @@ struct ib_qp *ipath_create_qp(struct ib_
 			     init_attr->qp_type == IB_QPT_RC ?
 			     ipath_do_rc_send : ipath_do_uc_send,
 			     (unsigned long)qp);
-		qp->piowait.next = LIST_POISON1;
-		qp->piowait.prev = LIST_POISON2;
-		qp->timerwait.next = LIST_POISON1;
-		qp->timerwait.prev = LIST_POISON2;
+		INIT_LIST_HEAD(&qp->piowait);
+		INIT_LIST_HEAD(&qp->timerwait);
 		qp->state = IB_QPS_RESET;
 		qp->s_wq = swq;
 		qp->s_size = init_attr->cap.max_send_wr + 1;
@@ -795,10 +793,10 @@ int ipath_destroy_qp(struct ib_qp *ibqp)
 
 	/* Make sure the QP isn't on the timeout list. */
 	spin_lock_irqsave(&dev->pending_lock, flags);
-	if (qp->timerwait.next != LIST_POISON1)
-		list_del(&qp->timerwait);
-	if (qp->piowait.next != LIST_POISON1)
-		list_del(&qp->piowait);
+	if (!list_empty(&qp->timerwait))
+		list_del_init(&qp->timerwait);
+	if (!list_empty(&qp->piowait))
+		list_del_init(&qp->piowait);
 	spin_unlock_irqrestore(&dev->pending_lock, flags);
 
 	/*
@@ -867,10 +865,10 @@ void ipath_sqerror_qp(struct ipath_qp *q
 
 	spin_lock(&dev->pending_lock);
 	/* XXX What if its already removed by the timeout code? */
-	if (qp->timerwait.next != LIST_POISON1)
-		list_del(&qp->timerwait);
-	if (qp->piowait.next != LIST_POISON1)
-		list_del(&qp->piowait);
+	if (!list_empty(&qp->timerwait))
+		list_del_init(&qp->timerwait);
+	if (!list_empty(&qp->piowait))
+		list_del_init(&qp->piowait);
 	spin_unlock(&dev->pending_lock);
 
 	ipath_cq_enter(to_icq(qp->ibqp.send_cq), wc, 1);
drivers/infiniband/hw/ipath/ipath_rc.c +7 -8
diff -r df954e47ff67 -r 947e92f4b370 drivers/infiniband/hw/ipath/ipath_rc.c
@@ -57,7 +57,7 @@ static void ipath_init_restart(struct ip
 	qp->s_len = wqe->length - len;
 	dev = to_idev(qp->ibqp.device);
 	spin_lock(&dev->pending_lock);
-	if (qp->timerwait.next == LIST_POISON1)
+	if (list_empty(&qp->timerwait))
 		list_add_tail(&qp->timerwait,
 			      &dev->pending[dev->pending_index]);
 	spin_unlock(&dev->pending_lock);
@@ -356,7 +356,7 @@ static inline int ipath_make_rc_req(stru
 		if ((int)(qp->s_psn - qp->s_next_psn) > 0)
 			qp->s_next_psn = qp->s_psn;
 		spin_lock(&dev->pending_lock);
-		if (qp->timerwait.next == LIST_POISON1)
+		if (list_empty(&qp->timerwait))
 			list_add_tail(&qp->timerwait,
 				      &dev->pending[dev->pending_index]);
 		spin_unlock(&dev->pending_lock);
@@ -726,8 +726,8 @@ void ipath_restart_rc(struct ipath_qp *q
 	 */
 	dev = to_idev(qp->ibqp.device);
 	spin_lock(&dev->pending_lock);
-	if (qp->timerwait.next != LIST_POISON1)
-		list_del(&qp->timerwait);
+	if (!list_empty(&qp->timerwait))
+		list_del_init(&qp->timerwait);
 	spin_unlock(&dev->pending_lock);
 
 	if (wqe->wr.opcode == IB_WR_RDMA_READ)
@@ -886,8 +886,8 @@ static int do_rc_ack(struct ipath_qp *qp
 	 * just won't find anything to restart if we ACK everything.
 	 */
 	spin_lock(&dev->pending_lock);
-	if (qp->timerwait.next != LIST_POISON1)
-		list_del(&qp->timerwait);
+	if (!list_empty(&qp->timerwait))
+		list_del_init(&qp->timerwait);
 	spin_unlock(&dev->pending_lock);
 
 	/*
@@ -1194,8 +1194,7 @@ static inline void ipath_rc_rcv_resp(str
 		     IB_WR_RDMA_READ))
 		goto ack_done;
 	spin_lock(&dev->pending_lock);
-	if (qp->s_rnr_timeout == 0 &&
-	    qp->timerwait.next != LIST_POISON1)
+	if (qp->s_rnr_timeout == 0 && !list_empty(&qp->timerwait))
 		list_move_tail(&qp->timerwait,
 			       &dev->pending[dev->pending_index]);
 	spin_unlock(&dev->pending_lock);
drivers/infiniband/hw/ipath/ipath_ruc.c +1 -1
diff -r df954e47ff67 -r 947e92f4b370 drivers/infiniband/hw/ipath/ipath_ruc.c
@@ -435,7 +435,7 @@ void ipath_no_bufs_available(struct ipat
 	unsigned long flags;
 
 	spin_lock_irqsave(&dev->pending_lock, flags);
-	if (qp->piowait.next == LIST_POISON1)
+	if (list_empty(&qp->piowait))
 		list_add_tail(&qp->piowait, &dev->piowait);
 	spin_unlock_irqrestore(&dev->pending_lock, flags);
 	/*
drivers/infiniband/hw/ipath/ipath_verbs.c +3 -3
diff -r df954e47ff67 -r 947e92f4b370 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -517,7 +517,7 @@ static void ipath_ib_timer(void *arg)
 	last = &dev->pending[dev->pending_index];
 	while (!list_empty(last)) {
 		qp = list_entry(last->next, struct ipath_qp, timerwait);
-		list_del(&qp->timerwait);
+		list_del_init(&qp->timerwait);
 		qp->timer_next = resend;
 		resend = qp;
 		atomic_inc(&qp->refcount);
@@ -527,7 +527,7 @@ static void ipath_ib_timer(void *arg)
 		qp = list_entry(last->next, struct ipath_qp, timerwait);
 		if (--qp->s_rnr_timeout == 0) {
 			do {
-				list_del(&qp->timerwait);
+				list_del_init(&qp->timerwait);
 				tasklet_hi_schedule(&qp->s_task);
 				if (list_empty(last))
 					break;
@@ -607,7 +607,7 @@ static int ipath_ib_piobufavail(void *ar
 	while (!list_empty(&dev->piowait)) {
 		qp = list_entry(dev->piowait.next, struct ipath_qp,
 				piowait);
-		list_del(&qp->piowait);
+		list_del_init(&qp->piowait);
 		tasklet_hi_schedule(&qp->s_task);
 	}
 	spin_unlock_irqrestore(&dev->pending_lock, flags);
#64 lore raw

PATCH 1/1 ipath - count dropped VL15 packets

In reply to Bryan O'Sullivan

We need to count these for IB conformance.

drivers/infiniband/hw/ipath/ipath_mad.c +8 -0
diff -r 8b882bb46a32 -r e468ad0bd83e drivers/infiniband/hw/ipath/ipath_mad.c
@@ -646,6 +646,7 @@ struct ib_pma_portcounters {
 #define IB_PMA_SEL_PORT_RCV_ERRORS		__constant_htons(0x0008)
 #define IB_PMA_SEL_PORT_RCV_REMPHYS_ERRORS	__constant_htons(0x0010)
 #define IB_PMA_SEL_PORT_XMIT_DISCARDS		__constant_htons(0x0040)
+#define IB_PMA_SEL_PORT_VL15_DROPPED		__constant_htons(0x0800)
 #define IB_PMA_SEL_PORT_XMIT_DATA		__constant_htons(0x1000)
 #define IB_PMA_SEL_PORT_RCV_DATA		__constant_htons(0x2000)
 #define IB_PMA_SEL_PORT_XMIT_PACKETS		__constant_htons(0x4000)
@@ -929,6 +930,10 @@ static int recv_pma_get_portcounters(str
 	else
 		p->port_xmit_discards =
 			cpu_to_be16((u16)cntrs.port_xmit_discards);
+	if (dev->n_vl15_dropped > 0xFFFFUL)
+		p->vl15_dropped = __constant_cpu_to_be16(0xFFFF);
+	else
+		p->vl15_dropped = cpu_to_be16((u16)dev->n_vl15_dropped);
 	if (cntrs.port_xmit_data > 0xFFFFFFFFUL)
 		p->port_xmit_data = __constant_cpu_to_be32(0xFFFFFFFF);
 	else
@@ -1022,6 +1027,9 @@ static int recv_pma_set_portcounters(str
 
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DISCARDS)
 		dev->n_port_xmit_discards = cntrs.port_xmit_discards;
+
+	if (p->counter_select & IB_PMA_SEL_PORT_VL15_DROPPED)
+		dev->n_vl15_dropped = 0;
 
 	if (p->counter_select & IB_PMA_SEL_PORT_XMIT_DATA)
 		dev->n_port_xmit_data = cntrs.port_xmit_data;
drivers/infiniband/hw/ipath/ipath_ud.c +5 -1
diff -r 8b882bb46a32 -r e468ad0bd83e drivers/infiniband/hw/ipath/ipath_ud.c
@@ -554,7 +554,11 @@ void ipath_ud_rcv(struct ipath_ibdev *de
 	spin_lock_irqsave(&rq->lock, flags);
 	if (rq->tail == rq->head) {
 		spin_unlock_irqrestore(&rq->lock, flags);
-		dev->n_pkt_drops++;
+		/* Count VL15 packets dropped due to no receive buffer */
+		if (qp->ibqp.qp_num == 0)
+			dev->n_vl15_dropped++;
+		else
+			dev->n_pkt_drops++;
 		goto bail;
 	}
 	/* Silently drop packets which are too big. */
drivers/infiniband/hw/ipath/ipath_verbs.h +1 -0
diff -r 8b882bb46a32 -r e468ad0bd83e drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -468,6 +468,7 @@ struct ipath_ibdev {
 	u32 n_other_naks;
 	u32 n_timeouts;
 	u32 n_pkt_drops;
+	u32 n_vl15_dropped;
 	u32 n_wqe_errs;
 	u32 n_rdma_dup_busy;
 	u32 n_piowait;
#65 lore raw

PATCH 1/1 ipath - forbid sending of bad packet sizes

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_ud.c +5 -0
diff -r b098b021b6fd -r 4868daa7f215 drivers/infiniband/hw/ipath/ipath_ud.c
@@ -273,6 +273,11 @@ int ipath_post_ud_send(struct ipath_qp *
 		}
 		len += wr->sg_list[i].length;
 		ss.num_sge++;
+	}
+	/* Check for invalid packet size. */
+	if (len > ipath_layer_get_ibmtu(dev->dd)) {
+		ret = -EINVAL;
+		goto bail;
 	}
 	extra_bytes = (4 - len) & 3;
 	nwords = (len + extra_bytes) >> 2;
#66 lore raw

PATCH 1/1 ipath - fix "many lost ticks" warning

In reply to Bryan O'Sullivan

Don't disable interrupts for long, or the kernel gets shirty.

drivers/infiniband/hw/ipath/ipath_keys.c +30 -24
diff -r 4e0a07d20868 -r 1887e7b3e2a3 drivers/infiniband/hw/ipath/ipath_keys.c
@@ -120,6 +120,7 @@ int ipath_lkey_ok(struct ipath_lkey_tabl
 		  struct ib_sge *sge, int acc)
 {
 	struct ipath_mregion *mr;
+	unsigned n, m;
 	size_t off;
 	int ret;
 
@@ -151,20 +152,22 @@ int ipath_lkey_ok(struct ipath_lkey_tabl
 	}
 
 	off += mr->offset;
+	m = 0;
+	n = 0;
+	while (off >= mr->map[m]->segs[n].length) {
+		off -= mr->map[m]->segs[n].length;
+		n++;
+		if (n >= IPATH_SEGSZ) {
+			m++;
+			n = 0;
+		}
+	}
 	isge->mr = mr;
-	isge->m = 0;
-	isge->n = 0;
-	while (off >= mr->map[isge->m]->segs[isge->n].length) {
-		off -= mr->map[isge->m]->segs[isge->n].length;
-		isge->n++;
-		if (isge->n >= IPATH_SEGSZ) {
-			isge->m++;
-			isge->n = 0;
-		}
-	}
-	isge->vaddr = mr->map[isge->m]->segs[isge->n].vaddr + off;
-	isge->length = mr->map[isge->m]->segs[isge->n].length - off;
+	isge->vaddr = mr->map[m]->segs[n].vaddr + off;
+	isge->length = mr->map[m]->segs[n].length - off;
 	isge->sge_length = sge->length;
+	isge->m = m;
+	isge->n = n;
 
 	ret = 1;
 
@@ -189,6 +192,7 @@ int ipath_rkey_ok(struct ipath_ibdev *de
 	struct ipath_lkey_table *rkt = &dev->lk_table;
 	struct ipath_sge *sge = &ss->sge;
 	struct ipath_mregion *mr;
+	unsigned n, m;
 	size_t off;
 	int ret;
 
@@ -206,20 +210,22 @@ int ipath_rkey_ok(struct ipath_ibdev *de
 	}
 
 	off += mr->offset;
+	m = 0;
+	n = 0;
+	while (off >= mr->map[m]->segs[n].length) {
+		off -= mr->map[m]->segs[n].length;
+		n++;
+		if (n >= IPATH_SEGSZ) {
+			m++;
+			n = 0;
+		}
+	}
 	sge->mr = mr;
-	sge->m = 0;
-	sge->n = 0;
-	while (off >= mr->map[sge->m]->segs[sge->n].length) {
-		off -= mr->map[sge->m]->segs[sge->n].length;
-		sge->n++;
-		if (sge->n >= IPATH_SEGSZ) {
-			sge->m++;
-			sge->n = 0;
-		}
-	}
-	sge->vaddr = mr->map[sge->m]->segs[sge->n].vaddr + off;
-	sge->length = mr->map[sge->m]->segs[sge->n].length - off;
+	sge->vaddr = mr->map[m]->segs[n].vaddr + off;
+	sge->length = mr->map[m]->segs[n].length - off;
 	sge->sge_length = len;
+	sge->m = m;
+	sge->n = n;
 	ss->sg_list = NULL;
 	ss->num_sge = 1;
 
drivers/infiniband/hw/ipath/ipath_qp.c +19 -32

121 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_rc.c +140 -194

567 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_ruc.c +12 -15
diff -r 4e0a07d20868 -r 1887e7b3e2a3 drivers/infiniband/hw/ipath/ipath_ruc.c
@@ -112,10 +112,11 @@ void ipath_insert_rnr_queue(struct ipath
  *
  * Return 0 if no RWQE is available, otherwise return 1.
  *
- * Called at interrupt level with the QP r_rq.lock held.
+ * Can be called from interrupt level.
  */
 int ipath_get_rwqe(struct ipath_qp *qp, int wr_id_only)
 {
+	unsigned long flags;
 	struct ipath_rq *rq;
 	struct ipath_srq *srq;
 	struct ipath_rwqe *wqe;
@@ -123,6 +124,8 @@ int ipath_get_rwqe(struct ipath_qp *qp, 
 
 	if (!qp->ibqp.srq) {
 		rq = &qp->r_rq;
+		spin_lock_irqsave(&rq->lock, flags);
+
 		if (unlikely(rq->tail == rq->head)) {
 			ret = 0;
 			goto bail;
@@ -137,15 +140,14 @@ int ipath_get_rwqe(struct ipath_qp *qp, 
 		}
 		if (++rq->tail >= rq->size)
 			rq->tail = 0;
-		ret = 1;
-		goto bail;
+		goto done;
 	}
 
 	srq = to_isrq(qp->ibqp.srq);
 	rq = &srq->rq;
-	spin_lock(&rq->lock);
+	spin_lock_irqsave(&rq->lock, flags);
+
 	if (unlikely(rq->tail == rq->head)) {
-		spin_unlock(&rq->lock);
 		ret = 0;
 		goto bail;
 	}
@@ -175,13 +177,13 @@ int ipath_get_rwqe(struct ipath_qp *qp, 
 			ev.event = IB_EVENT_SRQ_LIMIT_REACHED;
 			srq->ibsrq.event_handler(&ev,
 						 srq->ibsrq.srq_context);
-		} else
-			spin_unlock(&rq->lock);
-	} else
-		spin_unlock(&rq->lock);
+		}
+	}
+done:
 	ret = 1;
 
 bail:
+	spin_unlock_irqrestore(&rq->lock, flags);
 	return ret;
 }
 
@@ -247,10 +249,8 @@ again:
 		wc.imm_data = wqe->wr.imm_data;
 		/* FALLTHROUGH */
 	case IB_WR_SEND:
-		spin_lock_irqsave(&qp->r_rq.lock, flags);
 		if (!ipath_get_rwqe(qp, 0)) {
 		rnr_nak:
-			spin_unlock_irqrestore(&qp->r_rq.lock, flags);
 			/* Handle RNR NAK */
 			if (qp->ibqp.qp_type == IB_QPT_UC)
 				goto send_comp;
@@ -262,20 +262,17 @@ again:
 				sqp->s_rnr_retry--;
 			dev->n_rnr_naks++;
 			sqp->s_rnr_timeout =
-				ib_ipath_rnr_table[sqp->s_min_rnr_timer];
+				ib_ipath_rnr_table[sqp->r_min_rnr_timer];
 			ipath_insert_rnr_queue(sqp);
 			goto done;
 		}
-		spin_unlock_irqrestore(&qp->r_rq.lock, flags);
 		break;
 
 	case IB_WR_RDMA_WRITE_WITH_IMM:
 		wc.wc_flags = IB_WC_WITH_IMM;
 		wc.imm_data = wqe->wr.imm_data;
-		spin_lock_irqsave(&qp->r_rq.lock, flags);
 		if (!ipath_get_rwqe(qp, 1))
 			goto rnr_nak;
-		spin_unlock_irqrestore(&qp->r_rq.lock, flags);
 		/* FALLTHROUGH */
 	case IB_WR_RDMA_WRITE:
 		if (wqe->length == 0)
drivers/infiniband/hw/ipath/ipath_uc.c +1 -8
diff -r 4e0a07d20868 -r 1887e7b3e2a3 drivers/infiniband/hw/ipath/ipath_uc.c
@@ -240,7 +240,6 @@ void ipath_uc_rcv(struct ipath_ibdev *de
 	u32 hdrsize;
 	u32 psn;
 	u32 pad;
-	unsigned long flags;
 	struct ib_wc wc;
 	u32 pmtu = ib_mtu_enum_to_int(qp->path_mtu);
 	struct ib_reth *reth;
@@ -278,8 +277,6 @@ void ipath_uc_rcv(struct ipath_ibdev *de
 	wc.imm_data = 0;
 	wc.wc_flags = 0;
 
-	spin_lock_irqsave(&qp->r_rq.lock, flags);
-
 	/* Compare the PSN verses the expected PSN. */
 	if (unlikely(ipath_cmp24(psn, qp->r_psn) != 0)) {
 		/*
@@ -536,15 +533,11 @@ void ipath_uc_rcv(struct ipath_ibdev *de
 
 	default:
 		/* Drop packet for unknown opcodes. */
-		spin_unlock_irqrestore(&qp->r_rq.lock, flags);
 		dev->n_pkt_drops++;
-		goto bail;
+		goto done;
 	}
 	qp->r_psn++;
 	qp->r_state = opcode;
 done:
-	spin_unlock_irqrestore(&qp->r_rq.lock, flags);
-
-bail:
 	return;
 }
drivers/infiniband/hw/ipath/ipath_verbs.h +7 -6
diff -r 4e0a07d20868 -r 1887e7b3e2a3 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -306,32 +306,33 @@ struct ipath_qp {
 	u32 s_next_psn;		/* PSN for next request */
 	u32 s_last_psn;		/* last response PSN processed */
 	u32 s_psn;		/* current packet sequence number */
+	u32 s_ack_psn;		/* PSN for RDMA_READ */
 	u32 s_rnr_timeout;	/* number of milliseconds for RNR timeout */
-	u32 s_ack_psn;		/* PSN for next ACK or RDMA_READ */
-	u64 s_ack_atomic;	/* data for atomic ACK */
+	u32 r_ack_psn;		/* PSN for next ACK or atomic ACK */
 	u64 r_wr_id;		/* ID for current receive WQE */
 	u64 r_atomic_data;	/* data for last atomic op */
 	u32 r_atomic_psn;	/* PSN of last atomic op */
 	u32 r_len;		/* total length of r_sge */
 	u32 r_rcv_len;		/* receive data len processed */
 	u32 r_psn;		/* expected rcv packet sequence number */
+	u32 r_msn;		/* message sequence number */
 	u8 state;		/* QP state */
 	u8 s_state;		/* opcode of last packet sent */
 	u8 s_ack_state;		/* opcode of packet to ACK */
-	u8 s_nak_state;		/* non-zero if NAK is pending */
 	u8 r_state;		/* opcode of last packet received */
+	u8 r_ack_state;		/* opcode of packet to ACK */
+	u8 r_nak_state;		/* non-zero if NAK is pending */
+	u8 r_min_rnr_timer;	/* retry timeout value for RNR NAKs */
 	u8 r_reuse_sge;		/* for UC receive errors */
 	u8 r_sge_inx;		/* current index into sg_list */
+	u8 qp_access_flags;
 	u8 s_max_sge;		/* size of s_wq->sg_list */
-	u8 qp_access_flags;
 	u8 s_retry_cnt;		/* number of times to retry */
 	u8 s_rnr_retry_cnt;
-	u8 s_min_rnr_timer;
 	u8 s_retry;		/* requester retry counter */
 	u8 s_rnr_retry;		/* requester RNR retry counter */
 	u8 s_pkey_index;	/* PKEY index to use */
 	enum ib_mtu path_mtu;
-	atomic_t msn;		/* message sequence number */
 	u32 remote_qpn;
 	u32 qkey;		/* QKEY for this QP (for UD or RD) */
 	u32 s_size;		/* send work queue size */
#67 lore raw

PATCH 1/1 ipath - forbid setting of invalid MLID

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_sysfs.c +1 -1
diff -r 551966b88d7c -r 47f1df66d097 drivers/infiniband/hw/ipath/ipath_sysfs.c
@@ -221,7 +221,7 @@ static ssize_t store_mlid(struct device 
 	int ret;
 
 	ret = ipath_parse_ushort(buf, &mlid);
-	if (ret < 0)
+	if (ret < 0 || mlid < 0xc000)
 		goto invalid;
 
 	unit = dd->ipath_unit;
#68 lore raw

PATCH 1/1 ipath - fix occasional hangs in SDP

In reply to Bryan O'Sullivan

We were updating the head register multiple times in the rcvhdrq processing loop, and setting the counter on each update. Since that meant that the tail register was ahead of head for all but the last update,
we would get extra interrupts. The fix was to not write the counter
value except on the last update.

drivers/infiniband/hw/ipath/ipath_driver.c +22 -11
diff -r 5ddaf7c07cdf -r 09077b2f476f drivers/infiniband/hw/ipath/ipath_driver.c
@@ -918,7 +918,7 @@ void ipath_kreceive(struct ipath_devdata
 	const u32 maxcnt = dd->ipath_rcvhdrcnt * rsize;	/* words */
 	u32 etail = -1, l, hdrqtail;
 	struct ips_message_header *hdr;
-	u32 eflags, i, etype, tlen, pkttot = 0;
+	u32 eflags, i, etype, tlen, pkttot = 0, updegr=0;
 	static u64 totcalls;	/* stats, may eventually remove */
 	char emsg[128];
 
@@ -932,14 +932,14 @@ void ipath_kreceive(struct ipath_devdata
 	if (test_and_set_bit(0, &dd->ipath_rcv_pending))
 		goto bail;
 
-	if (dd->ipath_port0head ==
-	    (u32)le64_to_cpu(*dd->ipath_hdrqtailptr))
+	l = dd->ipath_port0head;
+	if(l == (u32)le64_to_cpu(*dd->ipath_hdrqtailptr))
 		goto done;
 
 	/* read only once at start for performance */
 	hdrqtail = (u32)le64_to_cpu(*dd->ipath_hdrqtailptr);
 
-	for (i = 0, l = dd->ipath_port0head; l != hdrqtail; i++) {
+	for (i = 0; l != hdrqtail; i++) {
 		u32 qp;
 		u8 *bthbytes;
 
@@ -1050,15 +1050,26 @@ void ipath_kreceive(struct ipath_devdata
 		l += rsize;
 		if (l >= maxcnt)
 			l = 0;
+		if (etype != RCVHQ_RCV_TYPE_EXPECTED)
+		    updegr = 1;
 		/*
-		 * update for each packet, to help prevent overflows if we
-		 * have lots of packets.
+		 * update head regs on last packet, and every 16 packets.
+		 * Reduce bus traffic, while still trying to prevent
+		 * rcvhdrq overflows, for when the queue is nearly full
 		 */
-		(void)ipath_write_ureg(dd, ur_rcvhdrhead,
-				       dd->ipath_rhdrhead_intr_off | l, 0);
-		if (etype != RCVHQ_RCV_TYPE_EXPECTED)
-			(void)ipath_write_ureg(dd, ur_rcvegrindexhead,
-					       etail, 0);
+		if(l == hdrqtail || (i && !(i&0xf))) {
+			u64 lval;
+			if(l == hdrqtail) /* want interrupt only on last */
+				lval = dd->ipath_rhdrhead_intr_off | l;
+			else
+				lval = l;
+			(void)ipath_write_ureg(dd, ur_rcvhdrhead, lval, 0);
+			if(updegr) {
+				(void)ipath_write_ureg(dd, ur_rcvegrindexhead,
+						       etail, 0);
+				updegr = 0;
+			}
+		}
 	}
 
 	pkttot += i;
drivers/infiniband/hw/ipath/ipath_intr.c +26 -25

113 lines not on this page · raw message

#69 lore raw

PATCH 1/1 ipath - more sharing between RC and UC code

In reply to Bryan O'Sullivan

Share more common code between RC and UC protocols.

drivers/infiniband/hw/ipath/ipath_qp.c +3 -5
diff -r 947e92f4b370 -r 201654fe1962 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -718,9 +718,7 @@ struct ib_qp *ipath_create_qp(struct ib_
 		spin_lock_init(&qp->r_rq.lock);
 		atomic_set(&qp->refcount, 0);
 		init_waitqueue_head(&qp->wait);
-		tasklet_init(&qp->s_task,
-			     init_attr->qp_type == IB_QPT_RC ?
-			     ipath_do_rc_send : ipath_do_uc_send,
+		tasklet_init(&qp->s_task, ipath_do_ruc_send,
 			     (unsigned long)qp);
 		INIT_LIST_HEAD(&qp->piowait);
 		INIT_LIST_HEAD(&qp->timerwait);
@@ -905,9 +903,9 @@ void ipath_get_credit(struct ipath_qp *q
 	 * as many packets as we like.  Otherwise, we have to
 	 * honor the credit field.
 	 */
-	if (credit == IPS_AETH_CREDIT_INVAL) {
+	if (credit == IPS_AETH_CREDIT_INVAL)
 		qp->s_lsn = (u32) -1;
-	} else if (qp->s_lsn != (u32) -1) {
+	else if (qp->s_lsn != (u32) -1) {
 		/* Compute new LSN (i.e., MSN + credit) */
 		credit = (aeth + credit_table[credit]) & IPS_MSN_MASK;
 		if (ipath_cmp24(credit, qp->s_lsn) > 0)
drivers/infiniband/hw/ipath/ipath_rc.c +210 -385

864 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_ruc.c +191 -54

333 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_uc.c +32 -127

212 lines not on this page · raw message

drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
diff -r 947e92f4b370 -r 201654fe1962 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -246,7 +246,7 @@ static int ipath_post_send(struct ib_qp 
 		switch (qp->ibqp.qp_type) {
 		case IB_QPT_UC:
 		case IB_QPT_RC:
-			err = ipath_post_rc_send(qp, wr);
+			err = ipath_post_ruc_send(qp, wr);
 			break;
 
 		case IB_QPT_SMI:
drivers/infiniband/hw/ipath/ipath_verbs.h +14 -6
diff -r 947e92f4b370 -r 201654fe1962 drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -585,10 +585,6 @@ void ipath_sqerror_qp(struct ipath_qp *q
 
 void ipath_get_credit(struct ipath_qp *qp, u32 aeth);
 
-void ipath_do_rc_send(unsigned long data);
-
-void ipath_do_uc_send(unsigned long data);
-
 void ipath_cq_enter(struct ipath_cq *cq, struct ib_wc *entry, int sig);
 
 int ipath_rkey_ok(struct ipath_ibdev *dev, struct ipath_sge_state *ss,
@@ -601,7 +597,7 @@ void ipath_copy_sge(struct ipath_sge_sta
 
 void ipath_skip_sge(struct ipath_sge_state *ss, u32 length);
 
-int ipath_post_rc_send(struct ipath_qp *qp, struct ib_send_wr *wr);
+int ipath_post_ruc_send(struct ipath_qp *qp, struct ib_send_wr *wr);
 
 void ipath_uc_rcv(struct ipath_ibdev *dev, struct ipath_ib_header *hdr,
 		  int has_grh, void *data, u32 tlen, struct ipath_qp *qp);
@@ -683,7 +679,19 @@ void ipath_insert_rnr_queue(struct ipath
 
 int ipath_get_rwqe(struct ipath_qp *qp, int wr_id_only);
 
-void ipath_ruc_loopback(struct ipath_qp *sqp, struct ib_wc *wc);
+u32 ipath_make_grh(struct ipath_ibdev *dev, struct ib_grh *hdr,
+		   struct ib_global_route *grh, u32 hwords, u32 nwords);
+
+void ipath_do_ruc_send(unsigned long data);
+
+u32 ipath_make_rc_ack(struct ipath_qp *qp, struct ipath_other_headers *ohdr,
+		      u32 pmtu);
+
+int ipath_make_rc_req(struct ipath_qp *qp, struct ipath_other_headers *ohdr,
+		      u32 pmtu, u32 *bth0p, u32 *bth2p);
+
+int ipath_make_uc_req(struct ipath_qp *qp, struct ipath_other_headers *ohdr,
+		      u32 pmtu, u32 *bth0p, u32 *bth2p);
 
 extern const enum ib_wc_opcode ib_ipath_wc_opcode[];
 
#70 lore raw

PATCH 1/1 ipath - count VL15 packet drops due to bad VL or lack of buffers

In reply to Bryan O'Sullivan
drivers/infiniband/hw/ipath/ipath_ud.c +7 -2
diff -r 23519e578bf0 -r b098b021b6fd drivers/infiniband/hw/ipath/ipath_ud.c
@@ -554,11 +554,16 @@ void ipath_ud_rcv(struct ipath_ibdev *de
 	spin_lock_irqsave(&rq->lock, flags);
 	if (rq->tail == rq->head) {
 		spin_unlock_irqrestore(&rq->lock, flags);
-		/* Count VL15 packets dropped due to no receive buffer */
+		/*
+		 * Count VL15 packets dropped due to no receive buffer.
+		 * Otherwise, count them as buffer overruns since usually,
+		 * the HW will be able to receive packets even if there are
+		 * no QPs with posted receive buffers.
+		 */
 		if (qp->ibqp.qp_num == 0)
 			dev->n_vl15_dropped++;
 		else
-			dev->n_pkt_drops++;
+			dev->rcv_errors++;
 		goto bail;
 	}
 	/* Silently drop packets which are too big. */
#71 lore raw

PATCH 1/1 ipath - make some maximum values more sane

In reply to Bryan O'Sullivan

Increase the limits on some maximum values.

drivers/infiniband/hw/ipath/ipath_verbs.c +4 -4
diff -r 5d9fbba3222e -r 480ceff18a88 drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -64,21 +64,21 @@ module_param_named(max_ahs, ib_ipath_max
 module_param_named(max_ahs, ib_ipath_max_ahs, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_ahs, "Maximum number of address handles to support");
 
-unsigned int ib_ipath_max_cqes = 0xFFFF;
+unsigned int ib_ipath_max_cqes = 0x2FFFF;
 module_param_named(max_cqes, ib_ipath_max_cqes, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_cqes,
 		 "Maximum number of completion queue entries to support");
 
-unsigned int ib_ipath_max_cqs = 0xFFFF;
+unsigned int ib_ipath_max_cqs = 0x1FFFF;
 module_param_named(max_cqs, ib_ipath_max_cqs, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_cqs, "Maximum number of completion queues to support");
 
-unsigned int ib_ipath_max_qp_wrs = 255;
+unsigned int ib_ipath_max_qp_wrs = 0x1FFFF;
 module_param_named(max_qp_wrs, ib_ipath_max_qp_wrs, uint,
 		   S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_qp_wrs, "Maximum number of QP WRs to support");
 
-unsigned int ib_ipath_max_sges = 255;
+unsigned int ib_ipath_max_sges = 0xFF;
 module_param_named(max_sges, ib_ipath_max_sges, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_sges, "Maximum number of SGEs to support");
 
#72 lore raw

Re: [PATCH 15 of 53] ipath - make some maximum values more sane

drivers/infiniband/hw/ipath/ipath_verbs.c module_param_named(max_ahs, ib_ipath_max · line 67

Quoted 2 lines
drivers/infiniband/hw/ipath/ipath_verbs.c +1 -1
-unsigned int ib_ipath_max_cqes = 0xFFFF;
+unsigned int ib_ipath_max_cqes = 0x2FFFF;

You just added this limit in patch 8/53. How about just fixing that patch to do what you want?

- R.

#73 lore raw

Re: [openib-general] [PATCH 15 of 53] ipath - make some maximum values more sane

In reply to Bryan O'Sullivan
Quoted 4 lines · Bryan O'Sullivan

Increase the limits on some maximum values.

I noticed a rdma/message max size limitation of 4096 the last time I ran some dapl tests. Are there plans to increase or did I miss it somewhere in all the patches?

Here are the max values returned from the ipath ibv_query_device:

query_hca: (ver=20401) ep 65535 ep_q 65535 evd 65535 evd_q 65535
query_hca: msg 4096 rdma 4096 iov 255 lmr 65535 rmr 0
query_hca: dto 65535 iov 255 rdma i1,o1

Thanks,

-arlin

#74 lore raw

PATCH 1/1 ipath - fix reporting of driver version to userspace

In reply to Bryan O'Sullivan

Fix the interface version that gets exported to userspace.

drivers/infiniband/hw/ipath/ipath_file_ops.c +1 -1
diff -r 480ceff18a88 -r 176d1f0c26a3 drivers/infiniband/hw/ipath/ipath_file_ops.c
@@ -139,7 +139,7 @@ static int ipath_get_base_info(struct ip
 	kinfo->spi_piosize = dd->ipath_ibmaxlen;
 	kinfo->spi_mtu = dd->ipath_ibmaxlen;	/* maxlen, not ibmtu */
 	kinfo->spi_port = pd->port_port;
-	kinfo->spi_sw_version = IPATH_USER_SWVERSION;
+	kinfo->spi_sw_version = IPATH_KERN_SWVERSION;
 	kinfo->spi_hw_version = dd->ipath_revision;
 
 	if (copy_to_user(ubase, kinfo, sizeof(*kinfo)))
#75 lore raw

PATCH 1/1 ipath - forbid empty MRs

In reply to Bryan O'Sullivan

Don't allow zero-length regions to be created.

drivers/infiniband/hw/ipath/ipath_mr.c +5 -0
diff -r 02a05b853d20 -r 5d9fbba3222e drivers/infiniband/hw/ipath/ipath_mr.c
@@ -168,6 +168,11 @@ struct ib_mr *ipath_reg_user_mr(struct i
 	struct ib_umem_chunk *chunk;
 	int n, m, i;
 	struct ib_mr *ret;
+
+	if (region->length == 0) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
 
 	n = 0;
 	list_for_each_entry(chunk, &region->chunk_list, list)
#76 lore raw

Re: [PATCH 14 of 53] ipath - forbid empty MRs

Quoted 1 line

Don't allow zero-length regions to be created.

Why are zero-length regions forbidden?

- R.

#77 lore raw

Re: [PATCH 14 of 53] ipath - forbid empty MRs

Quoted 3 lines · Roland Dreier
Quoted 1 line

Don't allow zero-length regions to be created.

Why are zero-length regions forbidden?

One of the gen2 basic tests checks for zero-length regions and barfs if someone creates them. There's no language in IBNA that forbids zero-length regions (I'll take a look at the spec itself to be sure), so it's possible that the test is wrong. On the other hand, a zero-length region doesn't seem terribly useful.

	<b
#78 lore raw

PATCH 1/1 ipath - require capabilities when creating a QP

In reply to Bryan O'Sullivan

You have to specify some capabilities when creating a QP.

drivers/infiniband/hw/ipath/ipath_qp.c +8 -0
diff -r a89145f4846c -r 2fea0d127a41 drivers/infiniband/hw/ipath/ipath_qp.c
@@ -667,6 +667,14 @@ struct ib_qp *ipath_create_qp(struct ib_
 		goto bail;
 	}
 
+	if (init_attr->cap.max_send_sge +
+	    init_attr->cap.max_recv_sge +
+	    init_attr->cap.max_send_wr +
+	    init_attr->cap.max_recv_wr == 0) {
+		ret = ERR_PTR(-EINVAL);
+		goto bail;
+	}
+
 	switch (init_attr->qp_type) {
 	case IB_QPT_UC:
 	case IB_QPT_RC:
#79 lore raw

PATCH 1/1 ipath - cap maximum number of AHs

In reply to Bryan O'Sullivan

Cap the maximum number of address handles.

drivers/infiniband/hw/ipath/ipath_verbs.c +17 -2
diff -r def81ab50644 -r e823378bd19c drivers/infiniband/hw/ipath/ipath_verbs.c
@@ -59,6 +59,11 @@ module_param_named(max_pds, ib_ipath_max
 module_param_named(max_pds, ib_ipath_max_pds, uint, S_IWUSR | S_IRUGO);
 MODULE_PARM_DESC(max_pds,
 		 "Maximum number of protection domains to support");
+
+static unsigned int ib_ipath_max_ahs = 0xFFFF;
+module_param_named(max_ahs, ib_ipath_max_ahs, uint, S_IWUSR | S_IRUGO);
+MODULE_PARM_DESC(max_ahs,
+		 "Maximum number of address handles to support");
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("PathScale <[email protected]>");
@@ -592,6 +597,7 @@ static int ipath_query_device(struct ib_
 	props->max_qp_wr = 0xffff;
 	props->max_sge = 255;
 	props->max_cq = 0xffff;
+	props->max_ah = ib_ipath_max_ahs;
 	props->max_cqe = 0xffff;
 	props->max_mr = dev->lk_table.max;
 	props->max_pd = ib_ipath_max_pds;
@@ -764,13 +770,13 @@ static struct ib_pd *ipath_alloc_pd(stru
 		goto bail;
 	}
 
-	dev->n_pds_allocated++;
-
 	pd = kmalloc(sizeof *pd, GFP_KERNEL);
 	if (!pd) {
 		ret = ERR_PTR(-ENOMEM);
 		goto bail;
 	}
+
+	dev->n_pds_allocated++;
 
 	/* ib_alloc_pd() will initialize pd->ibpd. */
 	pd->user = udata != NULL;
@@ -805,6 +811,12 @@ static struct ib_ah *ipath_create_ah(str
 {
 	struct ipath_ah *ah;
 	struct ib_ah *ret;
+	struct ipath_ibdev *dev = to_idev(pd->device);
+
+	if (dev->n_ahs_allocated == ib_ipath_max_ahs) {
+		ret = ERR_PTR(-ENOMEM);
+		goto bail;
+	}
 
 	/* A multicast address requires a GRH (see ch. 8.4.1). */
 	if (ah_attr->dlid >= IPS_MULTICAST_LID_BASE &&
@@ -848,7 +860,10 @@ bail:
  */
 static int ipath_destroy_ah(struct ib_ah *ibah)
 {
+	struct ipath_ibdev *dev = to_idev(ibah->device);
 	struct ipath_ah *ah = to_iah(ibah);
+
+	dev->n_ahs_allocated--;
 
 	kfree(ah);
 
drivers/infiniband/hw/ipath/ipath_verbs.h +1 -0
diff -r def81ab50644 -r e823378bd19c drivers/infiniband/hw/ipath/ipath_verbs.h
@@ -432,6 +432,7 @@ struct ipath_ibdev {
 	__be64 gid_prefix;	/* in network order */
 	__be64 mkey;
 	u32 n_pds_allocated;	/* number of PDs allocated for device */
+	u32 n_ahs_allocated;	/* number of AHs allocated for device */
 	u64 ipath_sword;	/* total dwords sent (sample result) */
 	u64 ipath_rword;	/* total dwords received (sample result) */
 	u64 ipath_spkts;	/* total packets sent (sample result) */
#80 lore raw
In reply to Bryan O'Sullivan
Quoted 11 lines · Bryan O'Sullivan

Hi, Roland -

Here is a series of patches to bring the ipath driver up to date. I believe you may already have two of them (but I've included them just in case), but the others should all be new.

They apply on top of Linus's current -git.

Cheers,

	<b

Bryan,

I notice there are several patches in 1-53 that appear to be missing, or at least I did not get them on the openib list.

They are: 3, 11, 12, 17, 27, 35, 36, 38, 41, 43, 46, 52.

Roger

#81 lore raw
In reply to Bryan O'Sullivan

Umm... dumping a 53 patch series into the kernel at this stage in the release cycle isn't going to work. You need to sort out the patches that need to go into 2.6.17 from patches that can wait. For example, a 1500+ line patch to factor out common code is clearly not appropriate now. Pretty much the only patches that should be going in now are changes that fix crashes or other serious bugs.

(You can send both sets of patches at the same time -- just let me which ones are for 2.6.17 and which ones can be queued for 2.6.18)

I have some more specific comments in reply to individual patches, although I didn't try to review all 53.

- R.

#82 lore raw
Quoted 7 lines · Roland Dreier

Umm... dumping a 53 patch series into the kernel at this stage in the release cycle isn't going to work. You need to sort out the patches that need to go into 2.6.17 from patches that can wait. For example, a 1500+ line patch to factor out common code is clearly not appropriate now. Pretty much the only patches that should be going in now are changes that fix crashes or other serious bugs.

Quoted 7 lines

(You can send both sets of patches at the same time -- just let me which ones are for 2.6.17 and which ones can be queued for 2.6.18)

I have some more specific comments in reply to individual patches, although I didn't try to review all 53.

- R.

Roland,

What should these patches apply against?

I have tried rc4 and a number of them fail, and I have also tried one of your gits (though maybe not the right one), and at least some of the the same patches seem to fail to apply there.

If I can get an idea what they should apply to I will apply them against that and see how things look.

I have at least one of the nasty bugs that I know about.

Roger

#83 lore raw

Roger> What should these patches apply against?

No idea. Bryan said they apply against Linus's current git, but I didn't actually try.

- R.

#84 lore raw
Quoted 7 lines · Roland Dreier

Roger> What should these patches apply against?

No idea. Bryan said they apply against Linus's current git, but I didn't actually try.

- R.

I checked the rc4 -> git patches (there is only 1 ipath patch in it), and I get a number of patch fails attempting to apply the patches, I have the older 5/12/06 patch that was sent and I also get a number of fails trying to apply that, though that may mean to be applied to rc3 and not rc4, but rc4 + older patch + newer patches fails, and rc4 + git + newer patches fails, looking through the code there are a few things that I cannot find where the code in the context diff came from.

I did attempt to resolve some of the funniness but there were things that I appear to be missing (things in the context diff that I cannot find exist in rc4 and I cannot find being added in any patch), so I don't think I can even get everything to apply even with manual adjusting.

Roger

#85 lore raw
Quoted 2 lines · Roger Heflin

I checked the rc4 -> git patches (there is only 1 ipath patch in it), and I get a number of patch fails attempting to apply the patches,

I've been using a Mercurial mirror of the git tree, but it should be basically identical to the git tree.

Quoted 5 lines

I did attempt to resolve some of the funniness but there were things that I appear to be missing (things in the context diff that I cannot find exist in rc4 and I cannot find being added in any patch), so I don't think I can even get everything to apply even with manual adjusting.

Please send me some more information off-list, and I'll try to help.

	<b
#86 lore raw
In reply to Roland Dreier
Quoted 2 lines · Roland Dreier

Umm... dumping a 53 patch series into the kernel at this stage in the release cycle isn't going to work.

Fair enough.

Quoted 2 lines

Pretty much the only patches that should be going in now are changes that fix crashes or other serious bugs.

OK, I'll filter those out and send them separately.

	<b