Skip to contentSkip to conversation

RDMA/nes: double CLOSE event indication crash

PATCH In review Faisal Latif 3 participants 5 messages Aug 14, 2010 – Sep 8, 2010 2 messages not in the archive InfiniBand

lore
Share & tools
lore

Faisal Latif

author
#0 lore raw

PATCH 1/1 RDMA/nes: double CLOSE event indication crash

During a stress testing in a large cluster, multiple close event is detected and BUG() is hit in core. The cause is that active node gave up while waitings for MPA response from the peer and tried to close the connection by sending RST. The passive node driver receives the RST but is waiting for MPA response from user. When MPA accept is receives, the driver send offloads the connection and sends CLOSE event. The driver get an AE indicating RESET receive and also send CLOSE event causing BUG() to hit in the core. RESET handling and sending CLOSE events are fixed.

Signed-off-by Faisal Latif <[email protected]>
---
 drivers/infiniband/hw/nes/nes_cm.c |   18 ++++++++++--------
 1 files changed, 10 insertions(+), 8 deletions(-)
drivers/infiniband/hw/nes/nes_cm.c +10 -8
index 986d6f3..10e4bde 100644
@@ -502,7 +502,9 @@ int schedule_nes_timer(struct nes_cm_node *cm_node, struct sk_buff *skb,
 static void nes_retrans_expired(struct nes_cm_node *cm_node)
 {
 	struct iw_cm_id *cm_id = cm_node->cm_id;
-	switch (cm_node->state) {
+	enum nes_cm_node_state    state = cm_node->state;
+	cm_node->state = NES_CM_STATE_CLOSED;
+	switch (state) {
 	case NES_CM_STATE_SYN_RCVD:
 	case NES_CM_STATE_CLOSING:
 		rem_ref_cm_node(cm_node->cm_core, cm_node);

@@ -511,7 +513,6 @@ static void nes_retrans_expired(struct nes_cm_node *cm_node)
 	case NES_CM_STATE_FIN_WAIT1:
 		if (cm_node->cm_id)
 			cm_id->rem_ref(cm_id);
-		cm_node->state = NES_CM_STATE_CLOSED;
 		send_reset(cm_node, NULL);
 		break;
 	default:

@@ -1439,9 +1440,6 @@ static void handle_rst_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb,
 		break;
 	case NES_CM_STATE_MPAREQ_RCVD:
 		passive_state = atomic_add_return(1, &cm_node->passive_state);
-		if (passive_state ==  NES_SEND_RESET_EVENT)
-			create_event(cm_node, NES_CM_EVENT_RESET);
-		cm_node->state = NES_CM_STATE_CLOSED;
 		dev_kfree_skb_any(skb);
 		break;
 	case NES_CM_STATE_ESTABLISHED:

@@ -1456,6 +1454,7 @@ static void handle_rst_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb,
 	case NES_CM_STATE_CLOSED:
 		drop_packet(skb);
 		break;
+	case NES_CM_STATE_FIN_WAIT2:
 	case NES_CM_STATE_FIN_WAIT1:
 	case NES_CM_STATE_LAST_ACK:
 		cm_node->cm_id->rem_ref(cm_node->cm_id);

@@ -2781,6 +2780,12 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param)
 		return -EINVAL;
 	}
 
+	passive_state = atomic_add_return(1, &cm_node->passive_state);
+	if (passive_state == NES_SEND_RESET_EVENT) {
+		rem_ref_cm_node(cm_node->cm_core, cm_node);
+		return -ECONNRESET;
+	}
+
 	/* associate the node with the QP */
 	nesqp->cm_node = (void *)cm_node;
 	cm_node->nesqp = nesqp;

@@ -2983,9 +2988,6 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param)
 		printk(KERN_ERR "%s[%u] OFA CM event_handler returned, "
 			"ret=%d\n", __func__, __LINE__, ret);
 
-	passive_state = atomic_add_return(1, &cm_node->passive_state);
-	if (passive_state == NES_SEND_RESET_EVENT)
-		create_event(cm_node, NES_CM_EVENT_RESET);
 	return 0;
 }
Signature · 6 lines
1.6.0

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#1 lore raw
Quoted 2 lines · Faisal Latif

During a stress testing in a large cluster, multiple close event is detected and BUG() is hit in core. The cause is [...]

Do you refer to the core of the IB stack? if not, to whose core?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4C677DD9.5030308-hKgKHo2Ms0FWk0Htik3J/[email protected]

author
#3 lore raw
Quoted 9 lines

-----Original Message-----
From: Or Gerlitz [mailto:ogerlitz-hKgKHo2Ms0FWk0Htik3J/[email protected]]
Sent: Sunday, August 15, 2010 12:41 AM
To: Latif, Faisal
Cc: Roland Dreier; [email protected]
Subject: Re: [PATCH] RDMA/nes: double CLOSE event indication crash

Faisal Latif wrote:

Quoted 8 lines
Quoted 1 line

During a stress testing in a large cluster, multiple close event is

detected

Quoted 1 line

and BUG() is hit in core. The cause is [...]

Do you refer to the core of the IB stack? if not, to whose core?

Or.

Hi Or

BUG() was in iw_cm.ko in its close handler mentioned as core in my email and caused by iw_nes.ko.

Faisal

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 2EFBCAEF10980645BBCFB605689E08E9049F5E881C-uLM7Qlg6MbdZtRGVdHMbwrfspsVTdybXVpNB7YpNyf8@public.gmane.org

#5 lore raw
Quoted 1 line · Faisal

BUG() was in iw_cm.ko in its close handler mentioned as core in my email and caused by iw_nes.ko.

I see, looks like iwcm.c accounts for most of the BUG* calls made from the core, could be nice to reduce them over time.

Or.

Quoted 20 lines
# grep -n BUG drivers/infiniband/core/*.c | grep "("
drivers/infiniband/core/cma.c:1262:             BUG_ON(1);
drivers/infiniband/core/cm.c:1169:      BUG_ON(cm_id->state != IB_CM_IDLE);
drivers/infiniband/core/cm.c:1318:              BUG_ON(!work);
drivers/infiniband/core/device.c:175:   BUG_ON(size < sizeof (struct ib_device));
drivers/infiniband/core/device.c:194:   BUG_ON(device->reg_state != IB_DEV_UNREGISTERED);
drivers/infiniband/core/iwcm.c:120:     BUG_ON(!list_empty(&cm_id_priv->work_free_list));
drivers/infiniband/core/iwcm.c:163:     BUG_ON(atomic_read(&cm_id_priv->refcount)==0);
drivers/infiniband/core/iwcm.c:165:             BUG_ON(!list_empty(&cm_id_priv->work_list));
drivers/infiniband/core/iwcm.c:186:             BUG_ON(!list_empty(&cm_id_priv->work_list));
drivers/infiniband/core/iwcm.c:241:     BUG_ON(qp == NULL);
drivers/infiniband/core/iwcm.c:298:             BUG();
drivers/infiniband/core/iwcm.c:374:             BUG();
drivers/infiniband/core/iwcm.c:397:     BUG_ON(test_bit(IWCM_F_CALLBACK_DESTROY, &cm_id_priv->flags));
drivers/infiniband/core/iwcm.c:518:             BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_RECV);
drivers/infiniband/core/iwcm.c:583:             BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_SENT);
drivers/infiniband/core/iwcm.c:620:     BUG_ON(iw_event->status);
drivers/infiniband/core/iwcm.c:695:     BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_RECV);
drivers/infiniband/core/iwcm.c:723:     BUG_ON(cm_id_priv->state != IW_CM_STATE_CONN_SENT);
drivers/infiniband/core/iwcm.c:795:             BUG();
Quoted 8 lines
drivers/infiniband/core/iwcm.c:824:             BUG();
drivers/infiniband/core/iwcm.c:865:             BUG_ON(atomic_read(&cm_id_priv->refcount)==0);
drivers/infiniband/core/iwcm.c:869:                             BUG_ON(!list_empty(&cm_id_priv->work_list));
drivers/infiniband/core/mad.c:587:      BUG_ON(!mad_list->mad_queue);
drivers/infiniband/core/mad.c:1396:                     BUG_ON(!*method);
drivers/infiniband/core/mad.c:1406:                     BUG_ON(*method);
drivers/infiniband/core/mad.c:2242:                             BUG_ON(1);
drivers/infiniband/core/verbs.c:91:             BUG();

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html