Problem running rping over Intel adapters

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count.

Quoted 6 lines

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

Quoted 4 lines

2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count.

On IB, retry count is a 3-bit value. See the CM REQ message.

The kernel rdma cm should probably check for a valid value and reduce it accordingly.

Quoted 13 lines

Quoted 12 lines

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count.

Quoted 3 lines

On IB, retry count is a 3-bit value. See the CM REQ message.

The kernel rdma cm should probably check for a valid value and reduce it

accordingly.

Then it's also fair to say that right now rping passes a known invalid value and needs fixed ;-)

Quoted 16 lines

Quoted 16 lines

Quoted 11 lines

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

conn_param.responder_resources = 1; conn_param.initiator_depth = 1; conn_param.retry_count = 10;

2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count.

On IB, retry count is a 3-bit value. See the CM REQ message.

The kernel rdma cm should probably check for a valid value and reduce it

Quoted 4 lines

accordingly.

Then it's also fair to say that right now rping passes a known invalid value and needs fixed ;-)

For IB, but I wasn't sure about iWarp. It doesn't look like it's used there though.

If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value.

Quoted 14 lines

Quoted 14 lines

Quoted 11 lines

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count.

On IB, retry count is a 3-bit value. See the CM REQ message.

The kernel rdma cm should probably check for a valid value and reduce it

Quoted 4 lines

accordingly.

Then it's also fair to say that right now rping passes a known invalid value and needs fixed ;-)

For IB, but I wasn't sure about iWarp. It doesn't look like it's used there though.

If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value. --

Not used by iwarp drivers...

Quoted 2 lines · Steve Wise

Not used by iwarp drivers...

Which one, the retry counter or the RNR retry counter?

Or.

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

retry_count, I assume, is for connection establishment? For the Chelsio devices, this is all up to the TCP stack and thus this counter is not used. If there is no TCP listener, then the connection establishment will fail immediately. If the node is not reachable, then normal TCP retransmit rules apply, but once TCP gives up, then the rdma connection establishment will fail.

Quoted 6 lines

retry_count, I assume, is for connection establishment? For the Chelsio devices, this is all up to the TCP stack and thus this counter is not used. If there is no TCP listener, then the connection establishment will fail immediately. If the node is not reachable, then normal TCP retransmit rules apply, but once TCP gives up, then the rdma connection establishment will fail.

On IB, this is a transport level retry of a lost request.

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

On the accept side, rping clears the conn_params and then sets just the responder_resources and initiator_depth, without even checking the incoming requested conn_param values from the incoming cm_id. So, OK, you can get away with that since this is a simple test program, but still not "best programming practices". However, the important part here is the retry_count of 10. That won't work on Intel/QLogic hardware.

-----Original Message-----
From: Hefty, Sean
Sent: Wednesday, October 03, 2012 6:39 PM
To: Doug Ledford; Marciniszyn, Mike; [email protected]
Subject: RE: Problem running rping over Intel adapters

Quoted 7 lines

1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets:

	conn_param.responder_resources = 1;
	conn_param.initiator_depth = 1;
	conn_param.retry_count = 10;

Quoted 6 lines

On the accept side, rping clears the conn_params and then sets just the responder_resources and initiator_depth, without even checking the incoming requested conn_param values from the incoming cm_id. So, OK, you can get away with that since this is a simple test program, but still not "best programming practices". However, the important part here is the retry_count of 10. That won't work on Intel/QLogic hardware.

I pushed in the following fix to rping. Thanks

---

rping: Reduce retry_count to fit in 3-bits

From: Sean Hefty <[email protected]>

retry_count is a 3 bit value on IB, reduce it from 10 to 7.

A value of 10 prevents rping from working over the Intel IB HCA. Problem reported by Doug Ledford <[email protected]>

The retry_count is also not set when calling rdma_accept.
Rather than passing different values into rdma_accept than what was specified by the remote side, use the values given in the connection request.

Signed-off-by Sean Hefty <[email protected]>

---
 examples/rping.c |    9 ++-------
 1 files changed, 2 insertions(+), 7 deletions(-)

examples/rping.c index 785338e..32bd70a renamed from examples/rping.c +0 -0

100644

examples/rping.c +1 -6

@@ -342,16 +342,11 @@ error:
 
 static int rping_accept(struct rping_cb *cb)  {
-	struct rdma_conn_param conn_param;
 	int ret;
 
 	DEBUG_LOG("accepting client connection request\n");
 
-	memset(&conn_param, 0, sizeof conn_param);
-	conn_param.responder_resources = 1;
-	conn_param.initiator_depth = 1;
-
-	ret = rdma_accept(cb->child_cm_id, &conn_param);
+	ret = rdma_accept(cb->child_cm_id, NULL);
 	if (ret) {
 		perror("rdma_accept");
 		return ret;