MLX4 Strangeness

DISCUSSION Tom Tucker 3 participants 8 messages Feb 15, 2010 – Feb 19, 2010 4 messages not in the archive InfiniBand

Share & tools

Tom Tucker

author

2010-02-15 20:24 UTC #0 lore raw

Hello,

I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.

The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.

From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:

- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above

I have also confirmed the following with a different application (i.e. rping):

server# rping -s
client# rping -c -a 192.168.80.129

fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...

However, if I run rping the other way, it works fine, that is,

client# rping -s
server# rping -c -a 192.168.80.135

It runs without error until I stop it.

Does anyone have any ideas on how I might debug this?

Thanks,
Tom

A message that is not in the archive (sent off-list or not yet received): 4B79AD88.8000101-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Tziporet Koren

2010-02-16 07:53 UTC #2 lore raw

Quoted 36 lines · Tom Tucker

Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...

Quoted 11 lines

However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
It runs without error until I stop it.
Does anyone have any ideas on how I might debug this?

Tom
What is the vendor syndrome error when you get a completion with error?

Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)

Tziporet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4B7A4EF4.9090209-VPRAkNaXOzVS1MOuV/[email protected]

Tom Tucker

author

2010-02-16 15:21 UTC #4 lore raw

Quoted 45 lines · Tziporet Koren

Quoted 43 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
It runs without error until I stop it.

Quoted 9 lines

Quoted 5 lines
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with error?

Hang on... compiling....

Quoted 3 lines

Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)

Only the MLX4 cards.

Quoted 7 lines

Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Tom Tucker

author

2010-02-16 21:31 UTC #5 lore raw

In reply to a message not in the archive

Quoted 45 lines · Tziporet Koren

Quoted 43 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
It runs without error until I stop it.

Quoted 9 lines

Quoted 5 lines
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with error?

Feb 16 15:08:29 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:51:27 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:01 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81003c9e3200 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:40 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002f2d8400 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Repeat forever....

So the vendor err is 244.

Quoted 9 lines

Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)
Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4B7B0E98.9030506-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Tom Tucker

author

2010-02-17 00:18 UTC #7 lore raw

Quoted 62 lines · Tom Tucker

Quoted 44 lines · Tziporet Koren
Quoted 42 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not
error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
Quoted 10 lines
Quoted 6 lines
It runs without error until I stop it.
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with error?
Feb 16 15:08:29 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:51:27 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:01 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81003c9e3200 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:40 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002f2d8400 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Quoted 4 lines

Repeat forever....
So the vendor err is 244.

Please ignore this. This log skips the failing WR (:-\). I need to do another trace.

Quoted 11 lines

Quoted 9 lines
Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)
Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4B7B35DD.1030609-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org

Tom Tucker

author

2010-02-17 00:25 UTC #9 lore raw

More info...

Reboot the client and try to reconnect to a server that has not been rebooted fails in the same way.

It must be an issue with the server. I see no completions on the server or any indication that an RDMA_SEND was incoming. Is there some way to dump adapter state or otherwise see if there was traffic on the wire?

Tom

Quoted 63 lines · Tom Tucker

Quoted 62 lines · Tom Tucker
Quoted 44 lines · Tziporet Koren
Quoted 42 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not
error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
Quoted 10 lines
Quoted 6 lines
It runs without error until I stop it.
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with error?
Feb 16 15:08:29 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:51:27 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:01 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81003c9e3200 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:40 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002f2d8400 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Quoted 22 lines

Quoted 4 lines
Repeat forever....
So the vendor err is 244.
Please ignore this. This log skips the failing WR (:-\). I need to do another trace.
Quoted 11 lines
Quoted 9 lines
Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)
Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Tom Tucker

author

2010-02-17 18:06 UTC #10 lore raw

In reply to a message not in the archive

Hi Tziporet:

Here is a trace with the data for WR failing with status 12. The vendor error is 129.

Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:154 wr_id 0000000000000000 status 12 opcode 0 vendor_err 129 byte_len 0 qp ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:154 wr_id ffff81002878d800 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:167 wr_id ffff81002878d800 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Any thoughts?
Tom

Quoted 63 lines · Tom Tucker

Quoted 62 lines · Tom Tucker
Quoted 44 lines · Tziporet Koren
Quoted 42 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running 2.7 firmware and the latest OFED 1.5.1. Two systems are involved and each have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems running bonnie++ and iozone concurrently. The test completes and there is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought up
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not
error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above
I have also confirmed the following with a different application (i.e. rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135
Quoted 10 lines
Quoted 6 lines
It runs without error until I stop it.
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with error?
Feb 16 15:08:29 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:51:27 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:01 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81003c9e3200 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 closed (-103)
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:40 vic10 kernel: rpcrdma_event_process:160 wr_id ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002f2d8400 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Quoted 22 lines

Quoted 4 lines
Repeat forever....
So the vendor err is 244.
Please ignore this. This log skips the failing WR (:-\). I need to do another trace.
Quoted 11 lines
Quoted 9 lines
Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)
Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Vu Pham

2010-02-19 00:03 UTC #11 lore raw

Hi Tom,

Status 12 = IB_WC_RETRY_EXC_ERR
Vendor_err = 129 --> Timeout and transport error counter exceeded

This indicates that we lost connection to the client ie. something went wrong on client side (bad operation cause QP error...) please try to catch any error on the client (qp async event, cq error status and vendor_err...)

Today I just run vdbench on big file and get error right away (lost connection and nfsrdma cannot recover from there)

Thanks,
-vu

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Tom Tucker
Sent: Wednesday, February 17, 2010 10:07 AM
To: Tziporet Koren
Cc: [email protected]; [email protected]
Subject: Re: [ewg] MLX4 Strangeness

Hi Tziporet:

Here is a trace with the data for WR failing with status 12. The vendor error is 129.

Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:154 wr_id 
0000000000000000 status 12 opcode 0 vendor_err 129 byte_len 0 qp 
ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:154 wr_id 
ffff81002878d800 status 5 opcode 0 vendor_err 244 byte_len 0 qp 
ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 17 12:27:33 vic10 kernel: rpcrdma_event_process:167 wr_id

ffff81002878d800 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002a13ec00 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Any thoughts?
Tom

Quoted 7 lines · Tom Tucker

Quoted 6 lines · Tom Tucker
Quoted 5 lines · Tziporet Koren
Quoted 3 lines · Tom Tucker
Hello,
I am seeing some very strange behavior on my MLX4 adapters running

2.7

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
firmware and the latest OFED 1.5.1. Two systems are involved and

each

Quoted 5 lines

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
have dual ported MTHCA DDR adapter and MLX4 adapters.
The scenario starts with NFSRDMA stress testing between the two systems
running bonnie++ and iozone concurrently. The test completes and

there

Quoted 4 lines

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
is no issue. Then 6 minutes pass and the server "times out" the connection and shuts down the RC connection to the client.
From this point on, using the RDMA CM, a new RC QP can be brought

Quoted 14 lines

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
and moved to RTS, however, the first RDMA_SEND to the NFS SERVER system
fails with IB_WC_RETRY_EXC_ERR. I have confirmed:
- that "arp" completed successfully and the neighbor entries are populated on both the client and server
- that the QP are in the RTS state on both the client and server
- that there are RECV WR posted to the RQ on the server and they did not
error out
- that no RECV WR completed successfully or in error on the server
- that there are SEND WR posted to the QP on the client
- the client side SEND_WR fails with error 12 as mentioned above

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
I have also confirmed the following with a different application

(i.e.

Quoted 16 lines

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
rping):
server# rping -s
client# rping -c -a 192.168.80.129
fails with the exact same error, i.e.
client# rping -c -a 192.168.80.129
cq completion failed status 12
wait for RDMA_WRITE_ADV state 10
client DISCONNECT EVENT...
However, if I run rping the other way, it works fine, that is,
client# rping -s
server# rping -c -a 192.168.80.135

Quoted 8 lines

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
It runs without error until I stop it.
Does anyone have any ideas on how I might debug this?
Tom
What is the vendor syndrome error when you get a completion with

error?

Quoted 17 lines

Quoted 2 lines

Feb 16 15:08:29 vic10 kernel: rpcrdma: connection to 
192.168.80.129:20049 closed (-103)
Feb 16 15:51:27 vic10 kernel: rpcrdma: connection to 
192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:01 vic10 kernel: rpcrdma_event_process:160 wr_id 
ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp 
ffff81003c9e3200 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 
192.168.80.129:20049 closed (-103)
Feb 16 15:52:06 vic10 kernel: rpcrdma: connection to 
192.168.80.129:20049 on mlx4_0, memreg 5 slots 32 ird 16
Feb 16 15:52:40 vic10 kernel: rpcrdma_event_process:160 wr_id

ffff81002879a000 status 5 opcode 0 vendor_err 244 byte_len 0 qp ffff81002f2d8400 ex 00000000 src_qp 00000000 wc_flags, 0 pkey_index

Quoted 24 lines

Quoted 4 lines
Repeat forever....
So the vendor err is 244.
Please ignore this. This log skips the failing WR (:-\). I need to do another trace.
Quoted 12 lines
Quoted 10 lines
Does the issue occurs only on the ConnectX cards (mlx4) or also on the InfiniHost cards (mthca)
Tziporet
_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg