CX314A WCE error: WR_FLUSH_ERR

DISCUSSION Liu, Changcheng 3 participants 5 messages Aug 21, 2019 – Aug 22, 2019 InfiniBand

Share & tools

Liu, Changcheng

author

2019-08-21 12:10 UTC #0 lore raw

Hi all,
In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR" in the WCE(work completion element) polled from completion queue bound with RQ(Receive Queue). Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR" problem?

With CX314A/40Gb NIC, I hit this error when using RC transport type with only Send Operation(IBV_WR_SEND) WR(work request) on SQ(Send Queue). Every WR only has one SGE(scatter/gather element) and all the SGE on RQ has the same size. The SGE size in SQ WR is not greater than the SGE size in RQ WR.

There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114 in the "RDMA Aware Networks Programming User Manual" http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
But I still didn't understand it well. How to trigger this error with a short demo program?
"
IBV_WC_WR_FLUSH_ERR
This event is generated when an invalid remote error is thrown when the responder detects an
invalid request. It may be that the operation is not supported by the request queue or there is
insufficient buffer space to receive the request.
"

B.R.
Changcheng

Tom Talpey

2019-08-21 13:37 UTC #1 lore raw

Quoted 7 lines · Changcheng

Hi all,
In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR" in the WCE(work completion element) polled from completion queue bound with RQ(Receive Queue). Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR" problem?
With CX314A/40Gb NIC, I hit this error when using RC transport type with only Send Operation(IBV_WR_SEND) WR(work request) on SQ(Send Queue). Every WR only has one SGE(scatter/gather element) and all the SGE on RQ has the same size. The SGE size in SQ WR is not greater than the SGE size in RQ WR.

Quoted 8 lines

There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114 in the "RDMA Aware Networks Programming User Manual" http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
But I still didn't understand it well. How to trigger this error with a short demo program?
"
IBV_WC_WR_FLUSH_ERR
This event is generated when an invalid remote error is thrown when the responder detects an
invalid request. It may be that the operation is not supported by the request queue or there is
insufficient buffer space to receive the request.
"

The most common reason for a flushed work request is loss of the connection to the remote peer. This can be caused by any number of conditions.

The second-most common is a programming error in the upper layer protocol. A shortage of posted receives on either peer, a protection error on some buffer, etc.

If you're looking to actually trigger this error for testing, well, try one of the above. If you're trying to figure out why it's happening, that can take some digging, but not in the RDMA stack, typically.

Tom.

Liu, Changcheng

author

2019-08-21 15:40 UTC #2 lore raw

Quoted 17 lines · Tom Talpey

Quoted 7 lines · Changcheng
Hi all,
In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR" in the WCE(work completion element) polled from completion queue bound with RQ(Receive Queue). Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR" problem?
With CX314A/40Gb NIC, I hit this error when using RC transport type with only Send Operation(IBV_WR_SEND) WR(work request) on SQ(Send Queue). Every WR only has one SGE(scatter/gather element) and all the SGE on RQ has the same size. The SGE size in SQ WR is not greater than the SGE size in RQ WR.
Quoted 8 lines
There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114 in the "RDMA Aware Networks Programming User Manual" http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
But I still didn't understand it well. How to trigger this error with a short demo program?
"
IBV_WC_WR_FLUSH_ERR
This event is generated when an invalid remote error is thrown when the responder detects an
invalid request. It may be that the operation is not supported by the request queue or there is
insufficient buffer space to receive the request.
"

Quoted 3 lines

The most common reason for a flushed work request is loss of the connection to the remote peer. This can be caused by any number of conditions.

Good diretion. I'll debug it in this way first.

Quoted 4 lines

The second-most common is a programming error in the upper layer protocol. A shortage of posted receives on either peer, a protection error on some buffer, etc.

Do you mean the protection key such as l_key/r_key isn't set well? What's kind of protection error could trigger IBV_WC_WR_FLUSH_ERR?

Quoted 5 lines

If you're looking to actually trigger this error for testing, well, try one of the above. If you're trying to figure out why it's happening, that can take some digging, but not in the RDMA stack, typically.

Many thanks.

--Changcheng

Quoted 3 lines

Tom.

Doug Ledford

2019-08-21 18:47 UTC #3 lore raw

Quoted 33 lines · Changcheng

Quoted 29 lines · Tom Talpey
Quoted 14 lines · Changcheng
Hi all,
In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR"
in the WCE(work completion element) polled from completion queue bound with RQ(Receive Queue).
Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR"
problem?
With CX314A/40Gb NIC, I hit this error when using RC transport
type with only Send Operation(IBV_WR_SEND) WR(work request) on SQ(Send Queue).
Every WR only has one SGE(scatter/gather element) and all the
SGE on RQ has the same size. The SGE size in SQ WR is not greater than the SGE size in RQ WR.
Quoted 13 lines
There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114
in the "RDMA Aware Networks Programming User Manual"
http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
But I still didn't understand it well. How to trigger this
error with a short demo program?
"
IBV_WC_WR_FLUSH_ERR
This event is generated when an invalid remote error is
thrown when the responder detects an
invalid request. It may be that the operation is not
supported by the request queue or there is
insufficient buffer space to receive the request.
"
Quoted 3 lines
The most common reason for a flushed work request is loss of the connection to the remote peer. This can be caused by any number of conditions.

Quoted 6 lines

Good diretion. I'll debug it in this way first.
Quoted 3 lines
The second-most common is a programming error in the upper layer protocol. A shortage of posted receives on either peer, a protection error on some buffer, etc.
Do you mean the protection key such as l_key/r_key isn't set well? What's kind of protection error could trigger IBV_WC_WR_FLUSH_ERR?

FLUSH_ERR is the error used whenever a queue pair goes into an error state and there are still WQEs posted to the queue pair. All outstanding WQEs are returned with the state IBV_WC_WR_FLUSH_ERR. This is how you make sure you don't loose WQEs when the QP hits an error state. So, literally *anything* that can cause a QP to go into an ERROR state will result in all WQEs currently posted to the QP being sent back with this FLUSH_ERR. FLUSH_ERR literally just means that the card is flushing out the QP's work queue because now that the QP is in an error state it can't process the WQEs and, presumably, the application needs to know which ones completed and which ones didn't so it knows what to requeue once the QP is no longer in an error state.

As Tom has already pointed out, all of these things will throw the queue pair into an error state and cause all posted WQEs to be flushed with the FLUSH_ERR condition:

1) Loss of queue pair connection
2) Any memory permission violation (attempt to write to read only memory, attempt to RDMA read/write to an invalid rkey, etc)
3) Receipt of any post_send message without a waiting post_recv buffer to accept the message
4) Receipt of a post_send message that is too large to fit in the first available post_recv buffer

A common cause of this sort of thing is when you don't do proper flow control on the queue pair and the sending side floods the receiving side and runs it out of posted recv WQEs. Although, in your case, you did say this was happening on the receive queue, so that implies this is happening on the receiving side, so if that is what's happenining here, the process would have to be something like:

sender starts sending data (maybe without any flow control)
receiver starts receiving data and refilling buffers
...
receiver runs totally dry of buffers and gets an incoming recv causing qp to go into error state

receiver then posts refill buffers to the RQ after the QP went into error state but before acknowledging the error state and shutting down the recv processing thread

all recv buffers posted as WQEs are flushed back to the process with FLUSH_ERR because they were posted to a QP in ERROR state

Quoted 9 lines

Quoted 4 lines
If you're looking to actually trigger this error for testing, well, try one of the above. If you're trying to figure out why it's happening, that can take some digging, but not in the RDMA stack, typically.
Many thanks.
--Changcheng
Quoted 2 lines
Tom.

Signature · 3 lines

Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

Liu, Changcheng

author

2019-08-22 15:03 UTC #4 lore raw

Thanks Doug Ledford & Tom. I've found that QP is force switched into Error status to flush outstandting WQEs into CQ with WR_FLUSH_ERR status.

Quoted 99 lines · Doug Ledford

Quoted 33 lines · Changcheng
Quoted 29 lines · Tom Talpey
Quoted 14 lines · Changcheng
Hi all,
In one system, it always frequently hit "IBV_WC_WR_FLUSH_ERR"
in the WCE(work completion element) polled from completion queue bound with RQ(Receive Queue).
Does anyone has some idea to debug "IBV_WC_WR_FLUSH_ERR"
problem?
With CX314A/40Gb NIC, I hit this error when using RC transport
type with only Send Operation(IBV_WR_SEND) WR(work request) on SQ(Send Queue).
Every WR only has one SGE(scatter/gather element) and all the
SGE on RQ has the same size. The SGE size in SQ WR is not greater than the SGE size in RQ WR.
Quoted 13 lines
There’s one explanation about IBV_WC_WR_FLUSH_ERR on page 114
in the "RDMA Aware Networks Programming User Manual"
http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf
But I still didn't understand it well. How to trigger this
error with a short demo program?
"
IBV_WC_WR_FLUSH_ERR
This event is generated when an invalid remote error is
thrown when the responder detects an
invalid request. It may be that the operation is not
supported by the request queue or there is
insufficient buffer space to receive the request.
"
Quoted 3 lines
The most common reason for a flushed work request is loss of the connection to the remote peer. This can be caused by any number of conditions.
Quoted 6 lines
Good diretion. I'll debug it in this way first.
Quoted 3 lines
The second-most common is a programming error in the upper layer protocol. A shortage of posted receives on either peer, a protection error on some buffer, etc.
Do you mean the protection key such as l_key/r_key isn't set well? What's kind of protection error could trigger IBV_WC_WR_FLUSH_ERR?
FLUSH_ERR is the error used whenever a queue pair goes into an error state and there are still WQEs posted to the queue pair. All outstanding WQEs are returned with the state IBV_WC_WR_FLUSH_ERR. This is how you make sure you don't loose WQEs when the QP hits an error state. So, literally *anything* that can cause a QP to go into an ERROR state will result in all WQEs currently posted to the QP being sent back with this FLUSH_ERR. FLUSH_ERR literally just means that the card is flushing out the QP's work queue because now that the QP is in an error state it can't process the WQEs and, presumably, the application needs to know which ones completed and which ones didn't so it knows what to requeue once the QP is no longer in an error state.
As Tom has already pointed out, all of these things will throw the queue pair into an error state and cause all posted WQEs to be flushed with the FLUSH_ERR condition:
1) Loss of queue pair connection
2) Any memory permission violation (attempt to write to read only memory, attempt to RDMA read/write to an invalid rkey, etc)
3) Receipt of any post_send message without a waiting post_recv buffer to accept the message
4) Receipt of a post_send message that is too large to fit in the first available post_recv buffer
A common cause of this sort of thing is when you don't do proper flow control on the queue pair and the sending side floods the receiving side and runs it out of posted recv WQEs. Although, in your case, you did say this was happening on the receive queue, so that implies this is happening on the receiving side, so if that is what's happenining here, the process would have to be something like:
sender starts sending data (maybe without any flow control)
receiver starts receiving data and refilling buffers
...
receiver runs totally dry of buffers and gets an incoming recv causing qp to go into error state
receiver then posts refill buffers to the RQ after the QP went into error state but before acknowledging the error state and shutting down the recv processing thread
all recv buffers posted as WQEs are flushed back to the process with FLUSH_ERR because they were posted to a QP in ERROR state
Quoted 9 lines
Quoted 4 lines
If you're looking to actually trigger this error for testing, well, try one of the above. If you're trying to figure out why it's happening, that can take some digging, but not in the RDMA stack, typically.
Many thanks.
--Changcheng
Quoted 2 lines
Tom.
Signature · 3 lines
Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD