question on the timewait event of the rdma-cm
Sean,
I'm debugging some disconnect related race in iser - and wanted to check with you something re the CM/RDMA-CM state machine: I see that when a disconnected is initiated by the passive side (iser target) of a connection, such that the active side (iser initiator) gets RDMA_CM_EVENT_DISCONNECTED, later we call rdma_disconnect (e.g to move the QP into error state, send DREP, etc) - SOMETIMES the active side gets also RDMA_CM_EVENT_TIMEWAIT_EXIT event, AFAIK this should happen if the rdma-cm ID isn't destroyed within X time, correct? if yes, so what is that X? if this description isn't accurate what would be the correct phrasing for the condition to get the timewait event?
Or.
the call to rdma_disconnect was made on jiffies count 4295608818
Quoted 7 lines
Nov 14 17:57:50 iser: iser_cma_handler:event 10 status 0 conn ffff88060c8fb2b0 id ffff88061054dc00 jiffies 4295608742 Nov 14 17:57:50 connection399:0: detected conn error (1011) Nov 14 17:57:50 iser: iser_disconnected_handler:more 32 rx 1 tx completions to reap Nov 14 17:57:50 iser: iscsi_iser_ep_disconnect:ib conn ffff88060c8fb2b0 state 3 jiffies 4295608818 Nov 14 17:57:51 Kernel reported iSCSI connection 399:0 error (1011) state (3) Nov 14 17:57:52 iser: iser_cma_handler:event 15 status 0 conn ffff88060c8fb2b0 id ffff88061054dc00 jiffies 4295609333
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4EC13EE9.1070103-VPRAkNaXOzVWk0Htik3J/[email protected]
Quoted 10 lines
I'm debugging some disconnect related race in iser - and wanted to check with you something re the CM/RDMA-CM state machine: I see that when a disconnected is initiated by the passive side (iser target) of a connection, such that the active side (iser initiator) gets
RDMA_CM_EVENT_DISCONNECTED, later we callrdma_disconnect(e.g to move the QP into error state, send DREP, etc) - SOMETIMES the active side gets alsoRDMA_CM_EVENT_TIMEWAIT_EXITevent, AFAIK this should happen if the rdma-cm ID isn't destroyed within X time, correct? if yes, so what is that X? if this description isn't accurate what would be the correct phrasing for the condition to get the timewait event?
After disconnecting, the QP should enter the timewait state for twice the packet lifetime. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AA8E-P5GAC/[email protected]
Quoted 1 line · Sean
After disconnecting, the QP should enter the timewait state for twice the packet lifetime.
Does going through timewait always holds? e.g no matter what's the return status of rdma_disconnect and/or the status of the rdma_cm disconnected event?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZJVs=NSky4hQ1sdoF+XiU8_Pg6FzS2AUMjgqVAzs7Oi6Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 3 lines
Does going through timewait always holds? e.g no matter what's the return status of
rdma_disconnectand/or the status of therdma_cmdisconnected event?
It usually holds. It will fail if rdma_disconnect() is called from a bogus state. But otherwise, I believe that it will enter timewait on failure to send or receive a disconnect message. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AAC4-P5GAC/[email protected]
Quoted 3 lines
It usually holds. It will fail if
rdma_disconnect()is called from a bogus state. But otherwise, I believe that it will enter timewait on failure to send or receive a disconnect message
mmm, so can these bogus states for rdma_disconnect to be called be better defined? basically, for the case where the rdma_cm manages the consumer QP, this call is the only way to move an RC QP into the error state when the QP is okay and the consumer want to flush, etc.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZKBaQ2HR4kbidW0Mec2xWhZ5H7-ezG20X-J_aYENd9ibw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 4 lines
mmm, so can these bogus states for
rdma_disconnectto be called be better defined? basically, for the case where therdma_cmmanages the consumer QP, this call is the only way to move an RC QP into the error state when the QP is okay and the consumer want to flush, etc.
By bogus I mean calling disconnect when the QP has never been connected, or calling disconnect twice. The ib_cm will check the state of the connection when disconnect is called. If disconnect is called from the wrong state, it simply fails. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AAF4-P5GAC/[email protected]
Quoted 2 lines
By bogus I mean calling disconnect when the QP has never been connected, or calling disconnect twice
what return value can serve as bogus indication for the application? is that -EINVAL? also, basically a QP could have buffers posted to it also before being connected (e.g after RTR or there's no point in time an rdma-cm consumer for which the cma manages the QP state is exposed to such QP in RTR and not RTS?) as for the twice case, still the IB CM will go through the timewait status in return for the 1st, not bogus call, correct?
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZLn3hEYp_kHRs22ttF3ca0iq4s5NtFi4iUYgEie81osBg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 7 lines
what return value can serve as bogus indication for the application? is that
-EINVAL? also, basically a QP could have buffers posted to it also before being connected (e.g after RTR or there's no point in time an rdma-cm consumer for which the cma manages the QP state is exposed to such QP in RTR and not RTS?) as for the twice case, still the IB CM will go through the timewait status in return for the 1st, not bogus call, correct?
rdma_disconnect() returns the value from modifying the QP state into error. It masks the return value of calling the ib_cm to send the DREQ or DREP. If rdma_disconnect() succeeds, a timewait event should occur, unless the app calls rdma_disconnect() without being connected.
Calling disconnect is one way that a QP may be transitioned into timewait. The ib_cm will also transition to timewait in certain connection failure cases: REP times out or is rejected, or remote side detects a stale connection. Without reading through the code, I don't know off the top of my head if a timewait event would follow an asynchronous rdma_connect() failure, even if rdma_disconnect() were called after the error. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAJZOPZLB-WvMQwhkx_o5kMD-A33ff8chA1AvQSj+npg4HiOAKQ@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAJZOPZLB-WvMQwhkx_o5kMD-A33ff8chA1AvQSj+npg4HiOAKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 1 line · Sean
[...] calling disconnect is one way that a QP may be transitioned into timewait [...]
I was talking on the QP "physical" state (e.g error that causes flushes) not the state w.r.t the IB CM.
Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html