Skip to contentSkip to conversation

question on the timewait event of the rdma-cm

DISCUSSION Or Gerlitz 3 participants 9 messages Nov 14, 2011 – Nov 15, 2011 9 messages not in the archive InfiniBand

lore
Share & tools
lore

Or Gerlitz

author
#0 lore raw

Sean,

I'm debugging some disconnect related race in iser - and wanted to check with you something re the CM/RDMA-CM state machine: I see that when a disconnected is initiated by the passive side (iser target) of a connection, such that the active side (iser initiator) gets RDMA_CM_EVENT_DISCONNECTED, later we call rdma_disconnect (e.g to move the QP into error state, send DREP, etc) - SOMETIMES the active side gets also RDMA_CM_EVENT_TIMEWAIT_EXIT event, AFAIK this should happen if the rdma-cm ID isn't destroyed within X time, correct? if yes, so what is that X? if this description isn't accurate what would be the correct phrasing for the condition to get the timewait event?

Or.

the call to rdma_disconnect was made on jiffies count 4295608818

Quoted 7 lines
Nov 14 17:57:50 iser: iser_cma_handler:event 10 status 0 conn ffff88060c8fb2b0 id ffff88061054dc00 jiffies 4295608742
Nov 14 17:57:50 connection399:0: detected conn error (1011)
Nov 14 17:57:50 iser: iser_disconnected_handler:more 32 rx 1 tx completions to reap
Nov 14 17:57:50 iser: iscsi_iser_ep_disconnect:ib conn ffff88060c8fb2b0 state 3 jiffies 4295608818
Nov 14 17:57:51 Kernel reported iSCSI connection 399:0 error (1011) state (3)
Nov 14 17:57:52 iser: iser_cma_handler:event 15 status 0 conn ffff88060c8fb2b0 id ffff88061054dc00 jiffies 4295609333

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4EC13EE9.1070103-VPRAkNaXOzVWk0Htik3J/[email protected]

#2 lore raw
Quoted 10 lines

I'm debugging some disconnect related race in iser - and wanted to check with you something re the CM/RDMA-CM state machine: I see that when a disconnected is initiated by the passive side (iser target) of a connection, such that the active side (iser initiator) gets RDMA_CM_EVENT_DISCONNECTED, later we call rdma_disconnect (e.g to move the QP into error state, send DREP, etc) - SOMETIMES the active side gets also RDMA_CM_EVENT_TIMEWAIT_EXIT event, AFAIK this should happen if the rdma-cm ID isn't destroyed within X time, correct? if yes, so what is that X? if this description isn't accurate what would be the correct phrasing for the condition to get the timewait event?

After disconnecting, the QP should enter the timewait state for twice the packet lifetime. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AA8E-P5GAC/[email protected]

#4 lore raw
Quoted 1 line · Sean

After disconnecting, the QP should enter the timewait state for twice the packet lifetime.

Does going through timewait always holds? e.g no matter what's the return status of rdma_disconnect and/or the status of the rdma_cm disconnected event?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAJZOPZJVs=NSky4hQ1sdoF+XiU8_Pg6FzS2AUMjgqVAzs7Oi6Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#6 lore raw
Quoted 3 lines

Does going through timewait always holds? e.g no matter what's the return status of rdma_disconnect and/or the status of the rdma_cm disconnected event?

It usually holds. It will fail if rdma_disconnect() is called from a bogus state. But otherwise, I believe that it will enter timewait on failure to send or receive a disconnect message. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AAC4-P5GAC/[email protected]

#8 lore raw
Quoted 3 lines

It usually holds.  It will fail if rdma_disconnect() is called from a bogus state.  But otherwise, I believe that it will enter timewait on failure to send or receive a disconnect message

mmm, so can these bogus states for rdma_disconnect to be called be better defined? basically, for the case where the rdma_cm manages the consumer QP, this call is the only way to move an RC QP into the error state when the QP is okay and the consumer want to flush, etc.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAJZOPZKBaQ2HR4kbidW0Mec2xWhZ5H7-ezG20X-J_aYENd9ibw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#10 lore raw
Quoted 4 lines

mmm, so can these bogus states for rdma_disconnect to be called be better defined? basically, for the case where the rdma_cm manages the consumer QP, this call is the only way to move an RC QP into the error state when the QP is okay and the consumer want to flush, etc.

By bogus I mean calling disconnect when the QP has never been connected, or calling disconnect twice. The ib_cm will check the state of the connection when disconnect is called. If disconnect is called from the wrong state, it simply fails. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A8237316E9AAF4-P5GAC/[email protected]

#12 lore raw
Quoted 2 lines

By bogus I mean calling disconnect when the QP has never been connected, or calling disconnect twice

what return value can serve as bogus indication for the application? is that -EINVAL? also, basically a QP could have buffers posted to it also before being connected (e.g after RTR or there's no point in time an rdma-cm consumer for which the cma manages the QP state is exposed to such QP in RTR and not RTS?) as for the twice case, still the IB CM will go through the timewait status in return for the 1st, not bogus call, correct?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAJZOPZLn3hEYp_kHRs22ttF3ca0iq4s5NtFi4iUYgEie81osBg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#14 lore raw
Quoted 7 lines

what return value can serve as bogus indication for the application? is that -EINVAL? also, basically a QP could have buffers posted to it also before being connected (e.g after RTR or there's no point in time an rdma-cm consumer for which the cma manages the QP state is exposed to such QP in RTR and not RTS?) as for the twice case, still the IB CM will go through the timewait status in return for the 1st, not bogus call, correct?

rdma_disconnect() returns the value from modifying the QP state into error. It masks the return value of calling the ib_cm to send the DREQ or DREP. If rdma_disconnect() succeeds, a timewait event should occur, unless the app calls rdma_disconnect() without being connected.

Calling disconnect is one way that a QP may be transitioned into timewait. The ib_cm will also transition to timewait in certain connection failure cases: REP times out or is rejected, or remote side detects a stale connection. Without reading through the code, I don't know off the top of my head if a timewait event would follow an asynchronous rdma_connect() failure, even if rdma_disconnect() were called after the error. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAJZOPZLB-WvMQwhkx_o5kMD-A33ff8chA1AvQSj+npg4HiOAKQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAJZOPZLB-WvMQwhkx_o5kMD-A33ff8chA1AvQSj+npg4HiOAKQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#17 lore raw
Quoted 1 line · Sean

[...] calling disconnect is one way that a QP may be transitioned into timewait [...]

I was talking on the QP "physical" state (e.g error that causes flushes) not the state w.r.t the IB CM.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html