ocfs2: o2net: fix packets lost issue when reconnect
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest0 of 3 patches 0 repliesNew
Patches in v1 No applies recorded yet
None of the 3 patches the cover letter announces are in this archive.
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [Ocfs2-devel] [PATCH 0/3] ocfs2: o2net: fix packets lost issue when reconnect
Hi Joseph,
Quoted 17 lines · Joseph Qi
Quoted 16 lines · Junxiao Bi
Hi,
After the tcp connection is established between two ocfs2 nodes, an idle timer will be set to check its state periodically, if no messages are received during this time, idle timer will timeout, it will shutdown the connection and try to rebuild, so pending message in tcp queues will be lost. This may cause the whole ocfs2 cluster hung.
This is very possible to happen when network state goes bad. Do the reconnect is useless, it will fail if network state doesn't recover. Just waiting there for network recovering may be a good idea, it will not lost messages and some node will be fenced until cluster goes into split-brain state, for this case, Tcp user timeout is used to override the tcp retransmit timeout. It will timeout after 25 days, user should have notice this through the provided log and fix the network, if they don't, ocfs2 will fall back to original reconnect way.
The following is the serial of patches to fix the bug. Please help review.
Quoted 6 lines
TCP RTT is auto-regressive, that means the following case may take place:
Suppose current retransmission interval is ?T (somewhat long), network recovers but down again before the next retransmission windows comes (< ?T), so the network recovery won't be detected and ocfs2 cluster still hungs.
Network recovers but down again, this means the network is still down. Ocfs2 hung is an expected behavior when network is down if split-brain case. What we need take care is how long can ocfs2 recover from hung after network recover(not down again). I didn't know tcp internal about how they retransmit the packets, I just test blocking the network for half an hour, it just need several seconds to recover from the hung. Of course, how long the hung recover may also depends on how hard it hung from dlm.
Thanks,
Junxiao.
Quoted 10 lines
Quoted 9 lines
Thanks,
Junxiao._______________________________________________
Ocfs2-devel mailing list
Ocfs2-devel at oss.oracle.com
https://oss.oracle.com/mailman/listinfo/ocfs2-devel