host/target keep alive timeout loop
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Sagi Grimberg
maintainerQuoted 8 lines
Hey Sagi/Christoph,
While running the same kato/recovery tests I've logged in a few other threads, occasionally I get some controllers on the host that will not reconnect. Even after I quiesce the test and have the interfaces up and everything is pingable. When it gets in this state, some of the 10 controllers are up and ok, and others are stuck in this reconnect/fail loop.
Quoted 6 lines
The host is stuck continually logging this for one or more controllers:
[ 7885.617176] nvme nvme10: failed nvme_keep_alive_end_io error=16385 [ 7886.837087] nvme nvme10: rdma_resolve_addr wait failed (-110). [ 7890.183979] nvme nvme10: failed to initialize i/o queue: -110 [ 7890.247538] nvme nvme10: Failed reconnect attempt, requeueing...
This looks like an underlying problem causing the host rdma_connect to timeout. Did it happen before or is it a new thing?
Quoted 4 lines
Quoted 4 lines
Hey Sagi/Christoph,
While running the same kato/recovery tests I've logged in a few other
threads,
Quoted 1 line
Quoted 1 line
occasionally I get some controllers on the host that will not reconnect.
Even
Quoted 1 line
Quoted 1 line
after I quiesce the test and have the interfaces up and everything is
pingable.
Quoted 1 line
Quoted 1 line
When it gets in this state, some of the 10 controllers are up and ok, and
others
Quoted 11 lines
Quoted 8 lines
are stuck in this reconnect/fail loop.
The host is stuck continually logging this for one or more controllers:
[ 7885.617176] nvme nvme10: failed nvme_keep_alive_end_io error=16385 [ 7886.837087] nvme nvme10: rdma_resolve_addr wait failed (-110). [ 7890.183979] nvme nvme10: failed to initialize i/o queue: -110 [ 7890.247538] nvme nvme10: Failed reconnect attempt, requeueing...This looks like an underlying problem causing the host
rdma_connectto timeout. Did it happen before or is it a new thing?
I can't say for sure, since until Christoph's recent fix, it was crashing in
other ways. I don't think the connection is failing at the driver level, but
I'll look into the stats for cxgb4 when it is in this state to see what is happening at that level.