State of ipoib cm mode

DISCUSSION Nikolay Borisov 4 participants 4 messages Jul 27, 2016 – Aug 1, 2016 7 messages not in the archive InfiniBand

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Nikolay Borisov

author

2016-07-27 12:05 UTC #3 lore raw

[Resending with the linux-rdma list cc'ed + some additional information]

Quoted 19 lines · Michael S. Tsirkin

Quoted 18 lines · Nikolay Borisov
Hello,
I've been running some production servers with ipoib cm but have observed various hangs, e.g. :
http://www.spinics.net/lists/linux-rdma/msg34577.html
http://www.spinics.net/lists/linux-rdma/msg37011.html
http://thread.gmane.org/gmane.linux.drivers.rdma/38899
Other people have also confirmed that there is a latent bug, which is very hard to debug (e.g. here:
http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
As the person who originally wrote the code and considering that git blame indicates most of it hasn't been touched does that mean it's considered stable? Also do you happen to have a hunch as to what might be causing such stalls?

Quoted 5 lines

Quoted 2 lines
Regards,
Nikolay
Please repost copying a mailing list.
I have a general policy against responding to off-list mail.

Ok.

In addition to that, here is the state of a node which has been hung for about 2 days now - no infiniband multicast connectivity, this is similar to the issue observed in the first mailing list entry I have referenced, but this time I managed to obtain the state of the ipoib_cm_rx and ib_cm_id structs (as well as any other structs which are referenced from those):

struct ipoib_cm_rx {
id = 0xffff8802128fa600,
qp = 0xffff880100e94000,
rx_ring = 0x0,

  list = {
    next = 0xffff88055f02bdd8,
    prev = 0xffff88055f02bdd8

},
dev = 0xffff880661f68000,
jiffies = 4367003834,
state = IPOIB_CM_RX_FLUSH,
recv_count = 0
}

struct ib_cm_id {
cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>, context = 0xffff880660f11780,
device = 0xffff8800378e4000,
service_id = 216172782113783824,
service_mask = 18446744073709551615,
state = IB_CM_IDLE,
lap_state = IB_CM_LAP_UNINIT,
local_id = 1741978561,
remote_id = 3782023797,
remote_cm_qpn = 1
}

And the backtrace is like that:

PID: 28224 TASK: ffff88064bdb5280 CPU: 5 COMMAND: "kworker/u24:2"
#0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
#1 [ffff88055f02bc70] schedule at ffffffff816103dc
#2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642 #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm] #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm] #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
[ib_ipoib]
#7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib] #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9 #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
#10 [ffff88055f02bed0] kthread at ffffffff8107161f
#11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff

ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);

Can you advise what other information might be helpful to debug this ?

Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Michael S. Tsirkin

2016-07-27 12:46 UTC #5 lore raw

Quoted 82 lines · Nikolay Borisov

[Resending with the linux-rdma list cc'ed + some additional information]
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Nikolay Borisov
Hello,
I've been running some production servers with ipoib cm but have observed various hangs, e.g. :
http://www.spinics.net/lists/linux-rdma/msg34577.html
http://www.spinics.net/lists/linux-rdma/msg37011.html
http://thread.gmane.org/gmane.linux.drivers.rdma/38899
Other people have also confirmed that there is a latent bug, which is very hard to debug (e.g. here:
http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
As the person who originally wrote the code and considering that git blame indicates most of it hasn't been touched does that mean it's considered stable? Also do you happen to have a hunch as to what might be causing such stalls?
Quoted 5 lines
Quoted 2 lines
Regards,
Nikolay
Please repost copying a mailing list.
I have a general policy against responding to off-list mail.
Ok.
In addition to that, here is the state of a node which has been hung for about 2 days now - no infiniband multicast connectivity, this is similar to the issue observed in the first mailing list entry I have referenced, but this time I managed to obtain the state of the ipoib_cm_rx and ib_cm_id structs (as well as any other structs which are referenced from those):
struct ipoib_cm_rx {
id = 0xffff8802128fa600,
qp = 0xffff880100e94000,
rx_ring = 0x0,
  list = {
    next = 0xffff88055f02bdd8,
    prev = 0xffff88055f02bdd8
},
dev = 0xffff880661f68000,
jiffies = 4367003834,
state = IPOIB_CM_RX_FLUSH,
recv_count = 0
}
struct ib_cm_id {
cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>, context = 0xffff880660f11780,
device = 0xffff8800378e4000,
service_id = 216172782113783824,
service_mask = 18446744073709551615,
state = IB_CM_IDLE,
lap_state = IB_CM_LAP_UNINIT,
local_id = 1741978561,
remote_id = 3782023797,
remote_cm_qpn = 1
}
And the backtrace is like that:
PID: 28224 TASK: ffff88064bdb5280 CPU: 5 COMMAND: "kworker/u24:2"
#0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
#1 [ffff88055f02bc70] schedule at ffffffff816103dc
#2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642 #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm] #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm] #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
[ib_ipoib]
#7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib] #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9 #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
#10 [ffff88055f02bed0] kthread at ffffffff8107161f
#11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff

Quoted 6 lines

ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
Can you advise what other information might be helpful to debug this ?
Regards,
Nikolay

I haven't looked at infiniband for ages, and won't be able to help you much. The links provided seem to indicate issues when SM or CM is not responsive. Try introducing delays by pausing the SM once in a while, or dropping packets to/from SM, or CM packets. Maybe add a mode that drops some of these packets once in a while.

Signature · 5 lines

MST
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAH_0vi9btY8PaDXnZOoP98hW_ZKkb6q11Fs+bEr_1VgRFgZjFQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): CAH_0vi9btY8PaDXnZOoP98hW_ZKkb6q11Fs+bEr_1VgRFgZjFQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Nikolay Borisov

2016-07-27 16:35 UTC #8 lore raw

Quoted 8 lines · Serge Ryabchun

Hi Nikolay,
very similar behavior we have experienced an half a year ago with CX2 and CX3 and QDR Mellanox switches.
It was fixed by this patch -
http://www.spinics.net/lists/linux-rdma/msg23811.html. Not fixed really but at least it can move multicast QP from SQE to RTS state and restore connectivity.

Thanks for chiming in. According to git describe this patch made it to 4.1 and the kernel I'm using is 4.4. So in my case this behavior is happening despite this patch being applied. One other element is that I'm seeing this with qlogic cards (ib_qib driver). Unfortunately I'm not able to pinpoint whether this is a problem of the card driver or with the middleware ib_ipoib driver.

Quoted 6 lines

It really was fixed replacing PSUs in the chassis by the more powerful. It appeared that Mellanox ASIC is very sensitive to the power. Under heavy loading those PSUs became slightly unstable and as a result built-in switch on the same PSUs produced damaged frames.

Quoted 101 lines

--
Regards,
Serge
Quoted 93 lines · Nikolay Borisov
[Resending with the linux-rdma list cc'ed + some additional information]
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Nikolay Borisov
Hello,
I've been running some production servers with ipoib cm but have observed various hangs, e.g. :
http://www.spinics.net/lists/linux-rdma/msg34577.html
http://www.spinics.net/lists/linux-rdma/msg37011.html
http://thread.gmane.org/gmane.linux.drivers.rdma/38899
Other people have also confirmed that there is a latent bug, which is very hard to debug (e.g. here:
http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
As the person who originally wrote the code and considering that git blame indicates most of it hasn't been touched does that mean it's considered stable? Also do you happen to have a hunch as to what might be causing such stalls?
Quoted 5 lines
Quoted 2 lines
Regards,
Nikolay
Please repost copying a mailing list.
I have a general policy against responding to off-list mail.
Ok.
In addition to that, here is the state of a node which has been hung for about 2 days now - no infiniband multicast connectivity, this is similar to the issue observed in the first mailing list entry I have referenced, but this time I managed to obtain the state of the ipoib_cm_rx and ib_cm_id structs (as well as any other structs which are referenced from those):
struct ipoib_cm_rx {
id = 0xffff8802128fa600,
qp = 0xffff880100e94000,
rx_ring = 0x0,
  list = {
    next = 0xffff88055f02bdd8,
    prev = 0xffff88055f02bdd8
},
dev = 0xffff880661f68000,
jiffies = 4367003834,
state = IPOIB_CM_RX_FLUSH,
recv_count = 0
}
struct ib_cm_id {
cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>, context = 0xffff880660f11780,
device = 0xffff8800378e4000,
service_id = 216172782113783824,
service_mask = 18446744073709551615,
state = IB_CM_IDLE,
lap_state = IB_CM_LAP_UNINIT,
local_id = 1741978561,
remote_id = 3782023797,
remote_cm_qpn = 1
}
And the backtrace is like that:
PID: 28224 TASK: ffff88064bdb5280 CPU: 5 COMMAND: "kworker/u24:2"
#0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
#1 [ffff88055f02bc70] schedule at ffffffff816103dc
#2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642 #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm] #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm] #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
[ib_ipoib]
#7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib] #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9 #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
#10 [ffff88055f02bed0] kthread at ffffffff8107161f
#11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff
ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
Can you advise what other information might be helpful to debug this ?
Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): CAJFSNy7seqPM3rF_OaXQnTGgTBTNTAYjXCxkPHBs11Od__8P0g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

Erez Shitrit

2016-08-01 08:08 UTC #10 lore raw

Hi Nikolay,

There are few issues here, the tx_timeout is part of the error flow of the driver, it happened mostly because the HW is much faster than the SW/CPU, it is temporary occasion that should be freed after a short time (not including bugs ..)

according your issue, please check with your HW vendor, i think that it misses a completion.
that can lead to a livelock ..

Thanks, Erez

Quoted 125 lines · Nikolay Borisov

Quoted 8 lines · Serge Ryabchun
Hi Nikolay,
very similar behavior we have experienced an half a year ago with CX2 and CX3 and QDR Mellanox switches.
It was fixed by this patch -
http://www.spinics.net/lists/linux-rdma/msg23811.html. Not fixed really but at least it can move multicast QP from SQE to RTS state and restore connectivity.
Thanks for chiming in. According to git describe this patch made it to 4.1 and the kernel I'm using is 4.4. So in my case this behavior is happening despite this patch being applied. One other element is that I'm seeing this with qlogic cards (ib_qib driver). Unfortunately I'm not able to pinpoint whether this is a problem of the card driver or with the middleware ib_ipoib driver.
Quoted 6 lines
It really was fixed replacing PSUs in the chassis by the more powerful. It appeared that Mellanox ASIC is very sensitive to the power. Under heavy loading those PSUs became slightly unstable and as a result built-in switch on the same PSUs produced damaged frames.
Quoted 101 lines
--
Regards,
Serge
Quoted 93 lines · Nikolay Borisov
[Resending with the linux-rdma list cc'ed + some additional information]
Quoted 19 lines · Michael S. Tsirkin
Quoted 18 lines · Nikolay Borisov
Hello,
I've been running some production servers with ipoib cm but have observed various hangs, e.g. :
http://www.spinics.net/lists/linux-rdma/msg34577.html
http://www.spinics.net/lists/linux-rdma/msg37011.html
http://thread.gmane.org/gmane.linux.drivers.rdma/38899
Other people have also confirmed that there is a latent bug, which is very hard to debug (e.g. here:
http://www.spinics.net/lists/linux-rdma/msg37022.html). Essentially
As the person who originally wrote the code and considering that git blame indicates most of it hasn't been touched does that mean it's considered stable? Also do you happen to have a hunch as to what might be causing such stalls?
Quoted 5 lines
Quoted 2 lines
Regards,
Nikolay
Please repost copying a mailing list.
I have a general policy against responding to off-list mail.
Ok.
In addition to that, here is the state of a node which has been hung for about 2 days now - no infiniband multicast connectivity, this is similar to the issue observed in the first mailing list entry I have referenced, but this time I managed to obtain the state of the ipoib_cm_rx and ib_cm_id structs (as well as any other structs which are referenced from those):
struct ipoib_cm_rx {
id = 0xffff8802128fa600,
qp = 0xffff880100e94000,
rx_ring = 0x0,
  list = {
    next = 0xffff88055f02bdd8,
    prev = 0xffff88055f02bdd8
},
dev = 0xffff880661f68000,
jiffies = 4367003834,
state = IPOIB_CM_RX_FLUSH,
recv_count = 0
}
struct ib_cm_id {
cm_handler = 0xffffffffa01e7b60 <ipoib_cm_rx_handler>, context = 0xffff880660f11780,
device = 0xffff8800378e4000,
service_id = 216172782113783824,
service_mask = 18446744073709551615,
state = IB_CM_IDLE,
lap_state = IB_CM_LAP_UNINIT,
local_id = 1741978561,
remote_id = 3782023797,
remote_cm_qpn = 1
}
And the backtrace is like that:
PID: 28224 TASK: ffff88064bdb5280 CPU: 5 COMMAND: "kworker/u24:2"
#0 [ffff88055f02bc28] __schedule at ffffffff8160fc6a
#1 [ffff88055f02bc70] schedule at ffffffff816103dc
#2 [ffff88055f02bc88] schedule_timeout at ffffffff81613642 #3 [ffff88055f02bd08] wait_for_completion at ffffffff816118df #4 [ffff88055f02bd68] cm_destroy_id at ffffffffa01d3759 [ib_cm] #5 [ffff88055f02bdc0] ib_destroy_cm_id at ffffffffa01d3a10 [ib_cm] #6 [ffff88055f02bdd0] ipoib_cm_free_rx_reap_list at ffffffffa01e7675
[ib_ipoib]
#7 [ffff88055f02be18] ipoib_cm_rx_reap at ffffffffa01e7705 [ib_ipoib] #8 [ffff88055f02be28] process_one_work at ffffffff8106bdf9 #9 [ffff88055f02be68] worker_thread at ffffffff8106c4a9
#10 [ffff88055f02bed0] kthread at ffffffff8107161f
#11 [ffff88055f02bf50] ret_from_fork at ffffffff816149ff
ffffffffa01d3759 is wait_for_completion(&cm_id_priv->comp);
Can you advise what other information might be helpful to debug this ?
Regards,
Nikolay
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html