Missing join mcast events causing full machine lockup

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

So Doug, what would
you say about the following as a proposed fix (not tested):

drivers/infiniband/ulp/ipoib/ipoib_multicast.c +1 -1

index 87799de90a1d..f6f15d36b02d 100644
@@ -947,7 +947,7 @@ void ipoib_mcast_restart_task(struct work_struct
 *work)
          */
         list_for_each_entry_safe(mcast, tmcast, &remove_list, list)
                 if (test_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags))
-                       wait_for_completion(&mcast->done);
+                       wait_for_completion_timeout(&mcast->done, 30
 * HZ);

        list_for_each_entry_safe(mcast, tmcast, &remove_list, list) { ipoib_mcast_leave(mcast->dev, mcast);

Given the loop afterwards which uses ipoib_mcast_(leave_free) that should work?
Looking at the code in ipoib_mcast_leave it seems we are going to trigger a warning,
which is preferable to putting the machine to a grinding halt?

Does the proposed patch break things horribly ?

Quoted 8 lines · Nikolay Borisov

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

[ snip long traces ]

Quoted 2 lines

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

Blindly relying on the join completions is not what we do. We are very careful to make sure we always have the right locking so that we never leave a join request in the BUSY state without running the completion at some time. If you are seeing us do that, then it means we have a bug in our locking or state processing. The answer then is to find that bug and not to paper over it with a timeout. Can you find some way to reproduce this with a 4.7 kernel?

Quoted 21 lines

So Doug, what would
you say about the following as a proposed fix (not tested):

drivers/infiniband/ulp/ipoib/ipoib_multicast.c +1 -1

index 87799de90a1d..f6f15d36b02d 100644
@@ -947,7 +947,7 @@ void ipoib_mcast_restart_task(struct work_struct
 *work)
         */
        list_for_each_entry_safe(mcast, tmcast, &remove_list, list)
                if (test_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags))
-                       wait_for_completion(&mcast->done);
+                       wait_for_completion_timeout(&mcast->done, 30
 * HZ);
 

        list_for_each_entry_safe(mcast, tmcast, &remove_list, list) {
                ipoib_mcast_leave(mcast->dev, mcast);

Quoted 7 lines

Given the loop afterwards which uses ipoib_mcast_(leave_free) that should work?
Looking at the code in ipoib_mcast_leave it seems we are going to trigger a warning,
which is preferable to putting the machine to a grinding halt?

Does the proposed patch break things horribly ?

It violates the intent of the join processing. And if we have the problem you are seeing, we really need to know if it's broken in IPoIB or deeper down in the core portion of the stack. Breaking out and continuing might be OK, but if we do, we are likely going to either leak something or have a use-after-free or something like that, so I would have to spend some time thinking about how things might go wrong and whether or not it's better to stop the machine when this happens, or continue and hope we don't corrupt memory somehow.

--
Doug Ledford <[email protected]>

              GPG KeyID: 0E572FDD

Quoted 19 lines · Doug Ledford

Quoted 10 lines · Nikolay Borisov

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to
missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

[ snip long traces ]

Quoted 3 lines

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

Quoted 12 lines

Blindly relying on the join completions is not what we do.  We are very
careful to make sure we always have the right locking so that we never
leave a join request in the BUSY state without running the completion
at some time.  If you are seeing us do that, then it means we have a
bug in our locking or state processing.  The answer then is to find that bug and not to paper over it with a timeout.  Can you find some
way to reproduce this with a 4.7 kernel?

Unfortunately my environment is constrained to 4.4 kernel. I will, however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and see
if something
shows up. And while I don't have a 100% reproducer for it I see those symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state that might be useful in debugging this i.e I can obtain crashdumps and
reverse the state of the ipoib stacks. I've seen this issue on 3.12 and on 4.4.
Some of my previous emails also show this manifesting in hangs in cm_destroy_id
as well. So clearly there is a problem there but it proves very elusive.

Quoted 34 lines · Nikolay Borisov

Quoted 19 lines · Doug Ledford

Quoted 10 lines · Nikolay Borisov

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to
missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

[ snip long traces ]

Quoted 3 lines

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

Quoted 12 lines

Blindly relying on the join completions is not what we do. We are very
careful to make sure we always have the right locking so that we never
leave a join request in the BUSY state without running the completion
at some time. If you are seeing us do that, then it means we have a
bug in our locking or state processing. The answer then is to find that bug and not to paper over it with a timeout. Can you find some
way to reproduce this with a 4.7 kernel?

Quoted 16 lines

Unfortunately my environment is constrained to 4.4 kernel. I will, however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and see
if something
shows up. And while I don't have a 100% reproducer for it I see those symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state that might be useful in debugging this i.e I can obtain crashdumps and
reverse the state of the ipoib stacks. I've seen this issue on 3.12 and on 4.4.
Some of my previous emails also show this manifesting in hangs in cm_destroy_id
as well. So clearly there is a problem there but it proves very elusive.

Can you give any clues as to what's causing it? Do you have link flap? SM bounces? Lots of multicast joins/leaves?

Quoted 52 lines · Doug Ledford

Quoted 34 lines · Nikolay Borisov

Quoted 19 lines · Doug Ledford

Quoted 10 lines · Nikolay Borisov

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to
missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

[ snip long traces ]

Quoted 3 lines

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

Quoted 12 lines

Blindly relying on the join completions is not what we do. We are very
careful to make sure we always have the right locking so that we never
leave a join request in the BUSY state without running the completion
at some time. If you are seeing us do that, then it means we have a
bug in our locking or state processing. The answer then is to find that bug and not to paper over it with a timeout. Can you find some
way to reproduce this with a 4.7 kernel?

Quoted 16 lines

Unfortunately my environment is constrained to 4.4 kernel. I will, however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and see
if something
shows up. And while I don't have a 100% reproducer for it I see those symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state that might be useful in debugging this i.e I can obtain crashdumps and
reverse the state of the ipoib stacks. I've seen this issue on 3.12 and on 4.4.
Some of my previous emails also show this manifesting in hangs in cm_destroy_id
as well. So clearly there is a problem there but it proves very elusive.

Quoted 2 lines

Can you give any clues as to what's causing it? Do you have link flap? SM bounces? Lots of multicast joins/leaves?

I spoke with the network admins and they said that the network is not flapping, we shouldn't have a lot of join/leaves since the network is not that big and is stable. E.g. once nodes joins they usually are not restarted.

Here are some messages which result after the said hangs happen:

Aug  1 04:53:51 node1 kernel: [29100.763267] ib0: Budget exhausted after napi rescheduled 
Jul 31 21:29:46 node1 kernel: [ 2457.666476] NETDEV WATCHDOG: ib0 (ib_qib): transmit queue 0 timed out
Jul 29 05:17:36 node1 kernel: [ 8797.968402] ib0: dev_queue_xmit failed to requeue packet
Jul 23 19:27:22 node1 kernel: ib0: packet len 2200 (> 2048) too long to send, dropping
Jul 25 01:01:52 node1 kernel: ib0: queue stopped 1, tx_head 124520708, tx_tail 124520580

Aug  2 10:05:26 node15 bird6: LocalIPv6: Socket error on ib0: No buffer space available

Also I'm being told that *sometimes* doing a remote port reset actually fixes the issue, but only sometimes as otherwise the port is completely inactive.

I realize this is not much information but this issue really just rears its ugly head out of nowhere and usually there doesn't seem to be that much information ;(

Quoted 2 lines

Quoted 34 lines · Nikolay Borisov

Quoted 19 lines · Doug Ledford

Quoted 10 lines · Nikolay Borisov

Hello,

With running the risk of sounding like a broken record, I came across
another case where ipoib can cause the machine to go haywire due to
missed join requests. This is on 4.4.14 kernel. Here is what I believe
happens:

[ snip long traces ]

Quoted 3 lines

This makes me wonder if using timeouts is actually better than blindly relying on completing the join.

Quoted 12 lines

Blindly relying on the join completions is not what we do. We are very
careful to make sure we always have the right locking so that we never
leave a join request in the BUSY state without running the completion
at some time. If you are seeing us do that, then it means we have a
bug in our locking or state processing. The answer then is to find that bug and not to paper over it with a timeout. Can you find some
way to reproduce this with a 4.7 kernel?

Quoted 16 lines

Unfortunately my environment is constrained to 4.4 kernel. I will, however,
try and check if I can get a couple of IB-enabled nodes on 4.7 and see
if something
shows up. And while I don't have a 100% reproducer for it I see those symptoms rather regularly
on production nodes. I'm able and happy to extract any runtime state that might be useful in debugging this i.e I can obtain crashdumps and
reverse the state of the ipoib stacks. I've seen this issue on 3.12 and on 4.4.
Some of my previous emails also show this manifesting in hangs in cm_destroy_id
as well. So clearly there is a problem there but it proves very elusive.

Can you give any clues as to what's causing it? Do you have link flap? SM bounces? Lots of multicast joins/leaves?