Skip to contentSkip to conversation

Possible process deadlock in RMPP flow

DISCUSSION Eli Cohen 6 participants 12 messages Sep 23, 2009 – Oct 20, 2009 6 messages not in the archive InfiniBand

lore
Share & tools
lore

Eli Cohen

author
#0 lore raw

Hi Sean,
one of our customers experiences problems when running ibnetdiscover. The problem happens from time to time.
Here is the call stack the he gets:

ibnetdiscover D ffffffff80149b8d 0 26968 26544
(L-TLB)
ffff8102c900bd88 0000000000000046 ffff81037e8e0000 ffff81037e8e02e8 ffff8102c900bd78 000000000000000a ffff8102c5b50820 ffff81038a929820 0000011837bf6105 0000000000000ede ffff8102c5b50a08 0000000100000000

Call Trace:
 [<ffffffff80064207>] wait_for_completion+0x79/0xa2
 [<ffffffff8008b4cc>] default_wake_function+0x0/0xe
 [<ffffffff882271d9>] :ib_mad:ib_cancel_rmpp_recvs+0x87/0xde
 [<ffffffff88224485>] :ib_mad:ib_unregister_mad_agent+0x30d/0x424
 [<ffffffff883983e9>] :ib_umad:ib_umad_close+0x9d/0xd6
 [<ffffffff80012e22>] __fput+0xae/0x198
 [<ffffffff80023de6>] filp_close+0x5c/0x64
 [<ffffffff800393df>] put_files_struct+0x63/0xae
 [<ffffffff80015b26>] do_exit+0x31c/0x911
 [<ffffffff8004971a>] cpuset_exit+0x0/0x6c
 [<ffffffff8005e116>] system_call+0x7e/0x83
Quoted 1 line

From the dump it seems that the process is waits on the call to

flush_workqueue() in ib_cancel_rmpp_recvs(). The package they use is OFED 1.4.2.

Do you have any idea or suggestions how to sort this out?

Thanks.

#1 lore raw
Quoted 18 lines

ibnetdiscover D ffffffff80149b8d 0 26968 26544
(L-TLB)
ffff8102c900bd88 0000000000000046 ffff81037e8e0000 ffff81037e8e02e8 ffff8102c900bd78 000000000000000a ffff8102c5b50820 ffff81038a929820 0000011837bf6105 0000000000000ede ffff8102c5b50a08 0000000100000000

Call Trace:
[<ffffffff80064207>] wait_for_completion+0x79/0xa2
[<ffffffff8008b4cc>] default_wake_function+0x0/0xe
[<ffffffff882271d9>] :ib_mad:ib_cancel_rmpp_recvs+0x87/0xde
[<ffffffff88224485>] :ib_mad:ib_unregister_mad_agent+0x30d/0x424
[<ffffffff883983e9>] :ib_umad:ib_umad_close+0x9d/0xd6
[<ffffffff80012e22>] __fput+0xae/0x198
[<ffffffff80023de6>] filp_close+0x5c/0x64
[<ffffffff800393df>] put_files_struct+0x63/0xae
[<ffffffff80015b26>] do_exit+0x31c/0x911
[<ffffffff8004971a>] cpuset_exit+0x0/0x6c
[<ffffffff8005e116>] system_call+0x7e/0x83
Quoted 3 lines

From the dump it seems that the process is waits on the call to flush_workqueue() in ib_cancel_rmpp_recvs(). The package they use is OFED 1.4.2.

Roland just submitted a patch in this area yesterday. I don't know if the patch would fix their issue, but it may be worth trying. What kernel does 1.4.2 map to?

What RMPP messages does ibnetdiscover use? If the program is completing successfully, there may be a different race with the rmpp cleanup. I'll see if anything else stands out in that area.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 7A32EEE20DF5432CADB60B8F8B1E0093-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org

#3 lore raw
Quoted 22 lines · Sean Hefty
Quoted 18 lines

ibnetdiscover D ffffffff80149b8d 0 26968 26544
(L-TLB)
ffff8102c900bd88 0000000000000046 ffff81037e8e0000 ffff81037e8e02e8 ffff8102c900bd78 000000000000000a ffff8102c5b50820 ffff81038a929820 0000011837bf6105 0000000000000ede ffff8102c5b50a08 0000000100000000

Call Trace:
[<ffffffff80064207>] wait_for_completion+0x79/0xa2
[<ffffffff8008b4cc>] default_wake_function+0x0/0xe
[<ffffffff882271d9>] :ib_mad:ib_cancel_rmpp_recvs+0x87/0xde
[<ffffffff88224485>] :ib_mad:ib_unregister_mad_agent+0x30d/0x424
[<ffffffff883983e9>] :ib_umad:ib_umad_close+0x9d/0xd6
[<ffffffff80012e22>] __fput+0xae/0x198
[<ffffffff80023de6>] filp_close+0x5c/0x64
[<ffffffff800393df>] put_files_struct+0x63/0xae
[<ffffffff80015b26>] do_exit+0x31c/0x911
[<ffffffff8004971a>] cpuset_exit+0x0/0x6c
[<ffffffff8005e116>] system_call+0x7e/0x83
Quoted 3 lines

From the dump it seems that the process is waits on the call to flush_workqueue() in ib_cancel_rmpp_recvs(). The package they use is OFED 1.4.2.

Quoted 7 lines

Roland just submitted a patch in this area yesterday. I don't know if the patch
would fix their issue, but it may be worth trying. What kernel does 1.4.2 map
to?

What RMPP messages does ibnetdiscover use?

None AFAIK.

-- Hal

Quoted 5 lines

If the program is completing
successfully, there may be a different race with the rmpp cleanup. I'll see if
anything else stands out in that area.

Quoted 7 lines

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

_______________________________________________
ewg mailing list [email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg

author
#4 lore raw
In reply to a message not in the archive
Quoted 4 lines · Sean Hefty

Roland just submitted a patch in this area yesterday. I don't know if the patch would fix their issue, but it may be worth trying. What kernel does 1.4.2 map to?

I think OFED 1.4.2 is based on kernel 2.6.27 but they're using RHEL 5.3.
Thanks, we'll try this.

Quoted 11 lines

What RMPP messages does ibnetdiscover use? If the program is completing successfully, there may be a different race with the rmpp cleanup. I'll see if anything else stands out in that area.

- Sean

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 3 lines · Eli Cohen
Quoted 1 line · Sean Hefty

What kernel does 1.4.2 map to?

I think OFED 1.4.2 is based on kernel 2.6.27 but they're using RHEL 5.3

Yes, the usual mess: ofed X is based on kernel Y1 but with some additions from kernel Y2 plus plenty of unreviwed and non-merged patches. Distro Z picks ofed X and the result is 99% unsupportable as Roland said. Somehow this ofed creature is still hanging around working on the the next damage its going to bring into this world (code name 1.5)

Eli, here's a little tip for you, I had the displeasure to resolve bunch of support cases originating from the fact that the below 2 years old commit missed some ofed version (sorry forgot the number...), maybe it would help you as well?

Under a normal setting, if this commit actually solves a bug being hit by many costumers, someone would have opened a distro bugzilla case saying, "please pick this commit for your kernel", the customers would have either wait for the next distro update or use a distro intermediate kernel. Currently, I understand that distros are picking ofed versions and that's it.

Or.

commit b61d92d8ae6aa13b17d1c31e69d123879cec2ee2

Author: Sean Hefty <[email protected]>
Date:   Fri Nov 30 17:30:18 2007 -0800

IB/mad: Fix incorrect access to items on local_list

A message that is not in the archive (sent off-list or not yet received): 4ABB13F3.1060702-hKgKHo2Ms0FWk0Htik3J/[email protected]

author
#7 lore raw
Quoted 7 lines · Or Gerlitz

commit b61d92d8ae6aa13b17d1c31e69d123879cec2ee2

Author: Sean Hefty <[email protected]>
Date:   Fri Nov 30 17:30:18 2007 -0800

IB/mad: Fix incorrect access to items on local_list

Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list.

#8 lore raw
Quoted 3 lines

Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list.

If ibnetdiscover doesn't use RMPP as Hal indicated, I don't think Roland's patch will help.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 53F5ED0C557B4667B22A27A0459B800B-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org

author
#10 lore raw
Quoted 6 lines · Sean Hefty
Quoted 3 lines

Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list.

If ibnetdiscover doesn't use RMPP as Hal indicated, I don't think Roland's patch will help.

Right, it doesn't help. Still it appears that ibnetdiscover triggers this problem and the lock seams to appear at ib_cancel_rmpp_recvs() waiting for flush_workqueue() to return. Do you know which apps or ULPs make use of RMPPs?
Any other ideas what this could be?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
In reply to Eli Cohen
Quoted 1 line · Eli Cohen

Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list.

Eli, did you find a commit that fixes the problem you reported on?

Or.

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 4AC8BE74.3050200-VPRAkNaXOzVS1MOuV/[email protected]

#14 lore raw
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

Thanks Or. This one is already in OFED 1.4.2 but apparently this is a different problem. Once I have information whether the patch Roland posted fixed it I will update the list.

Eli, did you find a commit that fixes the problem you reported on?

Or.

Not yet :-(

I can't find anything off in the code for this. It's odd, since unregister_mad_agent() does:

        flush_workqueue(port_priv->wq);
        ib_cancel_rmpp_recvs(mad_agent_priv);

and ib_cancel_rmpp_recvs() does:

        spin_lock_irqsave(&agent->lock, flags);
        list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
                cancel_delayed_work(&rmpp_recv->timeout_work);
                cancel_delayed_work(&rmpp_recv->cleanup_work);
        }
        spin_unlock_irqrestore(&agent->lock, flags);

        flush_workqueue(agent->qp_info->port_priv->wq);

which basically just flushes the same work queue.

I haven't been able to reproduce the problem, but I'm running the latest kernel
- not sure that matters in this case. Does ibnetdiscover just hang forever at the end of the test when this occurs? Is there any more information available?

- Sean

A message that is not in the archive (sent off-list or not yet received): BFC792E8570C48B8981C34F913243887-Zpru7NauK7drdx17CPfAsdBPR1lH4CV8@public.gmane.org

author
#17 lore raw
In reply to a message not in the archive
Quoted 18 lines · Sean Hefty

I can't find anything off in the code for this. It's odd, since unregister_mad_agent() does:

        flush_workqueue(port_priv->wq);
        ib_cancel_rmpp_recvs(mad_agent_priv);

and ib_cancel_rmpp_recvs() does:

        spin_lock_irqsave(&agent->lock, flags);
        list_for_each_entry(rmpp_recv, &agent->rmpp_list, list) {
                cancel_delayed_work(&rmpp_recv->timeout_work);
                cancel_delayed_work(&rmpp_recv->cleanup_work);
        }
        spin_unlock_irqrestore(&agent->lock, flags);

        flush_workqueue(agent->qp_info->port_priv->wq);
Quoted 6 lines

which basically just flushes the same work queue.

I haven't been able to reproduce the problem, but I'm running the latest kernel
- not sure that matters in this case. Does ibnetdiscover just hang forever at the end of the test when this occurs? Is there any more information available?

We are checking if the problem is a firmware bug, it looks like it. Once we verify this I will send an update.