IB/ipoib: Fix multicast join flow
v3 cover IB/ipoib: Fix multicast join flow
This patch series fixes the multicast join behavior problems introduced by my previous patchset. In particular, the original code did not use the send only join code from the multicast thread context, and so it did not need to restart the multicast thread. After my previous patchset, it does get called from the thread context, and so the send only join completion areas need to restart the join thread but they don't. This patchset makes them do so. It then adds in some cleanups for restarting the thread, and fixes the fact that one delayed join holds up the entire list of joins.
v3: Resend because the last send didn't register in patchworks properly
(because the subject-prefix was not on all of the emails, only the first) and because the Cc: list didn't not pass from cover letter to patches
v2: Added two new patches, the first creates a helper to restart the
multicast join thread and also adds using it in the two places where it should have been used but wasn't, the second allows the joins to proceed around a delayed join instead of stalling everything.
v1: Addressed the usage of the IPOIB_MCAST_RUN flag
Doug Ledford (3):
IB/ipoib: Fix failed multicast joins/sends IB/ipoib: Add a helper to restart the multicast task IB/ipoib: make delayed tasks not hold up everything
drivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 94 ++++++++++++++++++-------- 2 files changed, 66 insertions(+), 29 deletions(-)
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): cover.1421264928.git.dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
PATCH v3 1/3 IB/ipoib: Fix failed multicast joins/sends
The usage of IPOIB_MCAST_RUN as a flag is inconsistent. In some places it is used to mean "our device is administratively allowed to send multicast joins/leaves/packets" and in other places it means "our multicast join task thread is currently running and will process your request if you put it on the queue". However, this latter meaning is in fact flawed as there is a race condition between the join task testing the mcast list and finding it empty of remaining work, dropping the mcast mutex and also the priv->lock spinlock, and clearing the IPOIB_MCAST_RUN flag. Further, there are numerous locations that use the flag in the former fashion, and when all tasks complete and the task thread clears the RUN flag, all of those other locations will fail to ever again queue any work. This results in the interface coming up fine initially, but having problems adding new multicast groups after the first round of groups have all been added and the RUN flag is cleared by the join task thread when it thinks it is done. To resolve this issue, convert all locations in the code to treat the RUN flag as an indicator that the multicast portion of this interface is in fact administratively up and joins/leaves/sends can be performed. There is no harm (other than a slight performance penalty) to never clearing this flag and using it in this fashion as it simply means that a few places that used to micro-optimize how often this task was queued on a work queue will now queue the task a few extra times. We can address that suboptimal behavior in future patches.
--- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +5 -6
@@ -630,8 +630,6 @@ void ipoib_mcast_join_task(struct work_struct *work) }ipoib_dbg_mcast (priv,"successfully joined all multicast groups\n" ); - -clear_bit (IPOIB_MCAST_RUN , &priv->flags); }int ipoib_mcast_start_thread (struct net_device *dev)
@@ -641,8 +639,8 @@ int ipoib_mcast_start_thread(struct net_device *dev)ipoib_dbg_mcast (priv,"starting multicast thread\n" );mutex_lock (&mcast_mutex); -if (!test_and_set_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task,0 ); +set_bit (IPOIB_MCAST_RUN , &priv->flags); +queue_delayed_work (priv->wq, &priv->mcast_task,0 );mutex_unlock (&mcast_mutex);return 0 ;
@@ -725,7 +723,7 @@ void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)memcpy (mcast->mcmember.mgid.raw, mgid,sizeof (union ib_gid ));__ipoib_mcast_add (dev, mcast);list_add_tail (&mcast->list, &priv->multicast_list); -if (!test_and_set_bit (IPOIB_MCAST_RUN , &priv->flags)) +if (test_bit (IPOIB_MCAST_RUN , &priv->flags))queue_delayed_work (priv->wq, &priv->mcast_task,0 ); }
@@ -951,7 +949,8 @@ void ipoib_mcast_restart_task(struct work_struct *work)/* * Restart our join task if needed */ -ipoib_mcast_start_thread (dev); +if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) +queue_delayed_work (priv->wq, &priv->mcast_task,0 );rtnl_unlock (); }
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v3 2/3 IB/ipoib: Add a helper to restart the multicast task
In reply to a message not in the archiveAdd a simple helper to do the right thing when restarting the multicast thread. Call that helper from all the places that need to restart the thread. Add two places that didn't restart the thread, but should have.
--- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 46 ++++++++++++++++---------- 1 file changed, 29 insertions(+), 17 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +29 -17
@@ -66,6 +66,13 @@ struct ipoib_mcast_iter {unsigned int send_only; }; +static void __ipoib_mcast_continue_join_thread (struct ipoib_dev_priv *priv, +int delay) +{ +if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) +queue_delayed_work (priv->wq, &priv->mcast_task, delay); +} +static void ipoib_mcast_free (struct ipoib_mcast *mcast) {struct net_device *dev = mcast->dev;
@@ -270,6 +277,7 @@ ipoib_mcast_sendonly_join_complete(int status, {struct ipoib_mcast *mcast = multicast->context;struct net_device *dev = mcast->dev; +struct ipoib_dev_priv *priv =netdev_priv (dev);/* * We have to take the mutex to force mcast_sendonly_join to
@@ -309,6 +317,7 @@ out:complete (&mcast->done);if (status == -ENETRESET ) status =0 ; +__ipoib_mcast_continue_join_thread (priv,0 );mutex_unlock (&mcast_mutex);return status; }
@@ -360,6 +369,7 @@ static int ipoib_mcast_sendonly_join(struct ipoib_mcast *mcast)complete (&mcast->done);ipoib_warn (priv,"ib_sa_join_multicast for sendonly join " "failed (ret = %d)\n" , ret); +__ipoib_mcast_continue_join_thread (priv,0 ); }else {ipoib_dbg_mcast (priv,"no multicast record for %pI6, starting " "sendonly join\n" , mcast->mcmember.mgid.raw);
@@ -431,8 +441,7 @@ static int ipoib_mcast_join_complete(int status,if (!status) { mcast->backoff =1 ; -if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task,0 ); +__ipoib_mcast_continue_join_thread (priv,0 );/* * Defer carrier on work to priv->wq to avoid a
@@ -454,6 +463,13 @@ static int ipoib_mcast_join_complete(int status, mcast->backoff *=2 ;if (mcast->backoff >IPOIB_MAX_BACKOFF_SECONDS ) mcast->backoff =IPOIB_MAX_BACKOFF_SECONDS ; +/* +* XXX - This is wrong. *Our* join failed, but because the +* join thread does the joins in a serial fashion, if there +* are any joins behind ours waiting to complete, they should +* not be subjected to our backoff delay. +*/ +__ipoib_mcast_continue_join_thread (priv, mcast->backoff *HZ ); } out:spin_lock_irq (&priv->lock);
@@ -463,9 +479,6 @@ out:complete (&mcast->done);if (status == -ENETRESET ) status =0 ; -if (status &&test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task, - mcast->backoff *HZ );spin_unlock_irq (&priv->lock);mutex_unlock (&mcast_mutex);
@@ -532,10 +545,13 @@ static void ipoib_mcast_join(struct net_device *dev, struct ipoib_mcast *mcast, mcast->backoff *=2 ;if (mcast->backoff >IPOIB_MAX_BACKOFF_SECONDS ) mcast->backoff =IPOIB_MAX_BACKOFF_SECONDS ; - -if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task, - mcast->backoff *HZ ); +/* +* XXX - This is wrong. *Our* join failed, but because the +* join thread does the joins in a serial fashion, if there +* are any joins behind ours waiting to complete, they should +* not be subjected to our backoff delay. +*/ +__ipoib_mcast_continue_join_thread (priv, mcast->backoff *HZ ); }mutex_unlock (&mcast_mutex); }
@@ -573,9 +589,7 @@ void ipoib_mcast_join_task(struct work_struct *work)if (!broadcast) {ipoib_warn (priv,"failed to allocate broadcast group\n" );mutex_lock (&mcast_mutex); -if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task, -HZ ); +__ipoib_mcast_continue_join_thread (priv,HZ );mutex_unlock (&mcast_mutex);return ; }
@@ -723,8 +737,7 @@ void ipoib_mcast_send(struct net_device *dev, u8 *daddr, struct sk_buff *skb)memcpy (mcast->mcmember.mgid.raw, mgid,sizeof (union ib_gid ));__ipoib_mcast_add (dev, mcast);list_add_tail (&mcast->list, &priv->multicast_list); -if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task,0 ); +__ipoib_mcast_continue_join_thread (priv,0 ); }if (!mcast->ah) {
@@ -947,10 +960,9 @@ void ipoib_mcast_restart_task(struct work_struct *work)ipoib_mcast_free (mcast); }/* -* Restart our join task if needed +* Restart our join task thread if needed */ -if (test_bit (IPOIB_MCAST_RUN , &priv->flags)) -queue_delayed_work (priv->wq, &priv->mcast_task,0 ); +__ipoib_mcast_continue_join_thread (priv,0 );rtnl_unlock (); }
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v3 3/3 IB/ipoib: make delayed tasks not hold up everything
In reply to a message not in the archiveCurrently, the multicast join thread only processes one task at a time. It does this by having itself be scheduled as a delayed work item, when it runs, it finds one, and only one, piece of work to process, it then kicks that off via either the normal join process or the sendonly join process, and then it immediately exits. Both of those process chains are responsible for restarting the task when they have completed their specific action. This makes the entire join process serial with only one join supposedly ever outstanding at a time.
However, if we fail a join, and we need to initiate a backoff delay, that delay holds up the entire join process for all joins, not just the failed join. So modify the design such that we can have joins in delay, and the multicast thread will ignore them until their time comes around, and then we can process the rest of the queue without waiting for the delayed items.
--- drivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 85 +++++++++++++++++--------- 2 files changed, 56 insertions(+), 30 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib.h +1 -0
@@ -154,6 +154,7 @@ struct ipoib_mcast {unsigned long created;unsigned long backoff; +unsigned long delay_until;unsigned long flags;unsigned char logcount;
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +55 -30
166 lines not on this page · raw message
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Doug,
Thank you for the quick response.
Now I can see 2 issues, that I want to draw your attention to:
1. if there is a mcg that the driver failed to join, the mc_task enters to endless loop of re-queue, and the log will be full with the next messages:
[682560.569826] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.580136] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.590364] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.600504] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.610627] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.620769] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.631082] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.640835] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.651033] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.660758] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.670923] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.680676] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.690898] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.700630] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22
around 100 times a sec.
2. IPv6 still doesn't work for me, at the same case where it is not the first mcg in the list.
Thanks, Erez
-----Original Message-----
From: Doug Ledford [mailto:[email protected]]
Sent: Wednesday, January 14, 2015 9:53 PM
To: [email protected]; [email protected]
Cc: Amir Vadai; Eyal Perry; Erez Shitrit; Or Gerlitz; Doug Ledford
Subject: [PATCH V3 FIX For-3.19 0/3] IB/ipoib: Fix multicast join flow
This patch series fixes the multicast join behavior problems introduced by my previous patchset. In particular, the original code did not use the send only join code from the multicast thread context, and so it did not need to restart the multicast thread. After my previous patchset, it does get called from the thread context, and so the send only join completion areas need to restart the join thread but they don't. This patchset makes them do so. It then adds in some cleanups for restarting the thread, and fixes the fact that one delayed join holds up the entire list of joins.
v3: Resend because the last send didn't register in patchworks properly
(because the subject-prefix was not on all of the emails, only the first) and because the Cc: list didn't not pass from cover letter to patches
v2: Added two new patches, the first creates a helper to restart the
multicast join thread and also adds using it in the two places where it should have been used but wasn't, the second allows the joins to proceed around a delayed join instead of stalling everything.
v1: Addressed the usage of the IPOIB_MCAST_RUN flag
Doug Ledford (3):
IB/ipoib: Fix failed multicast joins/sends IB/ipoib: Add a helper to restart the multicast task IB/ipoib: make delayed tasks not hold up everything
drivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 94 ++++++++++++++++++-------- 2 files changed, 66 insertions(+), 29 deletions(-)
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): DBXPR05MB067182B666A7F24EE23DC8ADB64E0-c2uBOMY7wQg6ranl7A9sk9qRiQSDpxhJvxpqHgZTriW3zl9H0oFU5g@public.gmane.org
Quoted 22 lines · Erez Shitrit
Hi Doug,
Thank you for the quick response.
Now I can see 2 issues, that I want to draw your attention to:
1. if there is a mcg that the driver failed to join, the
mc_taskenters to endless loop of re-queue, and the log will be full with the next messages:[682560.569826] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.580136] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.590364] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.600504] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.610627] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.620769] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.631082] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.640835] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.651033] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.660758] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.670923] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.680676] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.690898] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.700630] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22
Quoted 1 line
around 100 times a sec.
OK, this looks like the send only joins that fail are not setting a fallback properly or something like that. There is a separate bug that I've isolated that I'm going to fix, then I we can see if that fix effects things here, as it very well might.
Quoted 1 line
2. IPv6 still doesn't work for me, at the same case where it is not the first mcg in the list.
Can you give me some sort of instructions on how to replicate your testing? Things are working for me here, but I don't have a complex IPv6 setup and mine may be too simple to reproduce what you are seeing.
Quoted 39 lines
Thanks, Erez
-----Original Message-----
From: Doug Ledford [mailto:[email protected]]
Sent: Wednesday, January 14, 2015 9:53 PM
To: [email protected]; [email protected]
Cc: Amir Vadai; Eyal Perry; Erez Shitrit; Or Gerlitz; Doug Ledford
Subject: [PATCH V3 FIX For-3.19 0/3] IB/ipoib: Fix multicast join flowThis patch series fixes the multicast join behavior problems introduced by my previous patchset. In particular, the original code did not use the send only join code from the multicast thread context, and so it did not need to restart the multicast thread. After my previous patchset, it does get called from the thread context, and so the send only join completion areas need to restart the join thread but they don't. This patchset makes them do so. It then adds in some cleanups for restarting the thread, and fixes the fact that one delayed join holds up the entire list of joins.
v3: Resend because the last send didn't register in patchworks properly
(because the subject-prefix was not on all of the emails, only the first) and because the Cc: list didn't not pass from cover letter to patchesv2: Added two new patches, the first creates a helper to restart the
multicast join thread and also adds using it in the two places where it should have been used but wasn't, the second allows the joins to proceed around a delayed join instead of stalling everything.v1: Addressed the usage of the
IPOIB_MCAST_RUNflagDoug Ledford (3):
IB/ipoib: Fix failed multicast joins/sends IB/ipoib: Add a helper to restart the multicast task IB/ipoib: make delayed tasks not hold up everythingdrivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 94 ++++++++++++++++++-------- 2 files changed, 66 insertions(+), 29 deletions(-)--
2.1.0--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Doug Ledford <[email protected]> GPG KeyID: 0E572FDD
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 29 lines · Doug Ledford
Quoted 22 lines · Erez Shitrit
Hi Doug,
Thank you for the quick response.
Now I can see 2 issues, that I want to draw your attention to:
1. if there is a mcg that the driver failed to join, the
mc_taskenters to endless loop of re-queue, and the log will be full with the next messages:[682560.569826] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.580136] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.590364] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.600504] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.610627] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.620769] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.631082] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.640835] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.651033] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.660758] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.670923] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.680676] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.690898] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.700630] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22Quoted 1 line
around 100 times a sec.
OK, this looks like the send only joins that fail are not setting a fallback properly or something like that. There is a separate bug that I've isolated that I'm going to fix, then I we can see if that fix effects things here, as it very well might.
Quoted 4 lines
Quoted 1 line
2. IPv6 still doesn't work for me, at the same case where it is not the first mcg in the list.
Can you give me some sort of instructions on how to replicate your testing? Things are working for me here, but I don't have a complex IPv6 setup and mine may be too simple to reproduce what you are seeing.
I don't have a complex setup, i have 2 devices, and i do a regular ping6 from device with the full series in it, to some other device. nothing special, the only thing i can say that in the list there is one sendonly mcg (
ff12:601b:ffff:0000:0000:0000:0000:0016) that is at the first place in the list.
anyway, i think it connected to the first issue,because it at some endless loop with the first mcg, it doesn't have the chance to handle the other mcg's.
Quoted 33 lines
Quoted 33 lines
Thanks, Erez
-----Original Message-----
From: Doug Ledford [mailto:[email protected]]
Sent: Wednesday, January 14, 2015 9:53 PM
To: [email protected]; [email protected]
Cc: Amir Vadai; Eyal Perry; Erez Shitrit; Or Gerlitz; Doug Ledford
Subject: [PATCH V3 FIX For-3.19 0/3] IB/ipoib: Fix multicast join flowThis patch series fixes the multicast join behavior problems introduced by my previous patchset. In particular, the original code did not use the send only join code from the multicast thread context, and so it did not need to restart the multicast thread. After my previous patchset, it does get called from the thread context, and so the send only join completion areas need to restart the join thread but they don't. This patchset makes them do so. It then adds in some cleanups for restarting the thread, and fixes the fact that one delayed join holds up the entire list of joins.
v3: Resend because the last send didn't register in patchworks properly
(because the subject-prefix was not on all of the emails, only the first) and because the Cc: list didn't not pass from cover letter to patchesv2: Added two new patches, the first creates a helper to restart the
multicast join thread and also adds using it in the two places where it should have been used but wasn't, the second allows the joins to proceed around a delayed join instead of stalling everything.v1: Addressed the usage of the
IPOIB_MCAST_RUNflagDoug Ledford (3):
IB/ipoib: Fix failed multicast joins/sends IB/ipoib: Add a helper to restart the multicast task IB/ipoib: make delayed tasks not hold up everythingdrivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 94 ++++++++++++++++++-------- 2 files changed, 66 insertions(+), 29 deletions(-)
Quoted 8 lines
Quoted 7 lines
--
2.1.0--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 54B81E2B.9030101-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
Quoted 34 lines · Erez Shitrit
Quoted 29 lines · Doug Ledford
Quoted 22 lines · Erez Shitrit
Hi Doug,
Thank you for the quick response.
Now I can see 2 issues, that I want to draw your attention to:
1. if there is a mcg that the driver failed to join, the
mc_taskenters to endless loop of re-queue, and the log will be full with the next messages:[682560.569826] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.580136] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.590364] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.600504] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.610627] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.620769] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.631082] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.640835] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.651033] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.660758] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.670923] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.680676] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22 [682560.690898] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting sendonly join [682560.700630] ib0: sendonly multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22Quoted 1 line
around 100 times a sec.
OK, this looks like the send only joins that fail are not setting a fallback properly or something like that. There is a separate bug that I've isolated that I'm going to fix, then I we can see if that fix effects things here, as it very well might.
Quoted 4 lines
Quoted 1 line
2. IPv6 still doesn't work for me, at the same case where it is not the first mcg in the list.
Can you give me some sort of instructions on how to replicate your testing? Things are working for me here, but I don't have a complex IPv6 setup and mine may be too simple to reproduce what you are seeing.
Quoted 7 lines
I don't have a complex setup, i have 2 devices, and i do a regular ping6 from device with the full series in it, to some other device. nothing special, the only thing i can say that in the list there is one sendonly mcg (
ff12:601b:ffff:0000:0000:0000:0000:0016) that is at the first place in the list.
anyway, i think it connected to the first issue,because it at some endless loop with the first mcg, it doesn't have the chance to handle the other mcg's.
OK, well, I have this all working here. However, there is still one lingering issue (not reported on this thread yet) that needs addressed, so I don't yet consider the patchset complete. But, I'll post it as it stands so far for you to try your tests again.
The outstanding issue is that it is possible for ipoib_mcast_flush_dev to race with ipoib_mcast_join and cause ipoib_mcast_join to oops. It's rare, I've only seen it once, but I was afraid that it was possible by looking at the code, and now I have confirmation that it is indeed possible. So, it needs to be fixed.
Quoted 46 lines
Quoted 33 lines
Quoted 33 lines
Thanks, Erez
-----Original Message-----
From: Doug Ledford [mailto:[email protected]]
Sent: Wednesday, January 14, 2015 9:53 PM
To: [email protected]; [email protected]
Cc: Amir Vadai; Eyal Perry; Erez Shitrit; Or Gerlitz; Doug Ledford
Subject: [PATCH V3 FIX For-3.19 0/3] IB/ipoib: Fix multicast join flowThis patch series fixes the multicast join behavior problems introduced by my previous patchset. In particular, the original code did not use the send only join code from the multicast thread context, and so it did not need to restart the multicast thread. After my previous patchset, it does get called from the thread context, and so the send only join completion areas need to restart the join thread but they don't. This patchset makes them do so. It then adds in some cleanups for restarting the thread, and fixes the fact that one delayed join holds up the entire list of joins.
v3: Resend because the last send didn't register in patchworks properly
(because the subject-prefix was not on all of the emails, only the first) and because the Cc: list didn't not pass from cover letter to patchesv2: Added two new patches, the first creates a helper to restart the
multicast join thread and also adds using it in the two places where it should have been used but wasn't, the second allows the joins to proceed around a delayed join instead of stalling everything.v1: Addressed the usage of the
IPOIB_MCAST_RUNflagDoug Ledford (3):
IB/ipoib: Fix failed multicast joins/sends IB/ipoib: Add a helper to restart the multicast task IB/ipoib: make delayed tasks not hold up everythingdrivers/infiniband/ulp/ipoib/ipoib.h | 1 + drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 94 ++++++++++++++++++-------- 2 files changed, 66 insertions(+), 29 deletions(-)Quoted 8 lines
Quoted 7 lines
--
2.1.0--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Doug Ledford <[email protected]> GPG KeyID: 0E572FDD
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v3 4/6 IB/ipoib: Handle -ENETRESET properly in our callback
When the core layer calls our callback with ENETRESET as the error, we clear the status to 0 before returning indicating that we are going to handle the error ourselves. This causes the core layer to not free the mcast->mc structure, but we are releasing our reference to it by clearing mcast->mc. So, preserve our reference to the multicast structure so when we next run ipoib_mcast_dev_flush, it will be able to properly release the mcast->mc entry at the right time in the right way.
--- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +6 -6
@@ -315,8 +315,10 @@ ipoib_mcast_sendonly_join_complete(int status,mutex_lock (&mcast_mutex);/* We trap for port events ourselves. */ -if (status == -ENETRESET ) +if (status == -ENETRESET ) { + status =0 ; goto out; + }if (!status) status =ipoib_mcast_join_finish (mcast, &multicast->rec);
@@ -344,8 +346,6 @@ out:if (status) mcast->mc =NULL ;complete (&mcast->done); -if (status == -ENETRESET ) - status =0 ;__ipoib_mcast_continue_join_thread (priv,NULL ,0 );mutex_unlock (&mcast_mutex);return status;
@@ -462,8 +462,10 @@ static int ipoib_mcast_join_complete(int status,mutex_lock (&mcast_mutex);/* We trap for port events ourselves. */ -if (status == -ENETRESET ) +if (status == -ENETRESET ) { + status =0 ; goto out; + }if (!status) status =ipoib_mcast_join_finish (mcast, &multicast->rec);
@@ -499,8 +501,6 @@ out:if (status) mcast->mc =NULL ;complete (&mcast->done); -if (status == -ENETRESET ) - status =0 ;spin_unlock_irq (&priv->lock);mutex_unlock (&mcast_mutex);
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): f0d0830949eb57626baa20a1d311b8e4b4f7768d.1421355536.git.dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org
PATCH v3 5/6 IB/ipoib: don't restart our thread on ENETRESET
Make the send only join completion handler behave like the normal join completion handler in that we should restart the multicast join thread whenever our join completes with either an error or success, but don't restart the thread if we got a net reset event, let ipoib_event queue up a device flush, which will then call ipoib_mcast_dev_flush to remove all of the current mcast entries, and ipoib_mcase_restart_task to check out current mcast entry list against the netdev list of mcast entries we are supposed to have and requeue the ones we need to get back.
--- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +2 -1
@@ -336,17 +336,18 @@ ipoib_mcast_sendonly_join_complete(int status,dev_kfree_skb_any (skb_dequeue (&mcast->pkt_queue)); }netif_tx_unlock_bh (dev); +__ipoib_mcast_continue_join_thread (priv, mcast,1 ); }else {/* Join completed, so reset any backoff parameters */ mcast->backoff =1 ; mcast->delay_until = jiffies; +__ipoib_mcast_continue_join_thread (priv,NULL ,0 ); } out:clear_bit (IPOIB_MCAST_FLAG_BUSY , &mcast->flags);if (status) mcast->mc =NULL ;complete (&mcast->done); -__ipoib_mcast_continue_join_thread (priv,NULL ,0 );mutex_unlock (&mcast_mutex);return status; }
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v3 6/6 IB/ipoib: remove unneeded locks
In reply to a message not in the archiveDuring the various work I've done on this, some extra locking has crept in when things weren't working right. This is one of those spots. Remove the unneeded spinlocks. The mutex is enough to protect against what we need to protect against.
--- drivers/infiniband/ulp/ipoib/ipoib_multicast.c | 2 -- 1 file changed, 2 deletions(-)
drivers/infiniband/ulp/ipoib/ipoib_multicast.c +0 -2
@@ -497,12 +497,10 @@ static int ipoib_mcast_join_complete(int status,__ipoib_mcast_continue_join_thread (priv, mcast,1 ); } out: -spin_lock_irq (&priv->lock);clear_bit (IPOIB_MCAST_FLAG_BUSY , &mcast->flags);if (status) mcast->mc =NULL ;complete (&mcast->done); -spin_unlock_irq (&priv->lock);mutex_unlock (&mcast_mutex);return status;
Signature · 6 lines
2.1.0 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html