IB/cm: use rwlock for MAD agent lock
PATCH 1/1 IB/cm: use rwlock for MAD agent lock
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.
This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.
The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.
Reproducer:
Server:
for i in $(seq 1 512); do
ucmatose -c 32 -p $((i + 5000)) &
done
Client:
for i in $(seq 1 512); do
ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &
done
--- drivers/infiniband/core/cm.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/infiniband/core/cm.c +8 -8
@@ -167,7 +167,7 @@ struct cm_port {struct cm_device {struct kref kref;struct list_head list; -spinlock_t mad_agent_lock; +rwlock_t mad_agent_lock;struct ib_device *ib_device;u8 ack_delay;int going_down;
@@ -285,7 +285,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)if (!cm_id_priv->av.port)return ERR_PTR (-EINVAL ); -spin_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); mad_agent = cm_id_priv->av.port->mad_agent;if (!mad_agent) { m =ERR_PTR (-EINVAL );
@@ -311,7 +311,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv) m->ah = ah; out: -spin_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);return m; }
@@ -1297,10 +1297,10 @@ static __be64 cm_form_tid(struct cm_id_private *cm_id_priv)if (!cm_id_priv->av.port)return cpu_to_be64 (low_tid); -spin_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);if (cm_id_priv->av.port->mad_agent) hi_tid = ((u64 )cm_id_priv->av.port->mad_agent->hi_tid) <<32 ; -spin_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);return cpu_to_be64 (hi_tid | low_tid); }
@@ -4378,7 +4378,7 @@ static int cm_add_one(struct ib_device *ib_device)return -ENOMEM ;kref_init (&cm_dev->kref); -spin_lock_init (&cm_dev->mad_agent_lock); +rwlock_init (&cm_dev->mad_agent_lock); cm_dev->ib_device = ib_device; cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay; cm_dev->going_down =0 ;
@@ -4494,9 +4494,9 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data)* The above ensures no call paths from the work are running, * the remaining paths all take the mad_agent_lock. */ -spin_lock (&cm_dev->mad_agent_lock); +write_lock (&cm_dev->mad_agent_lock); port->mad_agent =NULL ; -spin_unlock (&cm_dev->mad_agent_lock); +write_unlock (&cm_dev->mad_agent_lock);ib_unregister_mad_agent (mad_agent);ib_port_unregister_client_groups (ib_device, i, cm_counter_groups);
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Quoted 27 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
Since the lock appears to be only for protection from
cm_remove_one, it can be changed to a rwlock to resolve these issues.Reproducer:
Server:
for i in $(seq 1 512); doucmatose -c 32 -p $((i + 5000)) &done
Quoted 7 lines
Client:
for i in $(seq 1 512); doucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &done
Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")Signed-off-by Jacob Moroni <[email protected]>
SGTM, thanks.
RCU could probably be used here, if we expect the read_lock()/read_unlock() operations to happen in a fast path.
Jason Gunthorpe
maintainerQuoted 12 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.
Quoted 2 lines
The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
While the patch and fix seems reasonable, I'm somewhat surprised to see it.
If you are running at such a high workload then I'm shocked you don't hit all the other nasty problems with RDMA CM scalability?
Is the issue that the AH creation is very slow for some reason? It has been a longstanding peeve of mine that this is done under a spinlock context, I've long felt that should be reworked and some of those spinlocks converted to mutex's.
Jason
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.
In the link: https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock() / spin_unlock()
must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does
must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user
hardware interrupt is ok unless the interrupt handler may try to lock this spin lock spin lock not recursive: same thread locking twice will deadlock
keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.
I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Thanks a lot.
Zhu Yanjun
Quoted 91 lines
The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
Since the lock appears to be only for protection from
cm_remove_one, it can be changed to a rwlock to resolve these issues.Reproducer:
Server:
for i in $(seq 1 512); do ucmatose -c 32 -p $((i + 5000)) & doneClient:
for i in $(seq 1 512); do ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 & doneFixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")Signed-off-by Jacob Moroni <[email protected]>--- drivers/infiniband/core/cm.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)drivers/infiniband/core/cm.c +8
-8@@ -167,7 +167,7 @@ struct cm_port {struct cm_device {struct kref kref;struct list_head list; -spinlock_t mad_agent_lock; +rwlock_t mad_agent_lock;struct ib_device *ib_device;u8 ack_delay;int going_down;@@ -285,7 +285,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)if (!cm_id_priv->av.port)return ERR_PTR (-EINVAL ); -spin_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); mad_agent = cm_id_priv->av.port->mad_agent;if (!mad_agent) { m =ERR_PTR (-EINVAL );@@ -311,7 +311,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv) m->ah = ah; out: -spin_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);return m; }@@ -1297,10 +1297,10 @@ static __be64 cm_form_tid(struct cm_id_private *cm_id_priv)if (!cm_id_priv->av.port)return cpu_to_be64 (low_tid); -spin_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_lock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);if (cm_id_priv->av.port->mad_agent) hi_tid = ((u64 )cm_id_priv->av.port->mad_agent->hi_tid) <<32 ; -spin_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock); +read_unlock (&cm_id_priv->av.port->cm_dev->mad_agent_lock);return cpu_to_be64 (hi_tid | low_tid); }@@ -4378,7 +4378,7 @@ static int cm_add_one(struct ib_device *ib_device)return -ENOMEM ;kref_init (&cm_dev->kref); -spin_lock_init (&cm_dev->mad_agent_lock); +rwlock_init (&cm_dev->mad_agent_lock); cm_dev->ib_device = ib_device; cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay; cm_dev->going_down =0 ;@@ -4494,9 +4494,9 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data)* The above ensures no call paths from the work are running, * the remaining paths all take the mad_agent_lock. */ -spin_lock (&cm_dev->mad_agent_lock); +write_lock (&cm_dev->mad_agent_lock); port->mad_agent =NULL ; -spin_unlock (&cm_dev->mad_agent_lock); +write_unlock (&cm_dev->mad_agent_lock);ib_unregister_mad_agent (mad_agent);ib_port_unregister_client_groups (ib_device, i, cm_counter_groups);
Quoted 39 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock()/spin_unlock()must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock()prevents kernel preemption by ++preempt_count in uniprocessor, that’s allspin_lock()doesmust NOT call any function that can potentially sleep ex) kmalloc,
copy_from_userhardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlockkeep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.
Quoted 5 lines
I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Look at drivers/infiniband/core/cm.c
spin_lock_irqsave(&cm_id_priv->lock, flags);
-> Then call cm_alloc_msg() while hard IRQ are masked.
Quoted 2 lines
If you are running at such a high workload then I'm shocked you don't hit all the other nasty problems with RDMA CM scalability?
It could be that we just haven't hit those issues yet :)
This serialization was slowing things down so much that I think it has been masking some other issues. For instance, I just discovered a bug in rping's persistent server mode (I'll be sending a Github PR soon), which seems to be due to a race condition we started hitting after this fix.
Quoted 4 lines
Is the issue that the AH creation is very slow for some reason? It has been a longstanding peeve of mine that this is done under a spinlock context, I've long felt that should be reworked and some of those spinlocks converted to mutex's.
Yes, that's exactly it. We have fairly high tail latencies for creating address handles. By removing the serialization, we can at least take advantage of queueing, which seems to help a lot. It would be really great if this could move out of an atomic context.
Thanks,
Jake
Quoted 52 lines · Eric Dumazet
Quoted 39 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock()/spin_unlock()must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock()prevents kernel preemption by ++preempt_count in uniprocessor, that’s allspin_lock()doesmust NOT call any function that can potentially sleep ex) kmalloc,
copy_from_userhardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlockkeep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.Quoted 5 lines
I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Look at
drivers/infiniband/core/cm.c
spin_lock_irqsave(&cm_id_priv->lock, flags);-> Then call
cm_alloc_msg()while hard IRQ are masked.
Zhu Yanjun
Reviewed-byQuoted 43 lines · Eric Dumazet
Quoted 42 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock()/spin_unlock()must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock()prevents kernel preemption by ++preempt_count in uniprocessor, that’s allspin_lock()doesmust NOT call any function that can potentially sleep ex) kmalloc,
copy_from_userhardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlockkeep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
Quoted 6 lines
Quoted 2 lines
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Look at
drivers/infiniband/core/cm.c
spin_lock_irqsave(&cm_id_priv->lock, flags);
Thanks a lot. spin_lock_irq should be spin_lock_irqsave?
Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.
Anyway, read_lock should be a lighter lock than spin_lock.
Thanks,
Zhu Yanjun
Quoted 2 lines
-> Then call
cm_alloc_msg()while hard IRQ are masked.
Quoted 45 lines · Zhu Yanjun
Quoted 43 lines · Eric Dumazet
Quoted 42 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock()/spin_unlock()must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock()prevents kernel preemption by ++preempt_count in uniprocessor, that’s allspin_lock()doesmust NOT call any function that can potentially sleep ex) kmalloc,
copy_from_userhardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlockkeep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Look at
drivers/infiniband/core/cm.c
spin_lock_irqsave(&cm_id_priv->lock, flags);Thanks a lot.
spin_lock_irqshould bespin_lock_irqsave?
Both spin_lock_irq and spin_lock_irqsave are masking all interrupts.
Quoted 14 lines
Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.
Anyway,
read_lockshould be a lighter lock thanspin_lock.Thanks,
Reviewed-by Zhu Yanjun <[email protected]>Zhu Yanjun
Quoted 2 lines
-> Then call
cm_alloc_msg()while hard IRQ are masked.
Quoted 49 lines · Eric Dumazet
Quoted 48 lines · Zhu Yanjun
Quoted 46 lines · Eric Dumazet
Quoted 44 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...spin_lock()/spin_unlock()must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time
spin_lock()prevents kernel preemption by ++preempt_count in uniprocessor, that’s allspin_lock()doesmust NOT call any function that can potentially sleep ex) kmalloc,
copy_from_userhardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlockkeep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?
I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Look at
drivers/infiniband/core/cm.c
spin_lock_irqsave(&cm_id_priv->lock, flags);Thanks a lot.
spin_lock_irqshould bespin_lock_irqsave?Both
spin_lock_irqandspin_lock_irqsaveare masking all interrupts.
Maybe the commit log should be written clearly. Thanks.
Zhu Yanjun
Quoted 17 lines
Quoted 17 lines
Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.
Anyway,
read_lockshould be a lighter lock thanspin_lock.Thanks,
Reviewed-by Zhu Yanjun <[email protected]>Zhu Yanjun
Quoted 2 lines
-> Then call
cm_alloc_msg()while hard IRQ are masked.
Quoted 28 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
Since the lock appears to be only for protection from
cm_remove_one, it can be changed to a rwlock to resolve these issues.Reproducer:
Server:
for i in $(seq 1 512); doucmatose -c 32 -p $((i + 5000)) &done
Client:
for i in $(seq 1 512); do
Quoted 8 lines
ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &done
Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")Signed-off-by Jacob Moroni <[email protected]>--- drivers/infiniband/core/cm.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
Though I strongly encourage someone to change the spinlocks in this area to mutex :)
Jason
Quoted 31 lines · Jacob Moroni
In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for
mad_agent_lockincm_alloc_msg.This contention can occur while inside of a
spin_lock_irqregion, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization ofrdma_create_ahcalls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.
Since the lock appears to be only for protection from
cm_remove_one, it can be changed to a rwlock to resolve these issues.Reproducer:
Server:
for i in $(seq 1 512); doucmatose -c 32 -p $((i + 5000)) &done
Client:
for i in $(seq 1 512); doucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &done
Quoted 8 lines
Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")Signed-off-by Jacob Moroni <[email protected]>Acked-by Eric Dumazet <[email protected]>Reviewed-by Zhu Yanjun <[email protected]>Reviewed-by Jason Gunthorpe <[email protected]>--- drivers/infiniband/core/cm.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
Applied to for-next, thanks
Jason