Skip to contentSkip to conversation

IB/cm: use rwlock for MAD agent lock

PATCH Applied Jacob Moroni 4 participants 11 messages Feb 20, 2025 – Apr 7, 2025 InfiniBand

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 IB/cm: use rwlock for MAD agent lock

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.

Reproducer:

Server:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) &

done

Client:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &

done

Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")
---
 drivers/infiniband/core/cm.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/infiniband/core/cm.c +8 -8
index 142170473e75..effa53dd6800 100644
@@ -167,7 +167,7 @@ struct cm_port {
 struct cm_device {
 	struct kref kref;
 	struct list_head list;
-	spinlock_t mad_agent_lock;
+	rwlock_t mad_agent_lock;
 	struct ib_device *ib_device;
 	u8 ack_delay;
 	int going_down;
@@ -285,7 +285,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)
 	if (!cm_id_priv->av.port)
 		return ERR_PTR(-EINVAL);
 
-	spin_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
 	mad_agent = cm_id_priv->av.port->mad_agent;
 	if (!mad_agent) {
 		m = ERR_PTR(-EINVAL);
@@ -311,7 +311,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)
 	m->ah = ah;
 
 out:
-	spin_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
 	return m;
 }
 
@@ -1297,10 +1297,10 @@ static __be64 cm_form_tid(struct cm_id_private *cm_id_priv)
 	if (!cm_id_priv->av.port)
 		return cpu_to_be64(low_tid);
 
-	spin_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
 	if (cm_id_priv->av.port->mad_agent)
 		hi_tid = ((u64)cm_id_priv->av.port->mad_agent->hi_tid) << 32;
-	spin_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
 	return cpu_to_be64(hi_tid | low_tid);
 }
 
@@ -4378,7 +4378,7 @@ static int cm_add_one(struct ib_device *ib_device)
 		return -ENOMEM;
 
 	kref_init(&cm_dev->kref);
-	spin_lock_init(&cm_dev->mad_agent_lock);
+	rwlock_init(&cm_dev->mad_agent_lock);
 	cm_dev->ib_device = ib_device;
 	cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay;
 	cm_dev->going_down = 0;
@@ -4494,9 +4494,9 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data)
 		 * The above ensures no call paths from the work are running,
 		 * the remaining paths all take the mad_agent_lock.
 		 */
-		spin_lock(&cm_dev->mad_agent_lock);
+		write_lock(&cm_dev->mad_agent_lock);
 		port->mad_agent = NULL;
-		spin_unlock(&cm_dev->mad_agent_lock);
+		write_unlock(&cm_dev->mad_agent_lock);
 		ib_unregister_mad_agent(mad_agent);
 		ib_port_unregister_client_groups(ib_device, i,
 						 cm_counter_groups);
Signature · 1 line
2.48.1.601.g30ceb7b040-goog
Acked-by on v1 1/1
#1 lore raw
Quoted 27 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.

Reproducer:

Server:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) &

done

Quoted 7 lines

Client:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &

done

Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")

SGTM, thanks.

RCU could probably be used here, if we expect the read_lock()/read_unlock() operations to happen in a fast path.

maintainer
#2 lore raw
In reply to Jacob Moroni
Quoted 12 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

Quoted 2 lines

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

While the patch and fix seems reasonable, I'm somewhat surprised to see it.

If you are running at such a high workload then I'm shocked you don't hit all the other nasty problems with RDMA CM scalability?

Is the issue that the AH creation is very slow for some reason? It has been a longstanding peeve of mine that this is done under a spinlock context, I've long felt that should be reworked and some of those spinlocks converted to mutex's.

Jason

#3 lore raw
In reply to Jacob Moroni
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link: https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Thanks a lot.
Zhu Yanjun

Quoted 91 lines

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.

Reproducer:

Server:

   for i in $(seq 1 512); do
     ucmatose -c 32 -p $((i + 5000)) &
   done

Client:

   for i in $(seq 1 512); do
     ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &
   done
Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")
---
  drivers/infiniband/core/cm.c | 16 ++++++++--------
  1 file changed, 8 insertions(+), 8 deletions(-)
drivers/infiniband/core/cm.c +8 -8
index 142170473e75..effa53dd6800 100644
@@ -167,7 +167,7 @@ struct cm_port {
  struct cm_device {
  	struct kref kref;
  	struct list_head list;
-	spinlock_t mad_agent_lock;
+	rwlock_t mad_agent_lock;
  	struct ib_device *ib_device;
  	u8 ack_delay;
  	int going_down;
@@ -285,7 +285,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)
  	if (!cm_id_priv->av.port)
  		return ERR_PTR(-EINVAL);
 
-	spin_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
  	mad_agent = cm_id_priv->av.port->mad_agent;
  	if (!mad_agent) {
  		m = ERR_PTR(-EINVAL);
@@ -311,7 +311,7 @@ static struct ib_mad_send_buf *cm_alloc_msg(struct cm_id_private *cm_id_priv)
  	m->ah = ah;
 
  out:
-	spin_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
  	return m;
  }
 
@@ -1297,10 +1297,10 @@ static __be64 cm_form_tid(struct cm_id_private *cm_id_priv)
  	if (!cm_id_priv->av.port)
  		return cpu_to_be64(low_tid);
 
-	spin_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_lock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
  	if (cm_id_priv->av.port->mad_agent)
  		hi_tid = ((u64)cm_id_priv->av.port->mad_agent->hi_tid) << 32;
-	spin_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
+	read_unlock(&cm_id_priv->av.port->cm_dev->mad_agent_lock);
  	return cpu_to_be64(hi_tid | low_tid);
  }
 
@@ -4378,7 +4378,7 @@ static int cm_add_one(struct ib_device *ib_device)
  		return -ENOMEM;
 
  	kref_init(&cm_dev->kref);
-	spin_lock_init(&cm_dev->mad_agent_lock);
+	rwlock_init(&cm_dev->mad_agent_lock);
  	cm_dev->ib_device = ib_device;
  	cm_dev->ack_delay = ib_device->attrs.local_ca_ack_delay;
  	cm_dev->going_down = 0;
@@ -4494,9 +4494,9 @@ static void cm_remove_one(struct ib_device *ib_device, void *client_data)
  		 * The above ensures no call paths from the work are running,
  		 * the remaining paths all take the mad_agent_lock.
  		 */
-		spin_lock(&cm_dev->mad_agent_lock);
+		write_lock(&cm_dev->mad_agent_lock);
  		port->mad_agent = NULL;
-		spin_unlock(&cm_dev->mad_agent_lock);
+		write_unlock(&cm_dev->mad_agent_lock);
  		ib_unregister_mad_agent(mad_agent);
  		ib_port_unregister_client_groups(ib_device, i,
  						 cm_counter_groups);
#4 lore raw
Quoted 39 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

Quoted 5 lines

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Look at drivers/infiniband/core/cm.c

spin_lock_irqsave(&cm_id_priv->lock, flags);

-> Then call cm_alloc_msg() while hard IRQ are masked.

#5 lore raw
Quoted 2 lines

If you are running at such a high workload then I'm shocked you don't hit all the other nasty problems with RDMA CM scalability?

It could be that we just haven't hit those issues yet :)

This serialization was slowing things down so much that I think it has been masking some other issues. For instance, I just discovered a bug in rping's persistent server mode (I'll be sending a Github PR soon), which seems to be due to a race condition we started hitting after this fix.

Quoted 4 lines

Is the issue that the AH creation is very slow for some reason? It has been a longstanding peeve of mine that this is done under a spinlock context, I've long felt that should be reworked and some of those spinlocks converted to mutex's.

Yes, that's exactly it. We have fairly high tail latencies for creating address handles. By removing the serialization, we can at least take advantage of queueing, which seems to help a lot. It would be really great if this could move out of an atomic context.

Thanks,
Jake

Quoted 52 lines · Eric Dumazet
Quoted 39 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

Quoted 5 lines

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Look at drivers/infiniband/core/cm.c

spin_lock_irqsave(&cm_id_priv->lock, flags);

-> Then call cm_alloc_msg() while hard IRQ are masked.

Reviewed-by
#6 lore raw
In reply to Eric Dumazet
Quoted 43 lines · Eric Dumazet
Quoted 42 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

Quoted 6 lines
Quoted 2 lines

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Look at drivers/infiniband/core/cm.c

spin_lock_irqsave(&cm_id_priv->lock, flags);

Thanks a lot. spin_lock_irq should be spin_lock_irqsave?

Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.

Anyway, read_lock should be a lighter lock than spin_lock.

Thanks,

Zhu Yanjun

Quoted 2 lines

-> Then call cm_alloc_msg() while hard IRQ are masked.

#7 lore raw
Quoted 45 lines · Zhu Yanjun
Quoted 43 lines · Eric Dumazet
Quoted 42 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Look at drivers/infiniband/core/cm.c

spin_lock_irqsave(&cm_id_priv->lock, flags);

Thanks a lot. spin_lock_irq should be spin_lock_irqsave?

Both spin_lock_irq and spin_lock_irqsave are masking all interrupts.

Quoted 14 lines

Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.

Anyway, read_lock should be a lighter lock than spin_lock.

Thanks,

Zhu Yanjun

Quoted 2 lines

-> Then call cm_alloc_msg() while hard IRQ are masked.

#8 lore raw
Quoted 49 lines · Eric Dumazet
Quoted 48 lines · Zhu Yanjun
Quoted 46 lines · Eric Dumazet
Quoted 44 lines · Zhu Yanjun
Quoted 11 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

In the link:
https://www.cs.columbia.edu/~jae/4118-LAST/L12-interrupt-spinlock.html
"
...
spin_lock() / spin_unlock()

must not lose CPU while holding a spin lock, other threads will wait for the lock for a long time

spin_lock() prevents kernel preemption by ++preempt_count in uniprocessor, that’s all spin_lock() does

must NOT call any function that can potentially sleep ex) kmalloc, copy_from_user

hardware interrupt is ok unless the interrupt handler may try to lock this spin lock
spin lock not recursive: same thread locking twice will deadlock

keep the critical section as small as possible
...
"
And from the source code, it seems that spin_lock/spin_unlock are not related with interrupts.

I wonder why "leading to interrupts being disabled for extended durations on many cores" with spin_lock/spin_unlock?

I am not against this commit. I am just obvious why spin_lock/spin_unlock are related with "interrupts being disabled".

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Look at drivers/infiniband/core/cm.c

spin_lock_irqsave(&cm_id_priv->lock, flags);

Thanks a lot. spin_lock_irq should be spin_lock_irqsave?

Both spin_lock_irq and spin_lock_irqsave are masking all interrupts.

Maybe the commit log should be written clearly. Thanks.

Zhu Yanjun

Quoted 17 lines
Quoted 17 lines

Follow the steps of reproducer, I can not reproduce this problem on KVMs. Maybe I need a powerful host.

Anyway, read_lock should be a lighter lock than spin_lock.

Thanks,

Zhu Yanjun

Quoted 2 lines

-> Then call cm_alloc_msg() while hard IRQ are masked.

maintainer Reviewed-by on v1 1/1
#9 lore raw
In reply to Jacob Moroni
Quoted 28 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.

Reproducer:

Server:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) &

done

Client:
for i in $(seq 1 512); do

Quoted 8 lines
    ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &

done

Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")
---
 drivers/infiniband/core/cm.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

Though I strongly encourage someone to change the spinlocks in this area to mutex :)

Jason

maintainer Applied
#10 lore raw
In reply to Jacob Moroni
Quoted 31 lines · Jacob Moroni

In workloads where there are many processes establishing connections using RDMA CM in parallel (large scale MPI), there can be heavy contention for mad_agent_lock in cm_alloc_msg.

This contention can occur while inside of a spin_lock_irq region, leading to interrupts being disabled for extended durations on many cores. Furthermore, it leads to the serialization of rdma_create_ah calls, which has negative performance impacts for NICs which are capable of processing multiple address handle creations in parallel.

The end result is the machine becoming unresponsive, hung task warnings, netdev TX timeouts, etc.

Since the lock appears to be only for protection from cm_remove_one, it can be changed to a rwlock to resolve these issues.

Reproducer:

Server:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) &

done

Client:
for i in $(seq 1 512); do

    ucmatose -c 32 -p $((i + 5000)) -s 10.2.0.52 &

done

Quoted 8 lines
Fixes 76039ac9095f5ee5 ("IB/cm: Protect cm_dev, cm_ports and mad_agent with kref and lock")
---
 drivers/infiniband/core/cm.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

Applied to for-next, thanks

Jason