Skip to contentSkip to conversation

latency improvement in __smp_call_single_queue

PATCH In review George Prekas 2 participants 3 messages Sep 23, 2020 – Sep 24, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 latency improvement in __smp_call_single_queue

If an interrupt arrives between llist_add and send_call_function_single_ipi in the following code snippet, then the remote CPU will not receive the IPI in a timely manner and subsequent SMP calls even from other CPUs for other functions will be delayed:

    if (llist_add(node, &per_cpu(call_single_queue, cpu))) send_call_function_single_ipi(cpu);

Note: llist_add returns 1 if it was empty before the operation.

CPU 0                           | CPU 1                     | CPU 2 __smp_call_single_q(2,f1)       | __smp_call_single_q(2,f2) | llist_add returns 1           |                           | interrupted                   |   llist_add returns 0     | ...                       |   branch not taken        | ...                       |                           | resumed                       |                           | send_call_function_single_ipi |                           |

                                |                           | f1
                                |                           | f2

The call from CPU 1 for function f2 will be delayed because CPU 0 was interrupted.

---
 kernel/smp.c | 4 ++++

 1 file changed, 4 insertions(+)
kernel/smp.c +0 -0
index aa17eedff5be..9dc679466cf0 100644
@@ -135,6 +135,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(call_single_data_t, csd_data);
 

 void __smp_call_single_queue(int cpu, struct llist_node *node)
 {
+    unsigned long flags;
+
     /*
      * The list addition should be visible before sending the IPI
      * handler locks the list to pull the entry off it because of
@@ -146,8 +148,10 @@ void __smp_call_single_queue(int cpu, struct llist_node *node)
      * locking and barrier primitives. Generic code isn't really
      * equipped to do the right thing...
      */
+    local_irq_save(flags);
     if (llist_add(node, &per_cpu(call_single_queue, cpu))) send_call_function_single_ipi(cpu);
+    local_irq_restore(flags);
 }

 /*

Signature · 1 line
2.16.6
#1 lore raw
Quoted 18 lines · George Prekas

If an interrupt arrives between llist_add and send_call_function_single_ipi in the following code snippet, then the remote CPU will not receive the IPI in a timely manner and subsequent SMP calls even from other CPUs for other functions will be delayed:

    if (llist_add(node, &per_cpu(call_single_queue, cpu))) send_call_function_single_ipi(cpu);

Note: llist_add returns 1 if it was empty before the operation.

CPU 0                           | CPU 1                     | CPU 2 __smp_call_single_q(2,f1)       | __smp_call_single_q(2,f2) | llist_add returns 1           |                           | interrupted                   |   llist_add returns 0     | ...                       |   branch not taken        | ...                       |                           | resumed                       |                           | send_call_function_single_ipi |                           |

Quoted 5 lines
                                |                           | f1
                                |                           | f2

The call from CPU 1 for function f2 will be delayed because CPU 0 was interrupted.

Do you happen to have any actual numbers and a use-case where this was relevant?

#2 lore raw
Quoted 22 lines · On 9/24/2020 3:42 AM, [email protected] wrote:
Quoted 21 lines · George Prekas

If an interrupt arrives between llist_add and send_call_function_single_ipi in the following code snippet, then the remote CPU will not receive the IPI in a timely manner and subsequent SMP calls even from other CPUs for other functions will be delayed:

     if (llist_add(node, &per_cpu(call_single_queue, cpu))) send_call_function_single_ipi(cpu);

Note: llist_add returns 1 if it was empty before the operation.

CPU 0                           | CPU 1 | CPU 2 __smp_call_single_q(2,f1)       | __smp_call_single_q(2,f2) | llist_add returns 1           | |
   interrupted                   |   llist_add returns 0 | ...                       |   branch not taken | ...                       | |
   resumed                       | | send_call_function_single_ipi | |

                                 | | f1
                                 | | f2
Quoted 5 lines
Quoted 2 lines

The call from CPU 1 for function f2 will be delayed because CPU 0 was interrupted.

Do you happen to have any actual numbers and a use-case where this was relevant?

Hi Peter,

I encountered this problem while developing a device driver that used smp_call_function_single to communicate with other cores. I observed latency spikes and after investigation I figured out the problem described above.

I have written a simple device driver to validate the above fix. It does smp_call_function_single and measures the latency. I can post it here if it is appropriate. The latency impact is equal to the duration of the CPU 0's interruption.

--
George