generic-ipi: add smp_mb() before sending the IPI

PATCH Applied Suresh Siddha 3 participants 5 messages Nov 4, 2008 – Nov 6, 2008 x86

Share & tools

Suresh Siddha

author

2008-11-04 22:08 UTC #0 lore raw

PATCH 1/3 generic-ipi: add smp_mb() before sending the IPI

smp_mb() (to make the memory operations visible globally) is needed before sending the ipi, if the receiver refers to the data(setup by the sender) in a lock-free fashion.

On x86, x2apic mode accesses for sending IPI's don't have serializing semantics. So the need for smp_mb() before sending the IPI becomes more critical in x2apic mode.

Signed-off-by Suresh Siddha <[email protected]>

---

kernel/smp.c +10 -0

Index: linux-2.6.git/kernel/smp.c
===================================================================
@@ -76,6 +76,11 @@
 	list_add_tail(&data->list, &dst->list);
 	spin_unlock_irqrestore(&dst->lock, flags);
 
+	/*
+	 * Make the list addition visible before sending the ipi.
+	 */
+	smp_mb();
+
 	if (ipi)
 		arch_send_call_function_single_ipi(cpu);

@@ -370,6 +375,11 @@
 	list_add_tail_rcu(&data->csd.list, &call_function_queue);
 	spin_unlock_irqrestore(&call_function_lock, flags);
 
+	/*
+	 * Make the list addition visible before sending the ipi.
+	 */
+	smp_mb();
+
 	/* Send a message to all CPUs in the map */
 	arch_send_call_function_ipi(mask);

Suresh Siddha

author

2008-11-04 22:08 UTC #1 lore raw

PATCH 2/3 x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

On x86, x2apic mode accesses for sending IPI's don't have serializing semantics. If the IPI receiver refers(in lock-free fashion) to some memory setup by the sender, the need for smp_mb() before sending the IPI becomes critical in x2apic mode.

Add the smp_mb() in native_flush_tlb_others() before sending the IPI.

Signed-off-by Suresh Siddha <[email protected]>

---

arch/x86/kernel/tlb_32.c +6 -0

Index: linux-2.6.git/arch/x86/kernel/tlb_32.c
===================================================================
@@ -154,6 +154,12 @@
 	flush_mm = mm;
 	flush_va = va;
 	cpus_or(flush_cpumask, cpumask, flush_cpumask);
+
+	/*
+	 * Make the above memory operations globally visible before
+	 * sending the IPI.
+	 */
+	smp_mb();
 	/*
 	 * We have to send the IPI only to
 	 * CPUs affected.

arch/x86/kernel/tlb_64.c +5 -0

Index: linux-2.6.git/arch/x86/kernel/tlb_64.c
===================================================================
@@ -183,6 +183,11 @@
 	cpus_or(f->flush_cpumask, cpumask, f->flush_cpumask);
 
 	/*
+	 * Make the above memory operations globally visible before
+	 * sending the IPI.
+	 */
+	smp_mb();
+	/*
 	 * We have to send the IPI only to
 	 * CPUs affected.
 	 */

Ingo Molnar

Applied

2008-11-06 08:42 UTC #2 lore raw

Re: [patch 2/3] x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

Quoted 9 lines · Suresh Siddha

On x86, x2apic mode accesses for sending IPI's don't have serializing semantics. If the IPI receiver refers(in lock-free fashion) to some memory setup by the sender, the need for smp_mb() before sending the IPI becomes critical in x2apic mode.
Add the smp_mb() in native_flush_tlb_others() before sending the IPI.
Signed-off-by Suresh Siddha <[email protected]>

applied to tip/x86/urgent, thanks Suresh!

Jens, you'll send 1/3 and 3/3 to Linus, right?

Ingo

Jens Axboe

2008-11-06 08:45 UTC #3 lore raw

Re: [patch 2/3] x86: Add smp_mb() before sending INVALIDATE_TLB_VECTOR

Quoted 8 lines · Ingo Molnar

Quoted 5 lines · Suresh Siddha
On x86, x2apic mode accesses for sending IPI's don't have serializing semantics. If the IPI receiver refers(in lock-free fashion) to some memory setup by the sender, the need for smp_mb() before sending the IPI becomes critical in x2apic mode.

Quoted 8 lines

Quoted 4 lines
Add the smp_mb() in native_flush_tlb_others() before sending the IPI.
Signed-off-by Suresh Siddha <[email protected]>
applied to tip/x86/urgent, thanks Suresh!
Jens, you'll send 1/3 and 3/3 to Linus, right?

Yes, it's in for-linus and I'll send the pull request today.

Signature · 1 line

Jens Axboe

Suresh Siddha

author

2008-11-04 22:08 UTC #4 lore raw

PATCH 3/3 generic-ipi: fix the smp_mb() usage

In reply to Suresh Siddha

Remove the unnecessary smp_mb() in csd_flag_wait(), as the presence of that smp_mb() doesn't mean anything on the sender, when the ipi receiver is not doing any thing special (like memory fence) after clearing the CSD_FLAG_WAIT.

IPI receiver (on Alpha atleast) needs smp_read_barrier_depends() in the handler before reading the call_single_queue list in a lock-free fashion. Replace the existing smp_mb() (which is unnecessary) with smp_read_barrier_depends()

Signed-off-by Suresh Siddha <[email protected]>

---

kernel/smp.c +2 -6

Index: linux-2.6.git/kernel/smp.c
===================================================================
@@ -51,10 +51,6 @@
 {
 	/* Wait for response */
 	do {
-		/*
-		 * We need to see the flags store in the IPI handler
-		 */
-		smp_mb();
 		if (!(data->flags & CSD_FLAG_WAIT))
 			break;
 		cpu_relax();

@@ -162,7 +158,7 @@
 	 * Need to see other stores to list head for checking whether
 	 * list is empty without holding q->lock
 	 */
-	smp_mb();
+	smp_read_barrier_depends();
 	while (!list_empty(&q->list)) {
 		unsigned int data_flags;

@@ -196,7 +192,7 @@
 		/*
 		 * See comment on outer loop
 		 */
-		smp_mb();
+		smp_read_barrier_depends();
 	}
 }