Skip to contentSkip to conversation

x86: Voluntary leave_mm before entering ACPI C3

PATCH Applied Venki Pallipadi 5 participants 15 messages Dec 19, 2007 – Dec 20, 2007

Subsystems (2)
x86 ACPI
lore
Share & tools
lore

Venki Pallipadi

author
#0 lore raw

PATCH 1/1 x86: Voluntary leave_mm before entering ACPI C3

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

On a 8 logical CPU system, running make -j2, the number of tlbflush IPIs goes down from 40 per second to ~ 0. Total number of interrupts during the run of this workload was ~1200 per second, which makes it ~3% savings in wakeups.

There was no measurable performance or power impact however.

Signed-off-by Venkatesh Pallipadi <[email protected]>
arch/x86/kernel/smp_64.c +1 -1
Index: linux-2.6.24-rc/arch/x86/kernel/smp_64.c
===================================================================
@@ -70,7 +70,7 @@ static DEFINE_PER_CPU(union smp_flush_st
  * We cannot call mmdrop() because we are in interrupt context, 
  * instead update mm->cpu_vm_mask.
  */
-static inline void leave_mm(int cpu)
+void leave_mm(int cpu)
 {
 	if (read_pda(mmu_state) == TLBSTATE_OK)
 		BUG();
include/asm-x86/acpi_32.h +3 -0
Index: linux-2.6.24-rc/include/asm-x86/acpi_32.h
===================================================================
@@ -31,6 +31,7 @@
 #include <acpi/pdc_intel.h>
 
 #include <asm/system.h>		/* defines cmpxchg */
+#include <asm/mmu.h>
 
 #define COMPILER_DEPENDENT_INT64   long long
 #define COMPILER_DEPENDENT_UINT64  unsigned long long

@@ -138,6 +139,8 @@ static inline void disable_acpi(void) { 
 
 #define ARCH_HAS_POWER_INIT	1
 
+#define acpi_unlazy_tlb(x)	leave_mm(x)
+
 #endif /*__KERNEL__*/
 
 #endif /*_ASM_ACPI_H*/
include/asm-x86/acpi_64.h +3 -0
Index: linux-2.6.24-rc/include/asm-x86/acpi_64.h
===================================================================
@@ -30,6 +30,7 @@
 
 #include <acpi/pdc_intel.h>
 #include <asm/numa.h>
+#include <asm/mmu.h>
 
 #define COMPILER_DEPENDENT_INT64   long long
 #define COMPILER_DEPENDENT_UINT64  unsigned long long

@@ -148,6 +149,8 @@ static inline void acpi_fake_nodes(const
 }
 #endif
 
+#define acpi_unlazy_tlb(x)	leave_mm(x)
+
 #endif /*__KERNEL__*/
 
 #endif /*_ASM_ACPI_H*/
drivers/acpi/processor_idle.c +2 -0
Index: linux-2.6.24-rc/drivers/acpi/processor_idle.c
===================================================================
@@ -530,6 +530,7 @@ static void acpi_processor_idle(void)
 		break;
 
 	case ACPI_STATE_C3:
+		acpi_unlazy_tlb(smp_processor_id());
 		/*
 		 * disable bus master
 		 * bm_check implies we need ARB_DIS

@@ -1485,6 +1486,7 @@ static int acpi_idle_enter_bm(struct cpu
 		return 0;
 	}
 
+	acpi_unlazy_tlb(smp_processor_id());
 	/*
 	 * Must be done before busmaster disable as we might need to
 	 * access HPET !
include/asm-ia64/acpi.h +2 -0
Index: linux-2.6.24-rc/include/asm-ia64/acpi.h
===================================================================
@@ -126,6 +126,8 @@ extern int __devinitdata pxm_to_nid_map[
 extern int __initdata nid_to_pxm_map[MAX_NUMNODES];
 #endif
 
+#define acpi_unlazy_tlb(x)
+
 #endif /*__KERNEL__*/
 
 #endif /*_ASM_ACPI_H*/
include/asm-x86/mmu.h +2 -0
Index: linux-2.6.24-rc/include/asm-x86/mmu.h
===================================================================
@@ -20,4 +20,6 @@ typedef struct { 
 	void *vdso;
 } mm_context_t;
 
+void leave_mm(int cpu);
+
 #endif /* _ASM_X86_MMU_H */
arch/x86/kernel/smp_32.c +1 -1
Index: linux-2.6.24-rc/arch/x86/kernel/smp_32.c
===================================================================
@@ -256,7 +256,7 @@ static DEFINE_SPINLOCK(tlbstate_lock);
  * We need to reload %cr3 since the page tables may be going
  * away from under us..
  */
-void leave_mm(unsigned long cpu)
+void leave_mm(int cpu)
 {
 	if (per_cpu(cpu_tlbstate, cpu).state == TLBSTATE_OK)
 		BUG();
include/asm-x86/mmu_context_32.h +0 -2
Index: linux-2.6.24-rc/include/asm-x86/mmu_context_32.h
===================================================================
@@ -32,8 +32,6 @@ static inline void enter_lazy_tlb(struct
 #endif
 }
 
-void leave_mm(unsigned long cpu);
-
 static inline void switch_mm(struct mm_struct *prev,
 			     struct mm_struct *next,
 			     struct task_struct *tsk)
Applied
#1 lore raw
Quoted 7 lines · Venki Pallipadi

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

Quoted 6 lines

On a 8 logical CPU system, running make -j2, the number of tlbflush IPIs goes down from 40 per second to ~ 0. Total number of interrupts during the run of this workload was ~1200 per second, which makes it ~3% savings in wakeups.

There was no measurable performance or power impact however.

thanks, applied to x86.git. Nice and elegant patch!

Btw., since the TLB flush state machine is really subtle and fragile, could you try to run the following mmap stresstest i wrote some time ago:

http://redhat.com/~mingo/threaded-mmap-stresstest/

for a couple of hours. It runs nr_cpus threads which then do a "random crazy mix" of mappings/unmappings/remappings of a 800 MB memory window. The more sockets/cores, the crazier the TLB races get ;-)

Ingo

reviewer
#2 lore raw
Quoted 6 lines · Ingo Molnar
Quoted 4 lines · Venki Pallipadi

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

Are there any CPUs around which *don't* flush the TLB across C3? (I guess it's not guaranteed by the spec, though, and as TLBs grow larger there might be incentive to keep them online.)

-hpa

#3 lore raw
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Venki Pallipadi

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

Are there any CPUs around which *don't* flush the TLB across C3? (I guess it's not guaranteed by the spec, though, and as TLBs grow larger there might be incentive to keep them online.)

i dont think it's required for C3 to even turn off any portion of the CPU - if an interrupt arrives after the C3 sequence is initiated but just before dirty cachelines have been flushed then the CPU can just return without touching anything (such as the TLB) - right? So i dont think there's any implicit guarantee of TLB flushing (nor should there be), but in practice, a good C3 sequence would (statistically) turn off large portions of the CPU and hence the TLB as well.

Ingo

reviewer
#4 lore raw
Quoted 3 lines · Ingo Molnar

i dont think it's required for C3 to even turn off any portion of the CPU - if an interrupt arrives after the C3 sequence is initiated but just before dirty cachelines have been flushed then the CPU can just return without touching anything (such as the TLB) - right? So i dont think there's any implicit guarantee of TLB flushing (nor should there be), but in practice, a good C3 sequence would (statistically) turn off large portions of the CPU and hence the TLB as well.

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

Otherwise, what you say here of course is absolutely correct.

-hpa

author
#5 lore raw
Quoted 11 lines · H. Peter Anvin
Quoted 9 lines · Ingo Molnar

i dont think it's required for C3 to even turn off any portion of the CPU - if an interrupt arrives after the C3 sequence is initiated but just before dirty cachelines have been flushed then the CPU can just return without touching anything (such as the TLB) - right? So i dont think there's any implicit guarantee of TLB flushing (nor should there be), but in practice, a good C3 sequence would (statistically) turn off large portions of the CPU and hence the TLB as well.

Quoted 6 lines

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

Otherwise, what you say here of course is absolutely correct.

C3 does not guarantee all cache contents. Infact, atleast on Intel, L1 will be almost always flushed. Newer more power efficient CPUs does dynamic cache sizing [1]

C3 just guarantees that the caches are coherent. That is, if they are intact, then DMA will keep cache consistent.

Thanks,
Venki

[1] - http://download.intel.com/products/processor/core2duo/mobile_prod_brief.pdf

#6 lore raw
In reply to H. Peter Anvin
Quoted 4 lines · On Wed, 19 Dec 2007 11:48:14 -0800 "H. Peter Anvin" <[email protected]> wrote:

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

that sounds nice. It's fiction though ;-)

The thing to realize is that linux only sees "ACPI C3"; the BIOS maps that C3 to.. well any of the C states the processor in the system has. What you're saying is afaik correct for the *hardware* C3, not for the "C3" that Linux sees..

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
reviewer
#7 lore raw
Quoted 11 lines · Arjan van de Ven
Quoted 3 lines · On Wed, 19 Dec 2007 11:48:14 -0800 "H. Peter Anvin" <[email protected]> wrote:

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

that sounds nice. It's fiction though ;-)

The thing to realize is that linux only sees "ACPI C3"; the BIOS maps that C3 to.. well any of the C states the processor in the system has. What you're saying is afaik correct for the *hardware* C3, not for the "C3" that Linux sees..

Well, it can only map ACPI C3 to a state which is no more "dead" than what would normally be permitted by C3. IIRC, C3 is allowed to require that DMA be turned off (unlike C2), but is not allowed to lose the CPU state.

-hpa

#8 lore raw
Quoted 16 lines · On Thu, 20 Dec 2007 08:16:54 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines · On Wed, 19 Dec 2007 11:48:14 -0800 "H. Peter Anvin" <[email protected]> wrote:

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

Quoted 7 lines

that sounds nice. It's fiction though ;-)

The thing to realize is that linux only sees "ACPI C3"; the BIOS maps that C3 to.. well any of the C states the processor in the system has. What you're saying is afaik correct for the *hardware* C3, not for the "C3" that Linux sees..

Quoted 4 lines

Well, it can only map ACPI C3 to a state which is no more "dead" than what would normally be permitted by C3. IIRC, C3 is allowed to require that DMA be turned off (unlike C2), but is not allowed to lose the CPU state.

state isn't lost if the tlb or the caches are flushed... (properly, eg all pending writebacks are written back first etc)

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
reviewer
#9 lore raw
Quoted 17 lines · Arjan van de Ven
Quoted 14 lines · On Thu, 20 Dec 2007 08:16:54 -0800 "H. Peter Anvin" <[email protected]> wrote:
Quoted 13 lines · Arjan van de Ven
Quoted 3 lines · On Wed, 19 Dec 2007 11:48:14 -0800 "H. Peter Anvin" <[email protected]> wrote:

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

that sounds nice. It's fiction though ;-)

The thing to realize is that linux only sees "ACPI C3"; the BIOS maps that C3 to.. well any of the C states the processor in the system has. What you're saying is afaik correct for the *hardware* C3, not for the "C3" that Linux sees..

Quoted 6 lines
Quoted 3 lines

Well, it can only map ACPI C3 to a state which is no more "dead" than what would normally be permitted by C3. IIRC, C3 is allowed to require that DMA be turned off (unlike C2), but is not allowed to
lose the CPU state.

state isn't lost if the tlb or the caches are flushed... (properly, eg all pending writebacks are written back first etc)

Oh, right. My bad.

Of course C3 doesn't guarantee cache retention, only cache coherency.

-hpa

reviewer
#10 lore raw
In reply to H. Peter Anvin
Quoted 15 lines · H. Peter Anvin
Quoted 7 lines · Arjan van de Ven
Quoted 3 lines · On Wed, 19 Dec 2007 11:48:14 -0800 "H. Peter Anvin" <[email protected]> wrote:

I think C3 guarantees that the cache contents stay intact, and thus it might make sense in some technology to preserve the TLB as well (being a kind of cache.)

Quoted 6 lines

that sounds nice. It's fiction though ;-)

The thing to realize is that linux only sees "ACPI C3";
the BIOS maps that C3 to..
well any of the C states the processor in the system has.
What you're saying is afaik correct for the *hardware* C3, not for the "C3" that Linux sees..

Quoted 4 lines

Well, it can only map ACPI C3 to a state which is no more "dead" than what would normally be permitted by C3. IIRC, C3 is allowed to require that DMA be turned off (unlike C2), but is not allowed to lose the CPU state.

Re: mapping HW to ACPI C-states.

Right, it is fair game for the BIOS to map a "shallower" hardware C-state to a "deeper" ACPI C-state.

Re: CPU state

All C-states preserve the CPU SW programming state. (eg. while it may be saved and restored in HW,
it appears to SW to be always intact).

Re: C3 guarantees that the cache contents stay intact

This is both true and false, depending on how you use the word "intact".

If "intact" == "stays valid in cache", then no, this not guaranteed. The HW reserves the right to flush some or all of the L1 and L2 caches whenever it wants to --
this includes both HW and ACPI C2 and C3 states.

If "intact" = "cache consistent", then yes, this guarantee is true. The way the guarantee is implemented varies by generation. Older systems would lock the bus in C3 to assure
the processor was woken up for DMA to snoop.
Newer hardware simply wakes the cache to snoop
without waking the cores, or if it flushes the caches then it doesn't have to snoop at all -- which also counts as "cache consistent":-)

cheers,
-Len

author
#11 lore raw
In reply to Ingo Molnar
Quoted 18 lines · Ingo Molnar
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Ingo Molnar
Quoted 3 lines · Venki Pallipadi

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

Are there any CPUs around which *don't* flush the TLB across C3? (I guess it's not guaranteed by the spec, though, and as TLBs grow larger there might be incentive to keep them online.)

Quoted 8 lines

i dont think it's required for C3 to even turn off any portion of the CPU - if an interrupt arrives after the C3 sequence is initiated but just before dirty cachelines have been flushed then the CPU can just return without touching anything (such as the TLB) - right? So i dont think there's any implicit guarantee of TLB flushing (nor should there be), but in practice, a good C3 sequence would (statistically) turn off large portions of the CPU and hence the TLB as well.

Yes. There are cases where hardware/BIOS can do C-state changes behind OS, with things like be in C1 for a while and then go to C2/C3 after a while etc. In such cases, there will be times when TLBs are not really flushed in hardware. But ideally, if C3 results in deep idle TLBs would be turned off. And in cases where we wake up earlier than expected, C-state policy should identify that and choose a lower C-state next time around.

I also tried one variation of this, where in I only do flush if there are more than one CPU sharing the mm. But, that did not help the test case I was using (which is probably the worst case). What I would see is:
Process runs on CPU x and mm is not shared
Goes idle (C3) waiting on something
Wakes up on CPU y which will now start sharing mm
and would send flush IPI anyway

Thanks,
Venki

#12 lore raw
In reply to Ingo Molnar

FYI, your patch needed the fix below for !SMP. It works fine otherwise.

Ingo

---
 include/asm-x86/mmu.h |    6 ++++++
 1 file changed, 6 insertions(+)
include/asm-x86/mmu.h +6 -0
Index: linux/include/asm-x86/mmu.h
===================================================================
@@ -20,6 +20,12 @@ typedef struct { 
 	void *vdso;
 } mm_context_t;
 
+#ifdef CONFIG_SMP
 void leave_mm(int cpu);
+#else
+static inline void leave_mm(int cpu)
+{
+}
+#endif
 
 #endif /* _ASM_X86_MMU_H */
author
#13 lore raw
In reply to Ingo Molnar
Quoted 23 lines · Ingo Molnar
Quoted 7 lines · Venki Pallipadi

Aviod TLB flush IPIs during C3 states by voluntary leave_mm() before entering C3.

The performance impact of TLB flush on C3 should not be significant with respect to C3 wakeup latency. Also, CPUs tend to flush TLB in hardware while in C3 anyways.

Quoted 6 lines

On a 8 logical CPU system, running make -j2, the number of tlbflush IPIs goes down from 40 per second to ~ 0. Total number of interrupts during the run of this workload was ~1200 per second, which makes it ~3% savings in wakeups.

There was no measurable performance or power impact however.

thanks, applied to x86.git. Nice and elegant patch!

Btw., since the TLB flush state machine is really subtle and fragile, could you try to run the following mmap stresstest i wrote some time ago:

Quoted 6 lines

http://redhat.com/~mingo/threaded-mmap-stresstest/

for a couple of hours. It runs nr_cpus threads which then do a "random crazy mix" of mappings/unmappings/remappings of a 800 MB memory window. The more sockets/cores, the crazier the TLB races get ;-)

Ingo,

I ran this stress test on two systems (8 cores and 2 cores) for over 4 hours without any issues. There was more than 20% C3 time during the run. So, this C3 tlbflush path must have been stressed well during the run.

And sorry about the patch not working on UP config. That was a silly oversight on my part.

Thanks,
Venki

#14 lore raw
Quoted 12 lines · Venki Pallipadi
Quoted 11 lines

Btw., since the TLB flush state machine is really subtle and fragile, could you try to run the following mmap stresstest i wrote some time ago:

http://redhat.com/~mingo/threaded-mmap-stresstest/

for a couple of hours. It runs nr_cpus threads which then do a "random crazy mix" of mappings/unmappings/remappings of a 800 MB memory window. The more sockets/cores, the crazier the TLB races get ;-)

Quoted 6 lines

Ingo,

I ran this stress test on two systems (8 cores and 2 cores) for over 4 hours without any issues. There was more than 20% C3 time during the run. So, this C3 tlbflush path must have been stressed well during the run.

ok, great. Regarding power consumption: i suspect a real difference will only show up on multi-socket systems that can do deeper C modes, or on multicore systems that will benefit from longer idle time on another core. (i suspect most multicore CPUs today will only truly save significant amounts of power if all cores are idle.) In any case, not doing these extra IPIs is definitely a plus.

Ingo