Skip to contentSkip to conversation

Performance overhead of paravirt_ops on native identified

PATCH Jeremy Fitzhardinge 22 participants 79 messages May 14, 2009 – Jun 17, 2009 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi Ingo,

Xiaohui Xin and some other folks at Intel have been looking into what's behind the performance hit of paravirt_ops when running native.

It appears that the hit is entirely due to the paravirtualized spinlocks; the extra call/return in the spinlock path is somehow causing an increase in the cycles/instruction of somewhere around 2-7% (seems to vary quite a lot from test to test). The working theory is that the CPU's pipeline is getting upset about the call->call->locked-op->return->return, and seems to be failing to speculate (though I haven't seen anything definitive about the precise reasons). This doesn't entirely make sense, because the performance hit is also visible on unlock and other operations which don't involve locked instructions. But spinlock operations clearly swamp all the other pvops operations, even though I can't imagine that they're nearly as common (there's only a .05% increase in instructions executed).

If I disable just the pv-spinlock calls, my tests show that pvops is identical to non-pvops performance on native (my measurements show that it is actually about .1% faster, but Xiaohui shows a .05% slowdown).

Summary of results, averaging 10 runs of the "mmperf" test, using a no-pvops build as baseline:

nopv Pv-nospin Pv-spin
CPU cycles 100.00% 99.89% 102.18%
instructions 100.00% 100.10% 100.15%
CPI 100.00% 99.79% 102.03%
cache ref 100.00% 100.84% 100.28%
cache miss 100.00% 90.47% 88.56%
cache miss rate 100.00% 89.72% 88.31%
branches 100.00% 99.93% 100.04%
branch miss 100.00% 103.66% 107.72%
branch miss rt 100.00% 103.73% 107.67%
wallclock 100.00% 99.90% 102.20%

The clear effect here is that the 2% increase in CPI is directly reflected in the final wallclock time.

(The other interesting effect is that the more ops are out of line calls via pvops, the lower the cache access and miss rates. Not too surprising, but it suggests that the non-pvops kernel is over-inlined. On the flipside, the branch misses go up correspondingly...)

So, what's the fix?

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

One possibility is to inline _spin_lock, etc, when building an optimised kernel (ie, when there's no spinlock/preempt instrumentation/debugging enabled). That will remove the outer call/return pair, returning the instruction stream to a single call/return, which will presumably execute the same as the non-pvops case. The downsides arel 1) it will replicate the preempt_disable/enable code at eack lock/unlock callsite; this code is fairly small, but not nothing; and 2) the spinlock definitions are already a very heavily tangled mass of #ifdefs and other preprocessor magic, and making any changes will be non-trivial.

The other obvious answer is to disable pv-spinlocks. Making them a separate config option is fairly easy, and it would be trivial to enable them only when Xen is enabled (as the only non-default user). But it doesn't really address the common case of a distro build which is going to have Xen support enabled, and leaves the open question of whether the native performance cost of pv-spinlocks is worth the performance improvement on a loaded Xen system (10% saving of overall system CPU when guests block rather than spin). Still it is a reasonable short-term workaround.

The best solution would be to work out whether this really is a problem interaction with Intel's pipelines, and come up with something that avoids it. It would be very interesting to see if there's a similar hit on AMD systems.

J

Quoted 1 line

From 839033e472c8f3b228be35e57a8b31fbb7f9cf98 Mon Sep 17 00:00:00 2001

From: Jeremy Fitzhardinge <[email protected]>
Date: Wed, 13 May 2009 11:58:17 -0700
Subject: [PATCH] x86: add config to disable PV spinlocks

Paravirtualized spinlocks seem to cause a 2-7% performance hit when running a pvops kernel native. Without them, the pvops kernel is identical to a non-pvops kernel in performance.

[ Impact: reduce overhead of pvops when running native ]
arch/x86/Kconfig +0 -0
index 5f50179..a99ed71 100644
@@ -498,6 +498,17 @@ config PARAVIRT

over full virtualization. However, when run without a hypervisor the kernel is theoretically slower and slightly larger.

+config PARAVIRT_SPINLOCKS
+ bool "Enable paravirtualized spinlocks"
+ depends on PARAVIRT && SMP
+ default XEN
+ ---help---

+	  Paravirtualized spinlocks allow a pvops backend to replace the
+	  spinlock implementation with something virtualization-friendly
+	  (for example, block the virtual CPU rather than spinning).
+	  Unfortunately the downside is an as-yet unexplained performance
+	  when running native.

+
config PARAVIRT_CLOCK
bool
default n

arch/x86/include/asm/paravirt.h +0 -0
index 1fe5837..4fb37c8 100644
@@ -1443,7 +1443,7 @@ u64 _paravirt_ident_64(u64);
 

#define paravirt_nop ((void *)_paravirt_nop)

-#ifdef CONFIG_SMP
+#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)

static inline int __raw_spin_is_locked(struct raw_spinlock *lock) {

arch/x86/include/asm/spinlock.h +0 -0
index e5e6caf..b7e5db8 100644
@@ -172,7 +172,7 @@ static inline int __ticket_spin_is_contended(raw_spinlock_t *lock)
	return (((tmp >> TICKET_SHIFT) - tmp) & ((1 << TICKET_SHIFT) - 1)) > 1;
}

-#ifndef CONFIG_PARAVIRT
+#ifndef CONFIG_PARAVIRT_SPINLOCKS

static inline int __raw_spin_is_locked(raw_spinlock_t *lock)
{
@@ -206,7 +206,7 @@ static __always_inline void __raw_spin_lock_flags(raw_spinlock_t *lock,

	__raw_spin_lock(lock);
}

-#endif
+#endif /* CONFIG_PARAVIRT_SPINLOCKS */

static inline void __raw_spin_unlock_wait(raw_spinlock_t *lock) {

arch/x86/kernel/Makefile +0 -0
index 68a4ff6..4f78bd6 100644
@@ -90,7 +90,8 @@ obj-$(CONFIG_DEBUG_NX_TEST)	+= test_nx.o

obj-$(CONFIG_VMI) += vmi_32.o vmiclock_32.o
obj-$(CONFIG_KVM_GUEST) += kvm.o
obj-$(CONFIG_KVM_CLOCK) += kvmclock.o
-obj-$(CONFIG_PARAVIRT) += paravirt.o paravirt_patch_$(BITS).o paravirt-spinlocks.o
+obj-$(CONFIG_PARAVIRT) += paravirt.o paravirt_patch_$(BITS).o
+obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= paravirt-spinlocks.o
obj-$(CONFIG_PARAVIRT_CLOCK) += pvclock.o

obj-$(CONFIG_PCSPKR_PLATFORM) += pcspeaker.o

arch/x86/kernel/paravirt.c +0 -0
index aa34423..70ec9b9 100644
@@ -134,7 +134,9 @@ static void *get_call_destination(u8 type)
		.pv_irq_ops = pv_irq_ops,
		.pv_apic_ops = pv_apic_ops,
		.pv_mmu_ops = pv_mmu_ops,
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
		.pv_lock_ops = pv_lock_ops,
+#endif
	};
	return *((void **)&tmpl + type);
}
arch/x86/xen/Makefile +0 -0
index 3b767d0..172438f 100644
@@ -9,5 +9,6 @@ obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
			time.o xen-asm.o xen-asm_$(BITS).o \
			grant-table.o suspend.o

-obj-$(CONFIG_SMP) += smp.o spinlock.o
-obj-$(CONFIG_XEN_DEBUG_FS) += debugfs.o
\ No newline at end of file
+obj-$(CONFIG_SMP) += smp.o
+obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= spinlock.o
+obj-$(CONFIG_XEN_DEBUG_FS) += debugfs.o

arch/x86/xen/xen-ops.h +0 -0
index 5c50a10..22494fd 100644
@@ -61,15 +61,26 @@ void xen_setup_vcpu_info_placement(void);

#ifdef CONFIG_SMP
void xen_smp_init(void);

-void __init xen_init_spinlocks(void);
-__cpuinit void xen_init_lock_cpu(int cpu);
-void xen_uninit_lock_cpu(int cpu);
-
extern cpumask_var_t xen_cpu_initialized_map;
#else
static inline void xen_smp_init(void) {}
#endif

+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+void __init xen_init_spinlocks(void);
+__cpuinit void xen_init_lock_cpu(int cpu);
+void xen_uninit_lock_cpu(int cpu);
+#else
+static inline void xen_init_spinlocks(void)
+{
+}
+static inline void xen_init_lock_cpu(int cpu)
+{
+}
+static inline void xen_uninit_lock_cpu(int cpu)
+{
+}
+#endif

/* Declare an asm function, along with symbols needed to make it
inlineable */

#1 lore raw
Quoted 19 lines · Jeremy Fitzhardinge

So, what's the fix?

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

Quoted 11 lines

One possibility is to inline _spin_lock, etc, when building an optimised kernel (ie, when there's no spinlock/preempt instrumentation/debugging enabled). That will remove the outer call/return pair, returning the instruction stream to a single call/return, which will presumably execute the same as the non-pvops case. The downsides arel 1) it will replicate the preempt_disable/enable code at eack lock/unlock callsite; this code is fairly small, but not nothing; and 2) the spinlock definitions are already a very heavily tangled mass of #ifdefs and other preprocessor magic, and making any changes will be non-trivial.

The other obvious option, it would seem to me, would be to eliminate the *inner* call/return pair, i.e. merging the _spin_lock setup code in with the internals of each available implementation (in the case above, __ticket_spin_lock). This is effectively what happens on native. The one problem with that is that every callsite now becomes a patching target.

That brings me to a somewhat half-arsed thought I have been walking around with for a while.

Consider a paravirt -- or for that matter any other call which is runtime-static; this isn't just limited to paravirt -- function which looks to the C compiler just like any other external function -- no indirection. We can point it by default to a function which is really just an indirect jump to the appropriate handler, that handles the prepatching case. However, a linktime pass over vmlinux.o can find all the points where this function is called, and turn it into a list of patch sites(*). The advantages are:

1. [minor] no additional nop padding due to indirect function calls.
2. [major] no need for a ton of wrapper macros manifest in the code.

paravirt_ops that turn into pure inline code in the native case is obviously another ball of wax entirely; there inline assembly wrappers are simply unavoidable.

-hpa

(*) if patching code on SMP was cheaper, we could actually do this lazily, and wouldn't have to store a list of patch sites. I don't feel brave enough to go down that route.

#6 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

In reply to Jeremy Fitzhardinge
Quoted 12 lines
Quoted 1 line
Quoted 1 line

Jeremy Fitzhardinge <[email protected]> 14.05.09 02:16 >>>

One possibility is to inline _spin_lock, etc, when building an optimised kernel (ie, when there's no spinlock/preempt instrumentation/debugging enabled). That will remove the outer call/return pair, returning the instruction stream to a single call/return, which will presumably execute the same as the non-pvops case. The downsides arel 1) it will replicate the preempt_disable/enable code at eack lock/unlock callsite; this code is fairly small, but not nothing; and 2) the spinlock definitions are already a very heavily tangled mass of #ifdefs and other preprocessor magic, and making any changes will be non-trivial.

Quoted 9 lines

The other obvious answer is to disable pv-spinlocks. Making them a separate config option is fairly easy, and it would be trivial to enable them only when Xen is enabled (as the only non-default user). But it doesn't really address the common case of a distro build which is going to have Xen support enabled, and leaves the open question of whether the native performance cost of pv-spinlocks is worth the performance improvement on a loaded Xen system (10% saving of overall system CPU when guests block rather than spin). Still it is a reasonable short-term workaround.

Wouldn't a third solution be to use ticket spinlocks everywhere, i.e. eliminate the current indirection, and replace it by an indirection for just the contention case? As I view it, the problem for Xen aren't really the ticket locks by themselves, but rather the extra spinning involved, which is of concern only if a lock is contended. We're using ticket locks quite happily in our kernels, with directed instead of global wakeup from the unlock path. The only open issue we currently have is that while for native keeping interrupts disabled while spinning may be acceptable (though I'm not sure how -rt folks are viewing this), in a pv environment one should really re-enable interrupts here due to the potentially much higher latency.

Jan

maintainer
#2 lore raw
In reply to H. Peter Anvin
Quoted 57 lines · H. Peter Anvin
Quoted 19 lines · Jeremy Fitzhardinge

So, what's the fix?

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

Quoted 11 lines

One possibility is to inline _spin_lock, etc, when building an optimised kernel (ie, when there's no spinlock/preempt instrumentation/debugging enabled). That will remove the outer call/return pair, returning the instruction stream to a single call/return, which will presumably execute the same as the non-pvops case. The downsides arel 1) it will replicate the preempt_disable/enable code at eack lock/unlock callsite; this code is fairly small, but not nothing; and 2) the spinlock definitions are already a very heavily tangled mass of #ifdefs and other preprocessor magic, and making any changes will be non-trivial.

The other obvious option, it would seem to me, would be to eliminate the *inner* call/return pair, i.e. merging the _spin_lock setup code in with the internals of each available implementation (in the case above, __ticket_spin_lock). This is effectively what happens on native. The one problem with that is that every callsite now becomes a patching target.

That brings me to a somewhat half-arsed thought I have been walking around with for a while.

Consider a paravirt -- or for that matter any other call which is runtime-static; this isn't just limited to paravirt -- function which looks to the C compiler just like any other external function -- no indirection. We can point it by default to a function which is really just an indirect jump to the appropriate handler, that handles the prepatching case. However, a linktime pass over vmlinux.o can find all the points where this function is called, and turn it into a list of patch sites(*). The advantages are:

1. [minor] no additional nop padding due to indirect function calls.
2. [major] no need for a ton of wrapper macros manifest in the code.

paravirt_ops that turn into pure inline code in the native case is obviously another ball of wax entirely; there inline assembly wrappers are simply unavoidable.

Quoted 5 lines

-hpa

(*) if patching code on SMP was cheaper, we could actually do this lazily, and wouldn't have to store a list of patch sites. I don't feel brave enough to go down that route.

This sounds remarkably like what the dynamic function call tracer does.

maintainer
#7 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

In reply to Jan Beulich
Quoted 10 lines · Jan Beulich

Wouldn't a third solution be to use ticket spinlocks everywhere, i.e. eliminate the current indirection, and replace it by an indirection for just the contention case? As I view it, the problem for Xen aren't really the ticket locks by themselves, but rather the extra spinning involved, which is of concern only if a lock is contended. We're using ticket locks quite happily in our kernels, with directed instead of global wakeup from the unlock path. The only open issue we currently have is that while for native keeping interrupts disabled while spinning may be acceptable (though I'm not sure how -rt folks are viewing this), in a pv environment one should really re-enable interrupts here due to the potentially much higher latency.

the -rt folks don't nearly have as many spinlocks, and for those we do like ticket locks, because they are much fairer and give better worst case contention behaviour.

Also, for the -rt folks, preempt disable is about as bad as irq disable.

#3 lore raw
In reply to Peter Zijlstra
Quoted 3 lines · Peter Zijlstra

This sounds remarkably like what the dynamic function call tracer does.

I'm sure this has been invented before... probably more than once. Far too many things we invent don't get generalized.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#4 lore raw
In reply to H. Peter Anvin
Quoted 6 lines · H. Peter Anvin

The other obvious option, it would seem to me, would be to eliminate the *inner* call/return pair, i.e. merging the _spin_lock setup code in with the internals of each available implementation (in the case above, __ticket_spin_lock). This is effectively what happens on native. The one problem with that is that every callsite now becomes a patching target.

Yes, that's an option. It has the downside of requiring changes to the common spinlock code in kernel/spinlock.c and linux/spinlock_api*.h. The amount of duplicated code is potentially quite large, but there aren't that many spinlock implementations.

Also, there's not much point in using pv spinlocks when all the instrumentation is on. Lock contention metering, for example, never does a proper lock operation, but does a spin with repeated trylocks; we can't optimise that, so there's no point in trying.

So maybe if we can fast-path the fast-path to pv spinlocks, the problem is more tractable...

Quoted 12 lines

That brings me to a somewhat half-arsed thought I have been walking around with for a while.

Consider a paravirt -- or for that matter any other call which is runtime-static; this isn't just limited to paravirt -- function which looks to the C compiler just like any other external function -- no indirection. We can point it by default to a function which is really just an indirect jump to the appropriate handler, that handles the prepatching case. However, a linktime pass over vmlinux.o can find all the points where this function is called, and turn it into a list of patch sites(*). The advantages are:

Quoted 7 lines

1. [minor] no additional nop padding due to indirect function calls.
2. [major] no need for a ton of wrapper macros manifest in the code.

paravirt_ops that turn into pure inline code in the native case is obviously another ball of wax entirely; there inline assembly wrappers are simply unavoidable.

We did consider something like this at the outset. As I remember, there were a few concerns:

* There was no relocation data available in the kernel. I played around with ways to make it work, but they ended up being fairly complex and brittle, with a tendency (of course) to trigger binutils bugs. Maybe that has changed.
* We didn't really want to implement two separate mechanisms for the same thing. Given that we wanted to inline things like cli/sti/pushf/popf, we needed to have something capable of full patching. Having a separate mechanisms for patching calls is harder to justify. Now that pvops is well settled, perhaps it makes sense to consider adding another more general patching mechanism to avoid the indirect calls (a dynamic linker, essentially).

I won't make any great claims about the beauty of the PV_CALL* gunk, but at the very least it is contained within paravirt.h.

Quoted 4 lines

(*) if patching code on SMP was cheaper, we could actually do this lazily, and wouldn't have to store a list of patch sites. I don't feel brave enough to go down that route.

The problem that the tracepoints people were trying to solve was harder, where they wanted to replace an arbitrary set of instructions with some other arbitrary instructions (or a call) - that would need some kind SMP synchronization, both for general sanity and to keep the Intel rules happy.

In theory relinking a call should just be a single word write into the instruction, but I don't know if that gets into undefined territory or not. On older P4 systems it would end up blowing away the trace cache on all cpus when you write to code like that, so you'd want to be sure that your references are getting resolved fairly quickly. But its hard to see how patching the offset in a call instruction would end up calling something other than the old or new function.

J

#8 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

In reply to Jan Beulich
Quoted 6 lines · Jan Beulich

Wouldn't a third solution be to use ticket spinlocks everywhere, i.e. eliminate the current indirection, and replace it by an indirection for just the contention case? As I view it, the problem for Xen aren't really the ticket locks by themselves, but rather the extra spinning involved, which is of concern only if a lock is contended. We're using ticket locks quite happily in our kernels, with directed instead of global wakeup from the unlock path.

Do you have a patch to illustrate what you mean? How do you keep track of the target vcpu for the directed wakeup? Are you using the event-poll mechanism to block?

J

#5 lore raw
In reply to Jeremy Fitzhardinge
Quoted 8 lines · Jeremy Fitzhardinge

We did consider something like this at the outset. As I remember, there were a few concerns:

* There was no relocation data available in the kernel. I played around with ways to make it work, but they ended up being fairly complex and brittle, with a tendency (of course) to trigger binutils bugs. Maybe that has changed.

We already do this pass (in fact, we do something like three passes of it.) It's basically the vmlinux.o pass.

Quoted 7 lines

* We didn't really want to implement two separate mechanisms for the same thing. Given that we wanted to inline things like cli/sti/pushf/popf, we needed to have something capable of full patching. Having a separate mechanisms for patching calls is harder to justify. Now that pvops is well settled, perhaps it makes sense to consider adding another more general patching mechanism to avoid the indirect calls (a dynamic linker, essentially).

Full patching is understandable (although I think sometimes the code generated was worse than out-of-line... I believe you have fixed that.)

Quoted 2 lines

I won't make any great claims about the beauty of the PV_CALL* gunk, but at the very least it is contained within paravirt.h.

There is still massive spillover into other code, though, at least some of which could possibly be avoided. I don't know.

Quoted 9 lines
Quoted 4 lines

(*) if patching code on SMP was cheaper, we could actually do this lazily, and wouldn't have to store a list of patch sites. I don't feel brave enough to go down that route.

The problem that the tracepoints people were trying to solve was harder, where they wanted to replace an arbitrary set of instructions with some other arbitrary instructions (or a call) - that would need some kind SMP synchronization, both for general sanity and to keep the Intel rules happy.

Quoted 7 lines

In theory relinking a call should just be a single word write into the instruction, but I don't know if that gets into undefined territory or not. On older P4 systems it would end up blowing away the trace cache on all cpus when you write to code like that, so you'd want to be sure that your references are getting resolved fairly quickly. But its hard to see how patching the offset in a call instruction would end up calling something other than the old or new function.

The problem is that since the call offset field can be arbitrarily aligned -- it could even cross page boundaries -- you still have absolutely no SMP atomicity guarantees. So you still have all the same problems. Without

-hpa

#9 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

In reply to Jeremy Fitzhardinge
Quoted 11 lines
Quoted 1 line
Quoted 1 line

Jeremy Fitzhardinge <[email protected]> 14.05.09 19:45 >>>

Quoted 6 lines · Jan Beulich

Wouldn't a third solution be to use ticket spinlocks everywhere, i.e. eliminate the current indirection, and replace it by an indirection for just the contention case? As I view it, the problem for Xen aren't really the ticket locks by themselves, but rather the extra spinning involved, which is of concern only if a lock is contended. We're using ticket locks quite happily in our kernels, with directed instead of global wakeup from the unlock path.

Do you have a patch to illustrate what you mean? How do you keep track of the target vcpu for the directed wakeup? Are you using the event-poll mechanism to block?

A patch for the pv-ops kernel would require some time. What I can give you right away - just for reference - are the sources we currently use in our kernel:
attached.

Jan

#13 lore raw

[tip:x86/urgent] x86: Fix performance regression caused by paravirt_ops on native kernels

In reply to Jeremy Fitzhardinge
Commit-ID:  b4ecc126991b30fe5f9a59dfacda046aeac124b2
Gitweb:     http://git.kernel.org/tip/b4ecc126991b30fe5f9a59dfacda046aeac124b2
Author:     Jeremy Fitzhardinge <[email protected]>
AuthorDate: Wed, 13 May 2009 17:16:55 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 15 May 2009 20:07:42 +0200

x86: Fix performance regression caused by paravirt_ops on native kernels

Xiaohui Xin and some other folks at Intel have been looking into what's behind the performance hit of paravirt_ops when running native.

It appears that the hit is entirely due to the paravirtualized spinlocks introduced by:

 | commit 8efcbab674de2bee45a2e4cdf97de16b8e609ac8
 | Date:   Mon Jul 7 12:07:51 2008 -0700
 |
 |     paravirt: introduce a "lock-byte" spinlock implementation

The extra call/return in the spinlock path is somehow causing an increase in the cycles/instruction of somewhere around 2-7% (seems to vary quite a lot from test to test). The working theory is that the CPU's pipeline is getting upset about the call->call->locked-op->return->return, and seems to be failing to speculate (though I haven't seen anything definitive about the precise reasons). This doesn't entirely make sense, because the performance hit is also visible on unlock and other operations which don't involve locked instructions. But spinlock operations clearly swamp all the other pvops operations, even though I can't imagine that they're nearly as common (there's only a .05% increase in instructions executed).

If I disable just the pv-spinlock calls, my tests show that pvops is identical to non-pvops performance on native (my measurements show that it is actually about .1% faster, but Xiaohui shows a .05% slowdown).

Summary of results, averaging 10 runs of the "mmperf" test, using a no-pvops build as baseline:

nopv Pv-nospin Pv-spin
CPU cycles 100.00% 99.89% 102.18%
instructions 100.00% 100.10% 100.15%
CPI 100.00% 99.79% 102.03%
cache ref 100.00% 100.84% 100.28%
cache miss 100.00% 90.47% 88.56%
cache miss rate 100.00% 89.72% 88.31%
branches 100.00% 99.93% 100.04%
branch miss 100.00% 103.66% 107.72%
branch miss rt 100.00% 103.73% 107.67%
wallclock 100.00% 99.90% 102.20%

The clear effect here is that the 2% increase in CPI is directly reflected in the final wallclock time.

(The other interesting effect is that the more ops are out of line calls via pvops, the lower the cache access and miss rates. Not too surprising, but it suggests that the non-pvops kernel is over-inlined. On the flipside, the branch misses go up correspondingly...)

So, what's the fix?

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

One possibility is to inline _spin_lock, etc, when building an optimised kernel (ie, when there's no spinlock/preempt instrumentation/debugging enabled). That will remove the outer call/return pair, returning the instruction stream to a single call/return, which will presumably execute the same as the non-pvops case. The downsides arel 1) it will replicate the preempt_disable/enable code at eack lock/unlock callsite; this code is fairly small, but not nothing; and 2) the spinlock definitions are already a very heavily tangled mass of #ifdefs and other preprocessor magic, and making any changes will be non-trivial.

The other obvious answer is to disable pv-spinlocks. Making them a separate config option is fairly easy, and it would be trivial to enable them only when Xen is enabled (as the only non-default user). But it doesn't really address the common case of a distro build which is going to have Xen support enabled, and leaves the open question of whether the native performance cost of pv-spinlocks is worth the performance improvement on a loaded Xen system (10% saving of overall system CPU when guests block rather than spin). Still it is a reasonable short-term workaround.

[ Impact: fix pvops performance regression when running native ]

LKML-Reference: <[email protected]>

[ fixed the help text ]
---
 arch/x86/Kconfig                |   13 +++++++++++++
 arch/x86/include/asm/paravirt.h |    2 +-
 arch/x86/include/asm/spinlock.h |    4 ++--
 arch/x86/kernel/Makefile        |    3 ++-
 arch/x86/kernel/paravirt.c      |    2 ++
 arch/x86/xen/Makefile           |    5 +++--
 arch/x86/xen/xen-ops.h          |   19 +++++++++++++++----
 7 files changed, 38 insertions(+), 10 deletions(-)
arch/x86/Kconfig +13 -0
index df9e885..a6efe0a 100644
@@ -498,6 +498,19 @@ config PARAVIRT
 	  over full virtualization.  However, when run without a hypervisor
 	  the kernel is theoretically slower and slightly larger.
 
+config PARAVIRT_SPINLOCKS
+	bool "Paravirtualization layer for spinlocks"
+	depends on PARAVIRT && SMP && EXPERIMENTAL
+	---help---
+	  Paravirtualized spinlocks allow a pvops backend to replace the
+	  spinlock implementation with something virtualization-friendly
+	  (for example, block the virtual CPU rather than spinning).
+
+	  Unfortunately the downside is an up to 5% performance hit on
+	  native kernels, with various workloads.
+
+	  If you are unsure how to answer this question, answer N.
+
 config PARAVIRT_CLOCK
 	bool
 	default n
arch/x86/include/asm/paravirt.h +1 -1
index 378e369..a53da00 100644
@@ -1443,7 +1443,7 @@ u64 _paravirt_ident_64(u64);
 
 #define paravirt_nop	((void *)_paravirt_nop)
 
-#ifdef CONFIG_SMP
+#if defined(CONFIG_SMP) && defined(CONFIG_PARAVIRT_SPINLOCKS)
 
 static inline int __raw_spin_is_locked(struct raw_spinlock *lock)
 {
arch/x86/include/asm/spinlock.h +2 -2
index e5e6caf..b7e5db8 100644
@@ -172,7 +172,7 @@ static inline int __ticket_spin_is_contended(raw_spinlock_t *lock)
 	return (((tmp >> TICKET_SHIFT) - tmp) & ((1 << TICKET_SHIFT) - 1)) > 1;
 }
 
-#ifndef CONFIG_PARAVIRT
+#ifndef CONFIG_PARAVIRT_SPINLOCKS
 
 static inline int __raw_spin_is_locked(raw_spinlock_t *lock)
 {
@@ -206,7 +206,7 @@ static __always_inline void __raw_spin_lock_flags(raw_spinlock_t *lock,
 	__raw_spin_lock(lock);
 }
 
-#endif
+#endif	/* CONFIG_PARAVIRT_SPINLOCKS */
 
 static inline void __raw_spin_unlock_wait(raw_spinlock_t *lock)
 {
arch/x86/kernel/Makefile +2 -1
index 145cce7..88d1bfc 100644
@@ -89,7 +89,8 @@ obj-$(CONFIG_DEBUG_NX_TEST)	+= test_nx.o
 obj-$(CONFIG_VMI)		+= vmi_32.o vmiclock_32.o
 obj-$(CONFIG_KVM_GUEST)		+= kvm.o
 obj-$(CONFIG_KVM_CLOCK)		+= kvmclock.o
-obj-$(CONFIG_PARAVIRT)		+= paravirt.o paravirt_patch_$(BITS).o paravirt-spinlocks.o
+obj-$(CONFIG_PARAVIRT)		+= paravirt.o paravirt_patch_$(BITS).o
+obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= paravirt-spinlocks.o
 obj-$(CONFIG_PARAVIRT_CLOCK)	+= pvclock.o
 
 obj-$(CONFIG_PCSPKR_PLATFORM)	+= pcspeaker.o
arch/x86/kernel/paravirt.c +2 -0
index 8e45f44..9faf43b 100644
@@ -134,7 +134,9 @@ static void *get_call_destination(u8 type)
 		.pv_irq_ops = pv_irq_ops,
 		.pv_apic_ops = pv_apic_ops,
 		.pv_mmu_ops = pv_mmu_ops,
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
 		.pv_lock_ops = pv_lock_ops,
+#endif
 	};
 	return *((void **)&tmpl + type);
 }
arch/x86/xen/Makefile +3 -2
index 3b767d0..172438f 100644
@@ -9,5 +9,6 @@ obj-y		:= enlighten.o setup.o multicalls.o mmu.o irq.o \
 			time.o xen-asm.o xen-asm_$(BITS).o \
 			grant-table.o suspend.o
 
-obj-$(CONFIG_SMP)		+= smp.o spinlock.o
-obj-$(CONFIG_XEN_DEBUG_FS)	+= debugfs.o
\ No newline at end of file
+obj-$(CONFIG_SMP)		+= smp.o
+obj-$(CONFIG_PARAVIRT_SPINLOCKS)+= spinlock.o
+obj-$(CONFIG_XEN_DEBUG_FS)	+= debugfs.o
arch/x86/xen/xen-ops.h +15 -4
index 2013946..ca6596b 100644
@@ -62,15 +62,26 @@ void xen_setup_vcpu_info_placement(void);
 #ifdef CONFIG_SMP
 void xen_smp_init(void);
 
-void __init xen_init_spinlocks(void);
-__cpuinit void xen_init_lock_cpu(int cpu);
-void xen_uninit_lock_cpu(int cpu);
-
 extern cpumask_var_t xen_cpu_initialized_map;
 #else
 static inline void xen_smp_init(void) {}
 #endif
 
+#ifdef CONFIG_PARAVIRT_SPINLOCKS
+void __init xen_init_spinlocks(void);
+__cpuinit void xen_init_lock_cpu(int cpu);
+void xen_uninit_lock_cpu(int cpu);
+#else
+static inline void xen_init_spinlocks(void)
+{
+}
+static inline void xen_init_lock_cpu(int cpu)
+{
+}
+static inline void xen_uninit_lock_cpu(int cpu)
+{
+}
+#endif
 
 /* Declare an asm function, along with symbols needed to make it
    inlineable */
#10 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

In reply to Jan Beulich
Quoted 3 lines · Jan Beulich

A patch for the pv-ops kernel would require some time. What I can give you right away - just for reference - are the sources we currently use in our kernel:
attached.

Hm, I see. Putting a call out to a pv-ops function in the ticket lock slow path looks pretty straightforward. The need for an extra lock on the contended unlock side is a bit unfortunate; have you measured to see what hit that has? Seems to me like you could avoid the problem by using per-cpu storage rather than stack storage (though you'd need to copy the per-cpu data to stack when handling a nested spinlock).

What's the thinking behind the xen_spin_adjust() stuff?

Quoted 1 line

static __always_inline void __ticket_spin_lock(raw_spinlock_t *lock) { unsigned int token, count; bool free; __ticket_spin_lock_preamble; if (unlikely(!free)) token = xen_spin_adjust(lock, token); do { count = 1 << 10; __ticket_spin_lock_body; } while (unlikely(!count) && !xen_spin_wait(lock, token)); }

How does this work? Doesn't it always go into the slowpath loop even if the preamble got the lock with no contention?

J

#11 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Jeremy Fitzhardinge <[email protected]> 15.05.09 20:50 >>>

Quoted 3 lines · Jan Beulich

A patch for the pv-ops kernel would require some time. What I can give you right away - just for reference - are the sources we currently use in our kernel:
attached.

Quoted 6 lines

Hm, I see. Putting a call out to a pv-ops function in the ticket lock slow path looks pretty straightforward. The need for an extra lock on the contended unlock side is a bit unfortunate; have you measured to see what hit that has? Seems to me like you could avoid the problem by using per-cpu storage rather than stack storage (though you'd need to copy the per-cpu data to stack when handling a nested spinlock).

Not sure how you'd imagine this to work: The unlock code has to look at all cpus' data in either case, so an inner lock would still be required imo.

Quoted 1 line

What's the thinking behind the xen_spin_adjust() stuff?

That's the placeholder for implementing interrupt re-enabling in the irq-save lock path. The requirement is that if a nested lock attempt hits the same lock on the same cpu that it failed to get acquired on earlier (but got a ticket already), tickets for the given (lock, cpu) pair need to be circularly shifted around so that the innermost requestor gets the earliest ticket. This is what that function's job will become if I ever get to implement this.

Quoted 8 lines
Quoted 5 lines

static __always_inline void __ticket_spin_lock(raw_spinlock_t *lock) { unsigned int token, count; bool free; __ticket_spin_lock_preamble; if (unlikely(!free)) token = xen_spin_adjust(lock, token); do { count = 1 << 10; __ticket_spin_lock_body; } while (unlikely(!count) && !xen_spin_wait(lock, token)); }

How does this work? Doesn't it always go into the slowpath loop even if the preamble got the lock with no contention?

It indeed always enters the slowpath loop, but only for a single pass through part of its body (the first compare in the body macro will make it exit the loop right away: 'token' is not only the ticket here, but the full lock->slock contents). But yes, I think you're right, one could avoid entering the body altogether by moving the containing loop into the if(!free) body. The logic went through a number of re-writes, so I must have overlooked that opportunity on the last round of adjustments.

Jan

#12 lore raw

Re: [Xen-devel] Performance overhead of paravirt_ops on nativeidentified

Quoted 8 lines · Jan Beulich
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines

Jeremy Fitzhardinge <[email protected]> 15.05.09 20:50 >>>

Quoted 4 lines · Jan Beulich

A patch for the pv-ops kernel would require some time. What I can give you right away - just for reference - are the sources we currently use in our kernel:
attached.

Quoted 6 lines
Quoted 2 lines

Hm, I see. Putting a call out to a pv-ops function in the ticket lock slow path looks pretty straightforward. The need for an extra lock on the contended unlock side is a bit unfortunate; have you measured to see what hit that has? Seems to me like you could avoid the problem by using per-cpu storage rather than stack storage (though you'd need to copy the per-cpu data to stack when handling a nested spinlock).

Not sure how you'd imagine this to work: The unlock code has to look at all cpus' data in either case, so an inner lock would still be required imo.

Well, rather than an explicit lock I was thinking of an optimistic scheme like the pv clock update algorithm.

"struct spinning" would have a version counter it could update using the even=stable/odd=unstable algorithm. The lock side would simply save the current per-cpu "struct spinning" state onto its own stack (assuming you can even have nested spinning), and then update the per-cpu copy with the current lock. The kick side can check the version counter to make sure it gets a consistent snapshot of the target cpu's current lock state.

I think that only the locking side requires locked instructions, and the kick side can be all unlocked.

Quoted 10 lines
Quoted 2 lines

What's the thinking behind the xen_spin_adjust() stuff?

That's the placeholder for implementing interrupt re-enabling in the irq-save lock path. The requirement is that if a nested lock attempt hits the same lock on the same cpu that it failed to get acquired on earlier (but got a ticket already), tickets for the given (lock, cpu) pair need to be circularly shifted around so that the innermost requestor gets the earliest ticket. This is what that function's job will become if I ever get to implement this.

Sounds fiddly.

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

static __always_inline void __ticket_spin_lock(raw_spinlock_t *lock) { unsigned int token, count; bool free; __ticket_spin_lock_preamble; if (unlikely(!free)) token = xen_spin_adjust(lock, token); do { count = 1 << 10; __ticket_spin_lock_body; } while (unlikely(!count) && !xen_spin_wait(lock, token)); }

How does this work? Doesn't it always go into the slowpath loop even if the preamble got the lock with no contention?

Quoted 8 lines

It indeed always enters the slowpath loop, but only for a single pass through part of its body (the first compare in the body macro will make it exit the loop right away: 'token' is not only the ticket here, but the full lock->slock contents). But yes, I think you're right, one could avoid entering the body altogether by moving the containing loop into the if(!free) body. The logic went through a number of re-writes, so I must have overlooked that opportunity on the last round of adjustments.

I was thinking of something like this: (completely untested)

void __ticket_spin_lock(raw_spinlock_t *lock)

{
	unsigned short inc = 0x100;
	unsigned short token;

	do {
		unsigned count = 1 << 10;
		asm volatile (
			"	lock xaddw %w0, %1\n"
			"1:	cmpb %h0, %b0\n"
			"	je 2f\n"
			"	rep ; nop\n"
			"	movb %1, %b0\n"
			/* don't need lfence here, because loads are in-order */
			"	sub $1,%2\n"
			"	jnz 1b\n"
			"2:"
			: "+Q" (inc), "+m" (lock->slock), "+r" (count)
			:
			: "memory", "cc");

		if (likely(count != 0))
			break;

		token = inc;
		inc = 0;
	} while (unlikely(!xen_spin_wait(lock, token)));
}

where xen_spin_wait() would actually be a pvops call, and perhaps the asm could be pulled out into an inline to deal with the 8/16 bit ticket difference.

J

#14 lore raw
In reply to Jeremy Fitzhardinge
Quoted 7 lines · On Wed, 13 May 2009 17:16:55 -0700 Jeremy Fitzhardinge <[email protected]> wrote:

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30
Quoted 9 lines

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

Can't those calls be changed to jumps?

#15 lore raw
Quoted 22 lines · Chuck Ebbert
Quoted 7 lines · On Wed, 13 May 2009 17:16:55 -0700 Jeremy Fitzhardinge <[email protected]> wrote:

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30
Quoted 10 lines

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

Quoted 2 lines

Can't those calls be changed to jumps?

In this specific instance of this example, yes. But if you start enabling various spinlock debug options then there'll be code following the call. It would be hard for the runtime patching machinery to know when it would be safe to do the substitution.

J

#16 lore raw
Quoted 6 lines · Jeremy Fitzhardinge

In this specific instance of this example, yes. But if you start enabling various spinlock debug options then there'll be code following the call. It would be hard for the runtime patching machinery to know when it would be safe to do the substitution.

"When it is immediately followed by a ret" seems like a straightforward rule to me?

-hpa

#17 lore raw
In reply to Jeremy Fitzhardinge

Remember we have done one experiment with "jump", the result shows seems the overhead is even more than the call.

Thanks
Xiaohui

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:[email protected]]
Sent: 2009年5月22日 6:49
To: Chuck Ebbert
Cc: Ingo Molnar; Xin, Xiaohui; Li, Xin; Nakajima, Jun; H. Peter Anvin; Nick Piggin; Linux Kernel Mailing List; Xen-devel
Subject: Re: Performance overhead of paravirt_ops on native identified

Quoted 22 lines · Chuck Ebbert
Quoted 7 lines · On Wed, 13 May 2009 17:16:55 -0700 Jeremy Fitzhardinge <[email protected]> wrote:

Paravirt patching turns all the pvops calls into direct calls, so _spin_lock etc do end up having direct calls. For example, the compiler generated code for paravirtualized _spin_lock is:

<_spin_lock+0>:		mov    %gs:0xb4c8,%rax
<_spin_lock+9>:		incl   0xffffffffffffe044(%rax)
<_spin_lock+15>:	callq  *0xffffffff805a5b30
Quoted 10 lines

<_spin_lock+22>: retq

The indirect call will get patched to:
<_spin_lock+0>: mov %gs:0xb4c8,%rax
<_spin_lock+9>: incl 0xffffffffffffe044(%rax)
<_spin_lock+15>: callq <__ticket_spin_lock>
<_spin_lock+20>: nop; nop /* or whatever 2-byte nop */
<_spin_lock+22>: retq

Quoted 2 lines

Can't those calls be changed to jumps?

In this specific instance of this example, yes. But if you start enabling various spinlock debug options then there'll be code following the call. It would be hard for the runtime patching machinery to know when it would be safe to do the substitution.

J

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#18 lore raw
Quoted 1 line · Xiaohui

Remember we have done one experiment with "jump", the result shows seems the overhead is even more than the call.

I didn't, no. That seems extremely weird to me.

(Unbalancing the call/ret stack is known to suck royally, of course.)

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

Can't those calls be changed to jumps?

In this specific instance of this example, yes. But if you start enabling various spinlock debug options then there'll be code following the call. It would be hard for the runtime patching machinery to know when it would be safe to do the substitution.

When there is code after the call, it's rather obviously not safe.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#19 lore raw
In reply to Xin, Xiaohui
Quoted 2 lines · Xiaohui

Remember we have done one experiment with "jump", the result shows seems the overhead is even more than the call.

I don't think you had mentioned that. You're saying that a call->jmp->ret is slower than call->call->ret->ret?

J

#20 lore raw

What I mean is that if the binary of _spin_lock is like this:
(gdb) disassemble _spin_lock
Dump of assembler code for function _spin_lock:

0xffffffff80497c0f <_spin_lock+0>:      mov    1252634(%rip),%r11        # #0xffffffff805c9930 <test_lock_ops+16>
0xffffffff80497c16 <_spin_lock+7>:      jmpq   *%r11

End of assembler dump.
(gdb) disassemble

In this situation the binary contains a jump, the overhead is more than the call.

Thanks
Xiaohui

-----Original Message-----
From: Jeremy Fitzhardinge [mailto:[email protected]]
Sent: 2009年5月22日 12:28
To: Xin, Xiaohui
Cc: Chuck Ebbert; Ingo Molnar; Li, Xin; Nakajima, Jun; H. Peter Anvin; Nick Piggin; Linux Kernel Mailing List; Xen-devel
Subject: Re: Performance overhead of paravirt_ops on native identified

Quoted 2 lines · Xiaohui

Remember we have done one experiment with "jump", the result shows seems the overhead is even more than the call.

I don't think you had mentioned that. You're saying that a call->jmp->ret is slower than call->call->ret->ret?

J

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel

#21 lore raw
Quoted 10 lines · Xiaohui

What I mean is that if the binary of _spin_lock is like this:
(gdb) disassemble _spin_lock
Dump of assembler code for function _spin_lock:

0xffffffff80497c0f <_spin_lock+0>:      mov    1252634(%rip),%r11        # #0xffffffff805c9930 <test_lock_ops+16>
0xffffffff80497c16 <_spin_lock+7>:      jmpq   *%r11

End of assembler dump.
(gdb) disassemble

In this situation the binary contains a jump, the overhead is more than the call.

That's an indirect jump, though. I don't think anyone was suggesting using an indirect jump; the final patched version should be a direct jump (instead of a direct call.)

I can see how indirect jumps might be slower, since they are probably not optimized as aggressively in hardware as indirect calls -- indirect jumps are generally used for switch tables, which often have low predictability, whereas indirect calls are generally used for method calls, which are (a) incredibly important for OOP languages, and (b) generally highly predictable on the dynamic scale.

However, direct jumps and calls don't need prediction at all (although of course rets do.)

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#22 lore raw
Quoted 11 lines · H. Peter Anvin

That's an indirect jump, though. I don't think anyone was suggesting using an indirect jump; the final patched version should be a direct jump (instead of a direct call.)

I can see how indirect jumps might be slower, since they are probably not optimized as aggressively in hardware as indirect calls -- indirect jumps are generally used for switch tables, which often have low predictability, whereas indirect calls are generally used for method calls, which are (a) incredibly important for OOP languages, and (b) generally highly predictable on the dynamic scale.

Quoted 2 lines

However, direct jumps and calls don't need prediction at all (although of course rets do.)

I did a quick experiment to see how many sites this optimisation could actually affect. Firstly, it does absolutely nothing with frame pointers enabled. Arranging for no frame pointers is quite tricky, since it means disabling all debugging, tracing and other things.

With no frame pointers, its about 26 of 5400 indirect calls are immediately followed by ret (not all of those sites are pvops calls). With preempt disabled, this goes up to 45 sites.

I haven't done any actual runtime tests, but a quick survey of the affected sites shows that only a couple are performance-sensitive; _spin_lock and _spin_lock_irq and _spin_lock_irqsave are the most obvious.

J

#23 lore raw
Quoted 6 lines · Jeremy Fitzhardinge

I did a quick experiment to see how many sites this optimisation could actually affect. Firstly, it does absolutely nothing with frame pointers enabled. Arranging for no frame pointers is quite tricky, since it means disabling all debugging, tracing and other things.

Quoted 8 lines

With no frame pointers, its about 26 of 5400 indirect calls are immediately followed by ret (not all of those sites are pvops calls). With preempt disabled, this goes up to 45 sites.

I haven't done any actual runtime tests, but a quick survey of the affected sites shows that only a couple are performance-sensitive; _spin_lock and _spin_lock_irq and _spin_lock_irqsave are the most obvious.

OK, that doesn't seem like a very productive avenue. Problem still remains, obviously.

-hpa

#24 lore raw

[benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Jeremy Fitzhardinge

I did more 'perf stat mmap-perf 1' measurements (bound to a single core, running single thread - to exclude cross-CPU noise), which in essence measures CONFIG_PARAVIRT=y overhead on native kernels:

Performance counter stats for './mmap-perf':

      [vanilla]  [PARAVIRT=y]

    1230.805297   1242.828348  task clock ticks     (msecs)   + 0.97%
     3602663413    3637329004  CPU cycles           (events)  + 0.96%
     1927074043    1958330813  instructions         (events)  + 1.62%

That's around 1% on really fast hardware (Core2 E6800 @ 2.93 GHz, 4MB L2 cache), i.e. still significant overhead. Distros generally enable CONFIG_PARAVIRT, even though a large majority of users never actually runs them as Xen guests.

Are there plans to analyze and fix this overhead too, beyond the paravirt-spinlocks overhead you analyzed? (Note that i had CONFIG_PARAVIRT_SPINLOCKS disabled in this test.)

I think only those users should get overhead who actually run such kernels in a virtualized environment.

I cannot cite a single other kernel feature that has so much performance impact when runtime-disabled. For example, an often cited bloat and overhead source is CONFIG_SECURITY=y.

Its runtime overhead (same system, same workload) is:

      [vanilla]  [SECURITY=y]

    1219.652255   1230.805297  task clock ticks     (msecs)   + 0.91%
     3574548461    3602663413  CPU cycles           (events)  + 0.78%
     1915177924    1927074043  instructions         (events)  + 0.62%

( With the difference that the distros that enable CONFIG_SECURITY=y
tend to install and use at least one security module by default. )

So everyone who runs a CONFIG_PARAVIRT=y distro kernel has 1% of overhead in this mmap-test workload - even if no Xen is used on that box, ever.

Config attached.

Ingo

#
# Automatically generated make config: don't edit # Linux kernel version: 2.6.30-rc7
# Mon May 25 10:52:09 2009
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64" CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig" CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y CONFIG_GENERIC_HWEIGHT=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_ARCH_HAS_CPU_RELAX=y CONFIG_ARCH_HAS_DEFAULT_IDLE=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_AUDIT_ARCH=y CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_PENDING_IRQ=y CONFIG_USE_GENERIC_SMP_HELPERS=y CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
CONFIG_CLASSIC_RCU=y
# CONFIG_TREE_RCU is not set
# CONFIG_PREEMPT_RCU is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
# CONFIG_CGROUPS is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_EXTRA_PASS=y
# CONFIG_STRIP_ASM_SYMS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_COUNTERS=y

#
# Performance Counters
#
CONFIG_PERF_COUNTERS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
# CONFIG_MARKERS is not set
CONFIG_OPROFILE=y
# CONFIG_OPROFILE_IBS is not set
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_API_DEBUG=y
# CONFIG_SLOW_WORK is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set CONFIG_X86_MPPARSE=y
# CONFIG_X86_EXTENDED_PLATFORM is not set CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_PARAVIRT_GUEST=y
CONFIG_XEN=y
CONFIG_XEN_MAX_DOMAIN_MEMORY=32 CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_KVM_CLOCK is not set
# CONFIG_KVM_GUEST is not set CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_SPINLOCKS is not set CONFIG_PARAVIRT_CLOCK=y
# CONFIG_PARAVIRT_DEBUG is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
CONFIG_MCORE2=y
# CONFIG_GENERIC_CPU is not set CONFIG_X86_CPU=y CONFIG_X86_L1_CACHE_BYTES=64 CONFIG_X86_INTERNODE_CACHE_BYTES=64 CONFIG_X86_CMPXCHG=y CONFIG_X86_L1_CACHE_SHIFT=6 CONFIG_X86_WP_WORKS_OK=y CONFIG_X86_INTEL_USERCOPY=y CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y CONFIG_X86_MINIMUM_CPU_FAMILY=64 CONFIG_X86_DEBUGCTLMSR=y CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y CONFIG_CPU_SUP_CENTAUR=y
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
CONFIG_HPET_TIMER=y CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
# CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT is not set # CONFIG_AMD_IOMMU is not set CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_IOMMU_API is not set
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=16
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
# CONFIG_X86_MCE_AMD is not set CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_X86_CPU_DEBUG is not set CONFIG_ARCH_PHYS_ADDR_T_64BIT=y CONFIG_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set CONFIG_ARCH_SPARSEMEM_DEFAULT=y CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000 CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y CONFIG_HAVE_MEMORY_PRESENT=y CONFIG_SPARSEMEM_EXTREME=y CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_PHYS_ADDR_T_64BIT=y CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y CONFIG_UNEVICTABLE_LRU=y
CONFIG_HAVE_MLOCK=y CONFIG_HAVE_MLOCKED_PAGE_BIT=y
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set CONFIG_X86_RESERVE_LOW_64K=y
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set CONFIG_PHYSICAL_START=0x1000000 CONFIG_RELOCATABLE=y CONFIG_PHYSICAL_ALIGN=0x1000000 CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
# CONFIG_CMDLINE_BOOL is not set CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
# CONFIG_ACPI_PROC_EVENT is not set
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y CONFIG_CPU_FREQ_DEBUG=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set # CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_SPEEDSTEP_CENTRINO=y
# CONFIG_X86_P4_CLOCKMOD is not set

#

# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_DMAR is not set
# CONFIG_INTR_REMAP is not set
CONFIG_PCIEPORTBUS=y
# CONFIG_HOTPLUG_PCI_PCIE is not set
CONFIG_PCIEAER=y
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
# CONFIG_HT_IRQ is not set
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
# CONFIG_PCCARD is not set
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_FAKE is not set
# CONFIG_HOTPLUG_PCI_ACPI is not set
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set CONFIG_IP_FIB_HASH=y CONFIG_IP_MULTIPLE_TABLES=y CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set CONFIG_INET_TUNNEL=y CONFIG_INET_XFRM_MODE_TRANSPORT=y CONFIG_INET_XFRM_MODE_TUNNEL=y
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_LRO is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set CONFIG_TCP_CONG_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
# CONFIG_IPV6_PRIVACY is not set
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set # CONFIG_INET6_XFRM_MODE_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_BEET is not set
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set CONFIG_IPV6_SIT=y CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_NETLABEL is not set CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
# CONFIG_NETFILTER_NETLINK_LOG is not set
CONFIG_NF_CONNTRACK=y
# CONFIG_NF_CONNTRACK_SECMARK is not set
CONFIG_NF_CONNTRACK_FTP=y
# CONFIG_NF_CONNTRACK_IRC is not set
# CONFIG_NF_CONNTRACK_SIP is not set
# CONFIG_NF_CT_NETLINK is not set
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
# CONFIG_NETFILTER_XT_TARGET_NFLOG is not set
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
# CONFIG_NETFILTER_XT_TARGET_TCPMSS is not set
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
# CONFIG_NETFILTER_XT_MATCH_POLICY is not set
CONFIG_NETFILTER_XT_MATCH_STATE=y
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_NF_NAT_FTP=y
# CONFIG_NF_NAT_IRC is not set
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
# CONFIG_NF_NAT_SIP is not set
CONFIG_IP_NF_MANGLE=y

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_IP6_NF_TARGET_LOG=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_REJECT=y
# CONFIG_IP6_NF_MANGLE is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
# CONFIG_NET_SCH_CBQ is not set
# CONFIG_NET_SCH_HTB is not set
# CONFIG_NET_SCH_HFSC is not set
# CONFIG_NET_SCH_PRIO is not set
# CONFIG_NET_SCH_MULTIQ is not set
# CONFIG_NET_SCH_RED is not set
# CONFIG_NET_SCH_SFQ is not set
# CONFIG_NET_SCH_TEQL is not set
# CONFIG_NET_SCH_TBF is not set
# CONFIG_NET_SCH_GRED is not set
# CONFIG_NET_SCH_DSMARK is not set
# CONFIG_NET_SCH_NETEM is not set
# CONFIG_NET_SCH_DRR is not set
# CONFIG_NET_SCH_INGRESS is not set

#
# Classification
#
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
# CONFIG_NET_CLS_FW is not set
# CONFIG_NET_CLS_U32 is not set
# CONFIG_NET_CLS_RSVP is not set
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
# CONFIG_NET_EMATCH_CMP is not set
# CONFIG_NET_EMATCH_NBYTE is not set
# CONFIG_NET_EMATCH_U32 is not set
# CONFIG_NET_EMATCH_META is not set
# CONFIG_NET_EMATCH_TEXT is not set
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=y
# CONFIG_NET_ACT_GACT is not set
# CONFIG_NET_ACT_MIRRED is not set
# CONFIG_NET_ACT_IPT is not set
# CONFIG_NET_ACT_NAT is not set
# CONFIG_NET_ACT_PEDIT is not set
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_SCH_FIFO=y
# CONFIG_DCB is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NET_DROP_MONITOR is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
CONFIG_SYS_HYPERVISOR=y
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
CONFIG_BLK_CPQ_DA=y
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_ISL29003 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
# CONFIG_EEPROM_LEGACY is not set
# CONFIG_EEPROM_93CX6 is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs #
CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_SCSI_CXGB3_ISCSI is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_MPT2SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=y
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
CONFIG_PATA_OLDPIIX=y
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
# CONFIG_MD_RAID1 is not set
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID456 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
# CONFIG_DM_CRYPT is not set
CONFIG_DM_SNAPSHOT=y
CONFIG_DM_MIRROR=y
CONFIG_DM_ZERO=y
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# Enable only one of the two stacks, unless you know what you are doing #
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_COMPAT_NET_DEV_OPS=y
# CONFIG_IFB is not set
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
# CONFIG_REALTEK_PHY is not set
# CONFIG_NATIONAL_PHY is not set
# CONFIG_STE10XP is not set
# CONFIG_LSI_ET1011C_PHY is not set
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=y
# CONFIG_TYPHOON is not set
# CONFIG_ETHOC is not set
# CONFIG_DNET is not set CONFIG_NET_TULIP=y
# CONFIG_DE2104X is not set
# CONFIG_TULIP is not set
# CONFIG_DE4X5 is not set
# CONFIG_WINBOND_840 is not set
# CONFIG_DM9102 is not set
# CONFIG_ULI526X is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set # CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set # CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
CONFIG_FORCEDETH=y
# CONFIG_FORCEDETH_NAPI is not set CONFIG_E100=y
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SMSC9420 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_ATL2 is not set CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=y
CONFIG_E1000E=y
# CONFIG_IP1000 is not set
CONFIG_IGB=y
# CONFIG_IGBVF is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set CONFIG_TIGON3=y
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_JME is not set CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set CONFIG_CHELSIO_T3_DEPENDS=y
# CONFIG_CHELSIO_T3 is not set
# CONFIG_ENIC is not set
# CONFIG_IXGBE is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set
# CONFIG_MYRI10GE is not set
# CONFIG_NIU is not set
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_QLGE is not set
# CONFIG_SFC is not set
# CONFIG_BE2NET is not set
CONFIG_TR=y
# CONFIG_IBMOL is not set
# CONFIG_3C359 is not set
# CONFIG_TMS380TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers #

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_FDDI=y
# CONFIG_DEFXX is not set
# CONFIG_SKFP is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set
CONFIG_XEN_KBDDEV_FRONTEND=y

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
CONFIG_INPUT_JOYSTICK=y
# CONFIG_JOYSTICK_ANALOG is not set
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_ZHENHUA is not set
# CONFIG_JOYSTICK_JOYDUMP is not set
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_AD7879_I2C is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_ELO is not set
# CONFIG_TOUCHSCREEN_WACOM_W8001 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_COMPUTONE is not set
# CONFIG_ROCKETPORT is not set
# CONFIG_CYCLADES is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
# CONFIG_MOXA_SMARTIO is not set
# CONFIG_ISI is not set
# CONFIG_SYNCLINK is not set
# CONFIG_SYNCLINKMP is not set
# CONFIG_SYNCLINK_GT is not set
# CONFIG_N_HDLC is not set
# CONFIG_RISCOM8 is not set
# CONFIG_SPECIALIX is not set
# CONFIG_SX is not set
# CONFIG_RIO is not set
CONFIG_STALDRV=y
# CONFIG_STALLION is not set
# CONFIG_ISTALLION is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_MMAP is not set
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
# CONFIG_I2C_CHARDEV is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip) #
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Graphics adapter I2C/DDC channel drivers
#
# CONFIG_I2C_VOODOO3 is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_BQ27x00 is not set
CONFIG_HWMON=y
# CONFIG_HWMON_VID is not set
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7473 is not set
# CONFIG_SENSORS_ADT7475 is not set
# CONFIG_SENSORS_K8TEMP is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATK0110 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
# CONFIG_THERMAL_HWMON is not set
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
# CONFIG_W83697UG_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_REGULATOR is not set

#
# Multimedia devices
#

#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set

#
# Multimedia drivers
#
# CONFIG_DAB is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
CONFIG_DRM_I915=y
# CONFIG_DRM_I915_KMS is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
# CONFIG_FB_DDC is not set
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=y
CONFIG_FB_SYS_COPYAREA=y
CONFIG_FB_SYS_IMAGEBLIT=y
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=y
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_VESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
# CONFIG_BACKLIGHT_CLASS_DEVICE is not set

#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=y

#
# Display hardware drivers
#

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set # CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set # CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_JACK=y
CONFIG_SND_SEQUENCER=y
# CONFIG_SND_SEQ_DUMMY is not set CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y CONFIG_SND_PCM_OSS_PLUGINS=y CONFIG_SND_SEQUENCER_OSS=y CONFIG_SND_HRTIMER=y CONFIG_SND_SEQ_HRTIMER_DEFAULT=y CONFIG_SND_RTCTIMER=y
# CONFIG_SND_DYNAMIC_MINORS is not set CONFIG_SND_SUPPORT_OLD_API=y
# CONFIG_SND_VERBOSE_PROCFS is not set
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set CONFIG_SND_VMASTER=y
CONFIG_SND_AC97_CODEC=y
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
# CONFIG_SND_AC97_POWER_SAVE is not set CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set CONFIG_SND_HDA_INTEL=y
# CONFIG_SND_HDA_HWDEP is not set CONFIG_SND_HDA_INPUT_BEEP=y CONFIG_SND_HDA_CODEC_REALTEK=y CONFIG_SND_HDA_CODEC_ANALOG=y CONFIG_SND_HDA_CODEC_SIGMATEL=y CONFIG_SND_HDA_CODEC_VIA=y CONFIG_SND_HDA_CODEC_ATIHDMI=y
# CONFIG_SND_HDA_CODEC_NVHDMI is not set CONFIG_SND_HDA_CODEC_INTELHDMI=y CONFIG_SND_HDA_ELD=y CONFIG_SND_HDA_CODEC_CONEXANT=y CONFIG_SND_HDA_CODEC_CMEDIA=y CONFIG_SND_HDA_CODEC_SI3054=y CONFIG_SND_HDA_GENERIC=y
# CONFIG_SND_HDA_POWER_SAVE is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set CONFIG_SND_INTEL8X0=y
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set CONFIG_SND_USB=y
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set CONFIG_AC97_BUS=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_CYPRESS=y
# CONFIG_DRAGONRISE_FF is not set
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_TOPSEED=y
CONFIG_THRUSTMASTER_FF=y
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=y
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may #

#

# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=y
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set

#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I5400 is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
# CONFIG_EDAC_AMD8131 is not set
# CONFIG_EDAC_AMD8111 is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
CONFIG_XEN_BALLOON=y
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_DELL_WMI is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_EEEPC_LAPTOP is not set
CONFIG_ACPI_WMI=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_DEFAULTS_TO_ORDERED is not set
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_BTRFS_FS=y
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=y
# CONFIG_FUSE_FS is not set
CONFIG_GENERIC_ACL=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
# CONFIG_CONFIGFS_FS is not set
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NETWORK_FILESYSTEMS=y
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
# CONFIG_KARMA_PARTITION is not set
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ALLOW_WARNINGS=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
# CONFIG_DETECT_HUNG_TASK is not set
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_HW_BRANCH_TRACER=y
CONFIG_HAVE_FTRACE_SYSCALLS=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_TRACING=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_CONTEXT_SWITCH_TRACER is not set
# CONFIG_ENABLE_EVENT_TRACING is not set
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_BOOT_TRACER is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_POWER_TRACER is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_HW_BRANCH_TRACER is not set
# CONFIG_KMEMTRACE is not set
# CONFIG_WORKQUEUE_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_X86_VERBOSE_BOOTUP is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
CONFIG_X86_DS_SELFTEST=y
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
# CONFIG_SECURITYFS is not set
CONFIG_SECURITY_NETWORK=y
# CONFIG_SECURITY_NETWORK_XFRM is not set
# CONFIG_SECURITY_PATH is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_SECURITY_ROOTPLUG is not set
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
# CONFIG_SECURITY_TOMOYO is not set
# CONFIG_IMA is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
# CONFIG_CRYPTO_FIPS is not set
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
CONFIG_CRYPTO_PCBC=y
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=y
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
# CONFIG_CRYPTO_AES is not set
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
# CONFIG_VIRTUALIZATION is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
# CONFIG_CRC_CCITT is not set
# CONFIG_CRC16 is not set
CONFIG_CRC_T10DIF=y
# CONFIG_CRC_ITU_T is not set
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y

#25 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 2 lines · Ingo Molnar

I did more 'perf stat mmap-perf 1' measurements (bound to a single core, running single thread - to exclude cross-CPU noise), which in essence measures CONFIG_PARAVIRT=y overhead on native kernels:

Thanks for taking the time to make these measurements. You'll agree they're much better numbers than the last time you ran these tests?

Quoted 10 lines

Performance counter stats for './mmap-perf':

      [vanilla]  [PARAVIRT=y]

    1230.805297   1242.828348  task clock ticks     (msecs)   + 0.97%
     3602663413    3637329004  CPU cycles           (events)  + 0.96%
     1927074043    1958330813  instructions         (events)  + 1.62%

That's around 1% on really fast hardware (Core2 E6800 @ 2.93 GHz, 4MB L2 cache), i.e. still significant overhead. Distros generally enable CONFIG_PARAVIRT, even though a large majority of users never actually runs them as Xen guests.

Did you do only a single run, or is this the result of multiple runs? If so, what was your procedure? How did you control for page placement/cache effects/other boot-to-boot variations?

Your numbers are not dissimilar to my measurements, but I also saw up to 1% performance improvement vs native from boot to boot (I saw up to 10% reduction of cache misses with pvops, possibly because of its de-inlining effects).

I also saw about 1% boot to boot variation with the non-pvops kernel.

While I think pvops does add *some* overhead, I think the absolute magnitude is swamped in the noise. The best we can say is "somewhere under 1% on modern hardware".

Quoted 6 lines

Are there plans to analyze and fix this overhead too, beyond the paravirt-spinlocks overhead you analyzed? (Note that i had CONFIG_PARAVIRT_SPINLOCKS disabled in this test.)

I think only those users should get overhead who actually run such kernels in a virtualized environment.

I cannot cite a single other kernel feature that has so much performance impact when runtime-disabled. For example, an often cited bloat and overhead source is CONFIG_SECURITY=y.

Your particular benchmark does many, many mmap/mprotect/munmap/mremap calls, and takes a lot of pagefaults. That's going to hit the hot path with lots of pte updates and so on, but very few security hooks. How does it compare with a more balanced workload?

Quoted 8 lines

Its runtime overhead (same system, same workload) is:

      [vanilla]  [SECURITY=y]

    1219.652255   1230.805297  task clock ticks     (msecs)   + 0.91%
     3574548461    3602663413  CPU cycles           (events)  + 0.78%
     1915177924    1927074043  instructions         (events)  + 0.62%
Quoted 5 lines

( With the difference that the distros that enable CONFIG_SECURITY=y
tend to install and use at least one security module by default. )

So everyone who runs a CONFIG_PARAVIRT=y distro kernel has 1% of overhead in this mmap-test workload - even if no Xen is used on that box, ever.

So you're saying that:

* CONFIG_SECURITY adding +0.91% to wallclock time is OK, but pvops

     adding +0.97% is not,

* your test is sensitive enough to make 0.06% difference significant, and
* this benchmark is representative enough of real workloads that its results are overall meaningful?

Quoted 2 lines

Config attached.

Is this derived from a RH distro config?

J

#26 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 71 lines · Jeremy Fitzhardinge
Quoted 4 lines · Ingo Molnar

I did more 'perf stat mmap-perf 1' measurements (bound to a single core, running single thread - to exclude cross-CPU noise), which in essence measures CONFIG_PARAVIRT=y overhead on native kernels:

Thanks for taking the time to make these measurements. You'll agree they're much better numbers than the last time you ran these tests?

Quoted 4 lines

Performance counter stats for './mmap-perf':

[vanilla] [PARAVIRT=y]

Quoted 9 lines
   1230.805297   1242.828348  task clock ticks     (msecs)   + 0.97%
    3602663413    3637329004  CPU cycles           (events)  + 0.96%
    1927074043    1958330813  instructions         (events)  + 1.62%

That's around 1% on really fast hardware (Core2 E6800 @ 2.93 GHz, 4MB L2 cache), i.e. still significant overhead. Distros generally enable CONFIG_PARAVIRT, even though a large majority of users never actually runs them as Xen guests.

Did you do only a single run, or is this the result of multiple runs? If so, what was your procedure? How did you control for page placement/cache effects/other boot-to-boot variations?

Your numbers are not dissimilar to my measurements, but I also saw up to 1% performance improvement vs native from boot to boot (I saw up to 10% reduction of cache misses with pvops, possibly because of its de-inlining effects).

I also saw about 1% boot to boot variation with the non-pvops kernel.

While I think pvops does add *some* overhead, I think the absolute magnitude is swamped in the noise. The best we can say is "somewhere under 1% on modern hardware".

Quoted 11 lines

Are there plans to analyze and fix this overhead too, beyond the paravirt-spinlocks overhead you analyzed? (Note that i had CONFIG_PARAVIRT_SPINLOCKS disabled in this test.)

I think only those users should get overhead who actually run such kernels in a virtualized environment.

I cannot cite a single other kernel feature that has so much performance impact when runtime-disabled. For example, an often cited bloat and overhead source is CONFIG_SECURITY=y.

Your particular benchmark does many, many mmap/mprotect/munmap/mremap calls, and takes a lot of pagefaults. That's going to hit the hot path with lots of pte updates and so on, but very few security hooks. How does it compare with a more balanced workload?

Quoted 8 lines

Its runtime overhead (same system, same workload) is:

[vanilla] [SECURITY=y]

   1219.652255   1230.805297  task clock ticks     (msecs)   + 0.91%
    3574548461    3602663413  CPU cycles           (events)  + 0.78%
    1915177924    1927074043  instructions         (events)  + 0.62%
Quoted 7 lines

( With the difference that the distros that enable CONFIG_SECURITY=y
tend to install and use at least one security module by default. )

So everyone who runs a CONFIG_PARAVIRT=y distro kernel has 1% of overhead in this mmap-test workload - even if no Xen is used on that box, ever.

Quoted 8 lines

So you're saying that:

* CONFIG_SECURITY adding +0.91% to wallclock time is OK, but pvops

     adding +0.97% is not,

* your test is sensitive enough to make 0.06% difference significant, and
* this benchmark is representative enough of real workloads that its results are overall meaningful?

FWIW, we had to disable paravirt in our default SLES11 kernel. (admittedly this was before some of the recent improvements were made). But there are only so many 1% performance regressions you can introduce before customers won't upgrade (or vendors won't publish benchmarks with the new software).

But OTOH, almost any bit feature is going to cost performance. I don't think this is something new (as noted with CONFIG_SECURITY). It is just something people have to trade off and decide for themselves. If you make it configurable and keep performance as good as reasonably possible, then I don't think more can be asked.

If performance overhead is too much and/or too few users can take advantage of a feature, then distros can always special-case it. I think may did for pae...

#27 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 3 lines · Nick Piggin

FWIW, we had to disable paravirt in our default SLES11 kernel. (admittedly this was before some of the recent improvements were made).

Yes, I think you'll find it worth trying with it enabled again. The spinlock thing clearly slowed things down, but when that's disabled the difference to native is very small.

Quoted 6 lines

But OTOH, almost any bit feature is going to cost performance. I don't think this is something new (as noted with CONFIG_SECURITY). It is just something people have to trade off and decide for themselves. If you make it configurable and keep performance as good as reasonably possible, then I don't think more can be asked.

Yes, that's exactly my point. If I've worked on a feature, I clearly want people to use that feature. Part of making it useful is to make the distro/vendor/user decision to enable that feature as easy as possible, by making the tradeoffs simple.

But tradeoffs are always going to cut both ways: positive (kernel automatically works in a wider range of environments), and negative (performance questions, complexity, etc). Ultimately its the distro's decision to enable a particular feature, and the distro's responsibility to cope with the consequences of that.

Quoted 3 lines

If performance overhead is too much and/or too few users can take advantage of a feature, then distros can always special-case it. I think may did for pae...

I think that would be a clear sign of a problem. The whole point of pvops is to avoid needing multiple kernel builds.

J

#28 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Nick Piggin
Quoted 8 lines · Nick Piggin

FWIW, we had to disable paravirt in our default SLES11 kernel. (admittedly this was before some of the recent improvements were made). But there are only so many 1% performance regressions you can introduce before customers won't upgrade (or vendors won't publish benchmarks with the new software).

But OTOH, almost any bit feature is going to cost performance. I don't think this is something new (as noted with CONFIG_SECURITY). [...]

Yes in a way, but the difference is that:

- i noted CONFIG_SECURITY as the _worst current example_. It is the largest-overhead feature known to me in this area, and i benchmark the kernel a lot. CONFIG_PARAVIRT has _even more_ overhead so it takes the (dubious) top spot in this category.

- CONFIG_SECURITY, in the distros where it's enabled (most of them) it is actually being relied on by the default user-space. It's being configured and every default install of the distro has a real (or at least perceived) advantage from it.

Not so with CONFIG_PARAVIRT. That feature is almost fully parasitic to native environments: currently it brings no advantage on native hardware _at all_ (and 95% of the users dont care about Xen).

Still it's impractical for a distro to disable it because adding a separate kernel package has high costs too and PARAVIRT=y is needed for the weird execution environment that Xen requires to run Linux with acceptable speed.

It's as if we paid a full 1% overhead from the requirements of say Centaur CPUs, on all other CPUs (Intel and AMD). That would be inacceptable to owners of Intel and AMD CPUs: and so would it be inacceptable to distros to have a separate kernel package for it.

A distro is unable to hold the tide of creeping bloat - they dont have the long-term view (and probably shouldnt have it - a distro should care about and maximize the here-and-now utility of Linux mostly). Upstream maintainers are the ones to say "NO" to such crap, even if it's unpopular. In this current case that's me i guess.

Note what _is_ acceptable and what _is_ doable is to be a bit more inventive when dumping this optional, currently-high-overhead paravirt feature on us. My message to Xen folks is: use dynamic patching, fix your hypervisor and just use plain old-fashioned _restraint_ and common sense when engineering things, and for heaven's sake, _care_ about the native kernel's performance because in the long run it's your bread and butter too.

1% overhead (paid by _everyone_ who runs that distro kernel) for something 95%+ of the users _wont ever use_ is _not acceptable_.

So unless this overhead is brought down significantly i see the dom0 patches as outright harmful: upstream dom0 makes PARAVIRT=Y even _harder_ for distros to disable, and there will be even _less_ incentive for Xen folks to get the native bloat they caused under control.

Note that specific pieces of the dom0 patches, like the io-apic driver patches, are still acceptable of course, as they improve the overall quality of the kernel. Helping dom0 too as a 'side effect' is fully acceptable as long as everyone else is helped too.

And if the performance problems are largely fixed (say 0.1%-0.2% overhead in the benchmarks i did would be acceptable IMO), and if the patches are all squeaky-clean, i suspect we can do upstream dom0 as well.

Ingo

#29 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 24 lines · Ingo Molnar
Quoted 8 lines · Nick Piggin

FWIW, we had to disable paravirt in our default SLES11 kernel. (admittedly this was before some of the recent improvements were made). But there are only so many 1% performance regressions you can introduce before customers won't upgrade (or vendors won't publish benchmarks with the new software).

But OTOH, almost any bit feature is going to cost performance. I don't think this is something new (as noted with CONFIG_SECURITY). [...]

Yes in a way, but the difference is that:

- i noted CONFIG_SECURITY as the _worst current example_. It is the largest-overhead feature known to me in this area, and i benchmark the kernel a lot. CONFIG_PARAVIRT has _even more_ overhead so it takes the (dubious) top spot in this category.

- CONFIG_SECURITY, in the distros where it's enabled (most of them) it is actually being relied on by the default user-space. It's being configured and every default install of the distro has a real (or at least perceived) advantage from it.

Quoted 8 lines

Not so with CONFIG_PARAVIRT. That feature is almost fully parasitic to native environments: currently it brings no advantage on native hardware _at all_ (and 95% of the users dont care about Xen).

Still it's impractical for a distro to disable it because adding a separate kernel package has high costs too and PARAVIRT=y is needed for the weird execution environment that Xen requires to run Linux with acceptable speed.

I think all of the distros expect mainline to do a difficult balancing act. On the one hand we don't want any performance regressions, and on the other hand, we need to be able to work with mainline to get the major features that we're shipping and using every day in.

What we have here is a CONFIG_ that would be heavily used if it were included. It may be used as part of a separate kernel build, or it may be used for a single kernel image. Either way, this is something for the distros to decide. You're arguing to leave it out of mainline because making a separate kernel package for it is too hard, but by doing so forcing them to maintain the patches out of tree. They still have the same decision about the second package either way.

The xen developers are not dropping this code on mainline and running away to spend the rest of their lives on the beach. This is is a step in long term development, and keeping it out of the kernel is making testing and continued development much harder on the users.

I'm not suggesting we should take broken code, or that we should lower standards just for xen. But, expecting the xen developers to fix the 1% hit on a very specific micro-benchmark is not a way to promote new projects for the kernel, and it isn't a good way to convince people to do continued development in mainline instead of in private trees.

Please reconsider. Keeping these patches out is only making it harder on the people that want to make them better.

-chris

#30 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 5 lines · Chris Mason

I'm not suggesting we should take broken code, or that we should lower standards just for xen.  But, expecting the xen developers to fix the 1% hit on a very specific micro-benchmark is not a way to promote new projects for the kernel, and it isn't a good way to convince people to do continued development in mainline instead of in private trees.

It's not a new project which needs to be treated with kid's gloves. And one be sure that once the code is in the tree those interested parties will not be as strongly motivated to fix any problem like this. Ingo pointed to a way which doesn't negatively impact the performance of the Xen kernel and reduces the overhead (dynamic patching). Just get started on this (and general cleanup) and this whole argument will go away.

I find it ridiculous to use the "but it's used" argument to try to force the code into the kernel. By this argument you can say the same about crap like ndiswrapper and similarly harmful code.

#31 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 8 lines · Ulrich Drepper
Quoted 5 lines · Chris Mason

I'm not suggesting we should take broken code, or that we should lower standards just for xen.  But, expecting the xen developers to fix the 1% hit on a very specific micro-benchmark is not a way to promote new projects for the kernel, and it isn't a good way to convince people to do continued development in mainline instead of in private trees.

It's not a new project which needs to be treated with kid's gloves.

Sure, I'm not suggesting we send them flowers on mothers day or anything, and I'm not suggesting they skip out on important cleanups. But, I strongly object to a 1% hit on a random micro benchmark as a reason to keep the code out.

Quoted 3 lines

And one be sure that once the code is in the tree those interested parties will not be as strongly motivated to fix any problem like this.

The idea that people shipping xen aren't interested in performance regressions is really strange to me.

Quoted 4 lines

Ingo pointed to a way which doesn't negatively impact the performance of the Xen kernel and reduces the overhead (dynamic patching). Just get started on this (and general cleanup) and this whole argument will go away.

Dynamic patching is a big wad of duct tape over the problem.

Quoted 4 lines

I find it ridiculous to use the "but it's used" argument to try to force the code into the kernel. By this argument you can say the same about crap like ndiswrapper and similarly harmful code.

I'm not saying to take harmful code, I'm saying to take code with a small performance regression under a specific CONFIG_. Slub regresses more than 1% on database loads, CONFIG_SCHED_GROUPS, the list goes on and on.

The idea that we should take code that is heavily used is important. The best place to fix xen is in the kernel. It always has been, and keeping it out is just making it harder on everyone involved.

-chris

#32 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 2 lines · Chris Mason

The idea that people shipping xen aren't interested in performance regressions is really strange to me.

Why? They have a different base line. For them any regression to bare hardware performance is even a positive (since it means the gap between hardware and virt shrinks).

Quoted 1 line

Dynamic patching is a big wad of duct tape over the problem.

And what do you call the Xen model? It's a perfect fit IMO.

Quoted 4 lines

I'm not saying to take harmful code, I'm saying to take code with a small performance regression under a specific CONFIG_.  Slub regresses more than 1% on database loads, CONFIG_SCHED_GROUPS, the list goes on and on.

None of those have to be enabled in default kernels.

Quoted 1 line

The best place to fix xen is in the kernel.

No. The best way to fix things is _on the way into the kernel_.

#33 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 7 lines · Ulrich Drepper
Quoted 2 lines · Chris Mason

The idea that people shipping xen aren't interested in performance regressions is really strange to me.

Why? They have a different base line. For them any regression to bare hardware performance is even a positive (since it means the gap between hardware and virt shrinks).

And we would have gotten away with it too if it weren't for you meddling kids!

Quoted 11 lines
Quoted 3 lines

Dynamic patching is a big wad of duct tape over the problem.

And what do you call the Xen model? It's a perfect fit IMO.

Quoted 4 lines

I'm not saying to take harmful code, I'm saying to take code with a small performance regression under a specific CONFIG_.  Slub regresses more than 1% on database loads, CONFIG_SCHED_GROUPS, the list goes on and on.

Quoted 6 lines

None of those have to be enabled in default kernels.

Quoted 1 line

The best place to fix xen is in the kernel.

No. The best way to fix things is _on the way into the kernel_.

It all depends on which parts are causing problems. A 1% performance hit, under a CONFIG_ that can be disabled? If maintainers are focusing on details like this for long term and active projects, we're doing something very wrong.

-chris

#35 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Chris Mason

Hi Chris,

Quoted 8 lines · Chris Mason
Quoted 3 lines

I find it ridiculous to use the "but it's used" argument to try to force the code into the kernel.  By this argument you can say the same about crap like ndiswrapper and similarly harmful code.

I'm not saying to take harmful code, I'm saying to take code with a small performance regression under a specific CONFIG_.  Slub regresses more than 1% on database loads, CONFIG_SCHED_GROUPS, the list goes on and on.

Maybe it's just me but you make it sound like the SLUB regression is okay. It's not.

Unfortunately we're now in a position where we can't just remove SLUB (it's an improvement over SLAB for NUMA) so we're stuck with two allocators with third one on its way to the kernel. So yes, it makes a lot of sense to me to fix CONFIG_PARAVIRT regression before merging more of the xen stuff in the kernel. It's always easier to fix these things before they hit the kernel and people start to depend on them.

Pekka

#34 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Chris Mason

Hi Chris,

Quoted 1 line · Chris Mason
Quoted 1 line
Quoted 1 line

The best place to fix xen is in the kernel.

Quoted 1 line · Ulrich Drepper
Quoted 1 line

No.  The best way to fix things is _on the way into the kernel_.

Quoted 4 lines · Chris Mason

It all depends on which parts are causing problems.  A 1% performance hit, under a CONFIG_ that can be disabled?  If maintainers are focusing on details like this for long term and active projects, we're doing something very wrong.

The fact that CONFIG_PARAVIRT can be disabled doesn't really help. As a matter of fact, I'd argue that one of the primary reasons CONFIG_SLUB regression is still there is because people can just disable it and use CONFIG_SLAB instead.

So I think we have some evidence to suggest that people have less incentive to fix things once something is merged to the kernel. And I don't mean the authors of the code here but basically _everyone_ involved in kernel development. It usually takes effort from variety of people to get everything ironed out because, lets face it, we can't expect a handful of people to test out every configuration let alone fix them.

Pekka

#36 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Pekka Enberg
Quoted 7 lines · Pekka Enberg

Hi Chris,

Quoted 4 lines · Chris Mason
Quoted 3 lines

I find it ridiculous to use the "but it's used" argument to try to force the code into the kernel.  By this argument you can say the same about crap like ndiswrapper and similarly harmful code.

Quoted 7 lines
Quoted 4 lines

I'm not saying to take harmful code, I'm saying to take code with a small performance regression under a specific CONFIG_.  Slub regresses more than 1% on database loads, CONFIG_SCHED_GROUPS, the list goes on and on.

Maybe it's just me but you make it sound like the SLUB regression is okay. It's not.

Well, it is and it isn't. SLUB was implemented with specific workloads in mind. I'd prefer that regressions not get in at all, but sometimes it takes the broad exposure you get from being in mainline to finish things. Sometimes we finish things with rm, but without slub I don't think the issues it was trying to solve would have been discussed at all.

Quoted 7 lines

Unfortunately we're now in a position where we can't just remove SLUB (it's an improvement over SLAB for NUMA) so we're stuck with two allocators with third one on its way to the kernel. So yes, it makes a lot of sense to me to fix CONFIG_PARAVIRT regression before merging more of the xen stuff in the kernel. It's always easier to fix these things before they hit the kernel and people start to depend on them.

The problem is that people already depend on them ;) If people want to nack xen based on code structure, that's more than fair, I just hope we can keep the discussion around something the xen developers can work toward.

Micro benchmarks come and go, we tune as best we can based on the tradeoffs at hand.

-chris

#38 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Chris Mason
Quoted 8 lines · Chris Mason

I'm not suggesting we should take broken code, or that we should lower standards just for xen. But, expecting the xen developers to fix the 1% hit on a very specific micro-benchmark is not a way to promote new projects for the kernel, and it isn't a good way to convince people to do continued development in mainline instead of in private trees.

Please reconsider. Keeping these patches out is only making it harder on the people that want to make them better.

You are missing one subtle point.

I read several times, that A, B and C can not be changed design wise to allow newer kernels to run on older hypervisors. That's what frightens me:

dom0 imposes a kind of ABI which we can not change anymore.

So where is the room for the improvements which you expect when dom0 is merged ? It's not about micro benchmark results, it's about the inability to fix the existing design decisions in the near future.

You can change the internals of btrfs as often as you want, but you can not change the on disk layout at will. And while you can invent btrfs2 w/o any impact aside of grumpy users and a couple of thousand lines self contained code, dom0v2 would just add a different layer of intrusiveness into the x86 code base w/o removing the existing one.

Thanks,

tglx

#39 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 7 lines · Thomas Gleixner
Quoted 6 lines · Chris Mason

I'm not suggesting we should take broken code, or that we should lower standards just for xen. But, expecting the xen developers to fix the 1% hit on a very specific micro-benchmark is not a way to promote new projects for the kernel, and it isn't a good way to convince people to do continued development in mainline instead of in private trees.

Quoted 5 lines
Quoted 2 lines

Please reconsider. Keeping these patches out is only making it harder on the people that want to make them better.

You are missing one subtle point.

I'm sure I'm missing many more than one ;)

Quoted 10 lines

I read several times, that A, B and C can not be changed design wise to allow newer kernels to run on older hypervisors. That's what frightens me:

dom0 imposes a kind of ABI which we can not change anymore.

So where is the room for the improvements which you expect when dom0 is merged ? It's not about micro benchmark results, it's about the inability to fix the existing design decisions in the near future.

Quoted 5 lines

You can change the internals of btrfs as often as you want, but you can not change the on disk layout at will. And while you can invent btrfs2 w/o any impact aside of grumpy users and a couple of thousand lines self contained code, dom0v2 would just add a different layer of intrusiveness into the x86 code base w/o removing the existing one.

Well, if there's a line we want to draw in the sand based on firm and debatable criteria, great.

The problem I see here is that our line in the sand for the xen developers is fuzzy and winding (yeah, I saw Linus' reply in the other thread, full ack on that).

-chris

#37 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ulrich Drepper
Quoted 3 lines · Ulrich Drepper

Ingo pointed to a way which doesn't negatively impact the performance of the Xen kernel and reduces the overhead (dynamic patching)

The pvops code is already fully dynamically patched, which replaces all the indirect calls with either direct calls, inline instructions or nops. It has been this way from the initial implementation.

More recently I changed the calling convention on some of the most common critical-path ops to reduce the register pressure caused by the function call clobbers; you just don't need a pile of registers to disable interrupts.

Ingo knows all this, so I'm not sure what further patching he's suggesting. I don't see any more likely candidates, but I'm open to suggestions.

J

#40 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

Not so with CONFIG_PARAVIRT. That feature is almost fully parasitic to native environments: currently it brings no advantage on native hardware _at all_ (and 95% of the users dont care about Xen).

And VMI, and of course lguest. And a little bit of KVM, though not the paths you're talking about.

Your complaints are a little unfocussed: anything Xen could do to make this overhead go away, we would be able to do ourselves. Yet it's not clear where this 1% is going. We're more aggressive with our patching right now than any other subsystem; which call is the problem?

But your entire rant is willfully ignorant; if you've only just discovered that all commonly enabled config options have a cost when unused, I am shocked.

I took my standard config, and turned on AUDIT, CGROUP, all the sched options, all the namespace options, profiling, markers, kprobes, relocatable kernel, 1000Hz, preempt, support for every x86 variant (ie. PAE, NUMA, HIGHMEM64, DISCONTIGMEM). I turned off kernel debugging and paravirt. Booted with maxcpus=1.

I created another with SMP=n, and all that turned off. No highmem. 100Hz.

Then I ran virtbench which I had lying around, in local mode (ie. between processes running locally, rather than between guest OSes)

Distro-style maximal config:
Time for one context switch via pipe: 2285 (2276 - 2290) Time for one Copy-on-Write fault: 3415 (3264 - 4266)
Time to exec client once: 234656 (232906 - 253343)
Time for one fork/exit/wait: 82656 (82031 - 83218)
Time for gettimeofday(): 253 (253 - 254)
Time to send 4 MB from host: 6911750 (6901000 - 6925500) Time for one int-0x80 syscall: 284 (284 - 369)
Time for one syscall via libc: 139 (139 - 140)
Time to walk linear 64 MB: 760375 (754750 - 868125)
Time to walk random 64 MB: 955500 (947250 - 990000)
Time for two PTE updates: 3173 (3143 - 3196)
Time to read from disk (256 kB): 2395000 (2319000 - 2434500) Time for one disk read: 114156 (112906 - 114562)
Time to send 4 MB between guests: 7639000 (7568250 - 7739750) Time for inter-guest pingpong: 13900 (13800 - 13931)
Time to sendfile 4 MB between guests: 7187000 (7129000 - 46349000) Time to receive 1000 1k UDPs between guests: 6576000 (6500000 - 7232000)

Custom-style minimal config:
Time for one context switch via pipe: 1351 (1333 - 1405) Time for one Copy-on-Write fault: 2754 (2610 - 3586)
Time to exec client once: 190625 (189812 - 207500)
Time for one fork/exit/wait: 60968 (60875 - 61218)
Time for gettimeofday(): 248 (248 - 249)
Time to send 4 MB from host: 6643250 (6583750 - 6880750) Time for one int-0x80 syscall: 280 (280 - 334)
Time for one syscall via libc: 133 (133 - 144)
Time to walk linear 64 MB: 758750 (752375 - 835000)
Time to walk random 64 MB: 943500 (934500 - 1084000)
Time for two PTE updates: 1917 (1900 - 2401)
Time to read from disk (256 kB): 2390500 (2309000 - 2536000) Time for one disk read: 113250 (112937 - 113875)
Time to send 4 MB between guests: 7830500 (7740500 - 7946000) Time for inter-guest pingpong: 12566 (11621 - 13652)
Time to sendfile 4 MB between guests: 6533000 (5961000 - 76365000) Time to receive 1000 1k UDPs between guests: 5278000 (5194000 - 5431000)

Average slowdown: 15%
Worst: context switch, 69% slowdown.
Best: 4MB inter-process. 2% speedup (maybe due to more mem, 4G machine)

So in fact CONFIG_PARAVIRT's 1% makes it a paragon.

We should be praising Jeremy for his efforts and asking him to look at some of these others!

Sorry for the facts,
Rusty.

#41 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 6 lines · Rusty Russell

I took my standard config, and turned on AUDIT, CGROUP, all the sched options, all the namespace options, profiling, markers, kprobes, relocatable kernel, 1000Hz, preempt, support for every x86 variant (ie. PAE, NUMA, HIGHMEM64, DISCONTIGMEM). I turned off kernel debugging and paravirt. Booted with maxcpus=1.

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

It's one of those options that we do to support crazy hardware, and it is EXTREMELY expensive (but mainly only if you actually have the hardware, ie you actually have more than 1GB of RAM for HIGHMEM4G - HIGHMEM64G is always expensive for forks, but nobody sane ever enables it).

IOW, it's not at all comparable to the other options. It's not a software option, it's a real hardware option that hits you not depending on whether you want some sw capability, but on whether you want to use memory.

Because depending on the CPU, some loads will have 25% of time spent in just kmap/kunmap due to TLB flushes. Yes, really. There's a reason 32-bit kernels are shit for 1GB+ memory.

After you've turned off HIGHMEM (or run on a sane architecture like x86-64 that doesn't need it), re-run the benchmark, because it's interesting. But with HIGHMEM being different, your benchmark is totally invalid and pointless.

Linus

#43 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to a message not in the archive
Quoted 6 lines · Rusty Russell
Quoted 3 lines

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

Thanks, your point is demonstrated below. I don't think HIGHMEM4G is unreasonable for a distro tho, so I turned that on instead.

Well, I agree that HIGHMEM4G is a _reasonable_ thing to turn on.

The thing I disagree with is that it's at all valid to then compare to some all-software feature thing. HIGHMEM doesn't expand any esoteric capability that some people might use - it's about regular RAM for regular users.

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

So I think your minimum and maximum configs should at least _match_ in HIGHMEM. Limiting memory to not actually having any (with "mem=880M") will avoid the TLB flushing impact of HIGHMEM, which is clearly going to be the _bulk_ of the overhead, but HIGHMEM is still going to be noticeable on at least some microbenchmarks.

In other words, it's a lot like CONFIG_SMP, but at least CONFIG_SMP has a damn better reason for existing today than CONFIG_HIGHMEM.

That said, I suspect that now your context-switch test is likely no longer dominated by that thing, so looking at your numbers:

Quoted 3 lines

minimal config: ~0.001280
maximal config: ~0.002500 (with actual high mem)
maximum config: ~0.001925 (with mem=880M)

and I think that change from 0.001280 - 0.001925 (rough averages by eye-balling it, I didn't actually calculate anything) is still quite interesting, but I do wonder how much of it ends up being due to just code generation issues for CONFIG_HIGHMEM and CONFIG_SMP.

Quoted 2 lines

So we're paying a 48% overhead; microbenchmarks always suffer as code is added, and we've added a lot of code with these options.

I do agree that microbenchmarks are interesting, and tend to show these kinds of things clearly. It's just that when you look at the scheduler, for example, something like SMP support is a _big_ issue, and even if we get rid of the worst synchronization overhead with "maxcpus=1" at least removing the "lock" prefixes, I'm not sure how relevant it is to say that the scheduler is slower with SMP support.

(The same way I don't think it's relevant or interesting to see that it's slower with HIGHMEM).

They are simply so fundamental features that the two aren't comparable. Why would anybody compare a UP scheduler with a SMP scheduler? It's simply not the same problem. What does it mean to say that one is 48% slower? That's like saying that a squirrell is 48% juicier than an orange - maybe it's true, but anybody who puts the two in a blender to compare them is kind of sick. The comparison is ugly and pointless.

Now, other feature comparisons are way more interesting. For example, if statistics gathering is a noticeable portion of the 48%, then that really is a very relevant comparison, since scheduler statistics is something that is in no way "fundamental" to the hardware base, and most people won't care.

So comparing a "scheduler statistics" overhead vs "minimal config" overhead is very clearly a sane thing to do. Now we're talking about a feature that most people - even if it was somehow hardware related - wouldn't use or care about.

IOW, even if it were to use hardware features (say, something like oprofile, which is at least partly very much about exposing actual physical features of the hardware), if it's not fundamental to the whole usage for a huge percentage of people, then it's a "optional feature", and seeing slowdown is a big deal.

Something like CONFIG_HIGHMEM* or CONFIG_SMP is not really what I'd ever call "optional feature", although I hope to Dog that CONFIG_HIGHMEM can some day be considered that some day.

Linus

#44 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 7 lines · Linus Torvalds
Quoted 5 lines · Rusty Russell
Quoted 2 lines

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

Thanks, your point is demonstrated below. I don't think HIGHMEM4G is unreasonable for a distro tho, so I turned that on instead.

Quoted 6 lines

Well, I agree that HIGHMEM4G is a reasonable thing to turn on.

The thing I disagree with is that it's at all valid to then compare to some all-software feature thing. HIGHMEM doesn't expand any esoteric capability that some people might use - it's about regular RAM for regular users.

I think you're missing the point of Rusty's benchmark. I see his exercise as "compare a kernel configured as a distro would vs a custom-built kernel configured for the exact target environment". In that light, questions about the CONFIG options Rusty used should be based on whether most distros would use them in their stock kernels as opposed to how necessary they are.

What I see as the message of his benchmark is if you care about performance you should be customizing your kernel anyway. Distro kernels are slow. An option that makes the distro kernel a bit slower is no big deal since anyone who wants speed should already be rebuilding their kernel.

Don't get me wrong. I think it's always a good idea to minimize any performance penalty, even under specific configurations. I just think criticizing it because distros might enable it is a poor argument.

Dave McCracken
Oracle Corp.

#48 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

So I think your minimum and maximum configs should at least _match_ in HIGHMEM. Limiting memory to not actually having any (with "mem=880M") will avoid the TLB flushing impact of HIGHMEM, which is clearly going to be the _bulk_ of the overhead, but HIGHMEM is still going to be noticeable on at least some microbenchmarks.

Well, Ingo was ranting because (paraphrase) "no other config option when *unused* has as much impact as CONFIG_PARAVIRT!!!!!!!!!!".

That was the point of my mail; facts show it's simply untrue.

Quoted 1 line

The comparison is ugly and pointless.

(Re: SMP)

Distributions don't ship UP kernels any more; this shows what that costs if you're actually on a UP box. If we really don't care, perhaps we should make CONFIG_SMP=n an option under EMBEDDED for x86. And we can rip out the complex patching SMP patching stuff too.

Quoted 3 lines

Something like CONFIG_HIGHMEM* or CONFIG_SMP is not really what I'd ever call "optional feature", although I hope to Dog that CONFIG_HIGHMEM can some day be considered that some day.

Someone from a distro might know how many deployed machines don't need them. Kernel hackers tend to have modern machines; same with "enterprise" sites. I have no idea.

Without those facts, I'll leave further discussion to someone else :)

Thanks,
Rusty.

#45 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Dave McCracken

Hi,

Quoted 5 lines

I think you're missing the point of Rusty's benchmark. I see his exercise as "compare a kernel configured as a distro would vs a custom-built kernel configured for the exact target environment". In that light, questions about the CONFIG options Rusty used should be based on whether most distros would use them in their stock kernels as opposed to how necessary they are.

Well. The test ran on a machine with so much memory that you need HIGHMEM to use it all. I think it also was SMP. So a custom kernel for *that* machine would certainly include SMP and HIGHMEM ...

Quoted 2 lines

What I see as the message of his benchmark is if you care about performance you should be customizing your kernel anyway.

Sure. That wouldn't include turning off HIGHMEM and SMP though because you need them to make full use of your hardware. While it might be interesting by itself to see what the overhead of these config options is, it is IMHO quite pointless *in the context of this discussion*.

All the other options (namespaces, audit, statistics, whatnot) are different: You check whenever you want that $feature, if not you can turn it off. Distros tend to have them all turned on. So looking at the overhead of these config options when enabled + unused (and compare to the paravirt overhead) is certainly a valid thing.

cheers,
Gerd

#46 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 9 lines · Gerd Hoffmann
   Hi,
Quoted 6 lines

I think you're missing the point of Rusty's benchmark. I see his exercise as "compare a kernel configured as a distro would vs a custom-built kernel configured for the exact target environment". In that light, questions about the CONFIG options Rusty used should be based on whether most distros would use them in their stock kernels as opposed to how necessary they are.

Quoted 3 lines

Well. The test ran on a machine with so much memory that you need HIGHMEM to use it all. I think it also was SMP. So a custom kernel for *that* machine would certainly include SMP and HIGHMEM ...

I have a UP machine with 512M of RAM, but I wasn't going to take it out just to prove the point. Hence I used my test machine with mem=880 maxcpus=1 to simulate it, but that's a distraction here.

Quoted 3 lines

While it might be
interesting by itself to see what the overhead of these config options is, it is IMHO quite pointless *in the context of this discussion*.

No, you completely missed the point.
Rusty.

#49 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Rusty Russell
Quoted 5 lines · Rusty Russell

Distributions don't ship UP kernels any more; this shows what that costs if you're actually on a UP box. If we really don't care, perhaps we should make CONFIG_SMP=n an option under EMBEDDED for x86. And we can rip out the complex patching SMP patching stuff too.

The complex SMP patching is what makes it _possible_ to not ship UP kernels any more.

The SMP overhead exists, but it would be even higher if we didn't patch things to remove the "lock" prefix.

Linus

#47 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Dave McCracken
Quoted 4 lines · Dave McCracken

What I see as the message of his benchmark is if you care about performance you should be customizing your kernel anyway. Distro kernels are slow. An option that makes the distro kernel a bit slower is no big deal since anyone who wants speed should already be rebuilding their kernel.

And Oracle of course supports customers doing that?

Not in my experience, and the same goes for most other commercial enterprise software on Linux as well, so customers have to stick to distro kernels, if they want support.

Regards,
Anders K. Pedersen

#50 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Rusty Russell

Distributions don't ship UP kernels any more; this shows what that costs if you're actually on a UP box. If we really don't care, perhaps we should make CONFIG_SMP=n an option under EMBEDDED for x86. And we can rip out the complex patching SMP patching stuff too.

The complex SMP patching is what makes it _possible_ to not ship UP kernels any more.

"possible"? You mean "acceptable". Gray, not black and white.

1) Where's the line?
2) Where are we? Does patching claw back 5% of the loss? 50%? 90%?

No point benchmarking on my (SMP) laptop for this one. Gerd cc'd, maybe he has benchmarks from when he did the work originally?

Thanks,
Rusty.

#51 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 2 lines · Rusty Russell

"possible"? You mean "acceptable". Gray, not black and white.

I don't think we can possibly claim to support UP configurations if we don't patch.

Quoted 1 line

1) Where's the line?

"As good as we can make it". There is no line. There's "your code sucks so badly that it needs to get fixed, or we'll rip it out or disable it".

Quoted 1 line

2) Where are we? Does patching claw back 5% of the loss? 50%? 90%?

On some things, especially on P4, the lock overhead was tens of percent. Just a single locked instruction takes closer to two hundred instructions.

Of course, on those P4's, just kernel entry/exit is pretty high too (even with sysenter/exit), so I doubt you'll ever see something be 90% just because of that, unless it causes extra IO or other non-CPU issues.

Linus

#52 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 17 lines · Linus Torvalds
Quoted 6 lines · Rusty Russell
Quoted 3 lines

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

Thanks, your point is demonstrated below. I don't think HIGHMEM4G is unreasonable for a distro tho, so I turned that on instead.

Well, I agree that HIGHMEM4G is a _reasonable_ thing to turn on.

The thing I disagree with is that it's at all valid to then compare to some all-software feature thing. HIGHMEM doesn't expand any esoteric capability that some people might use - it's about regular RAM for regular users.

Quoted 4 lines

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

I was looking forward to be able to get rid of it... unfortunately other 32-bit architectures are starting to use it again :(

I guess it is not incredibly intrusive for generic mm code. A bit of kmap sprinkled around which is actually quite a useful delimiter of where pagecache is addressed via its kernel mapping.

Do you hate more the x86 code? Maybe that can be removed?

#53 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 23 lines · Nick Piggin
Quoted 17 lines · Linus Torvalds
Quoted 6 lines · Rusty Russell
Quoted 3 lines

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

Thanks, your point is demonstrated below. I don't think HIGHMEM4G is unreasonable for a distro tho, so I turned that on instead.

Well, I agree that HIGHMEM4G is a _reasonable_ thing to turn on.

The thing I disagree with is that it's at all valid to then compare to some all-software feature thing. HIGHMEM doesn't expand any esoteric capability that some people might use - it's about regular RAM for regular users.

Quoted 4 lines

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

Quoted 8 lines

I was looking forward to be able to get rid of it... unfortunately other 32-bit architectures are starting to use it again :(

I guess it is not incredibly intrusive for generic mm code. A bit of kmap sprinkled around which is actually quite a useful delimiter of where pagecache is addressed via its kernel mapping.

Do you hate more the x86 code? Maybe that can be removed?

IMHO what hurts most about highmem isnt even its direct source code overhead, but three factors:

- The buddy allocator allocates top down, with highmem pages first. So a lot of critical apps (the first ones started) will have highmem footprint, and that shows up every time they use it for file IO or other ops. kmap() overhead and more.

- Highmem is not really a 'solvable' problem in terms of good VM balancing. It gives conflicting constraints and there's no single 'good VM' that can really work - just a handful of bad solutions that differ in their level and area of suckiness.

- The kmap() cache itself can be depleted, and using atomic kmaps is fragile and error-prone. I think we still have a FIXME of a possibly triggerable deadlock somewhere in the core MM code ...

OTOH, highmem is clearly a useful hardware enablement feature with a slowly receding upside and a constant downside. The outcome is
clear: when a critical threshold is reached distros will stop enabling it. (or more likely, there will be pure 64-bit x86 distros)

Highmem simply enables a sucky piece of hardware so the code itself has an intrinsic level of suckage, so to speak. There's not much to be done about it but it's not a _big_ problem either: this type of hw is moving fast out of the distro attention span.

( What scares/worries me much more than sucky hardware is sucky
_software_ ABIs. Those have a half-life measured not in years but in decades and they get put into new products stubbornly, again and again. There's no Moore's Law getting rid of sucky software really and unlike the present set of sucky highmem hardware there's no influx of cosmic particles chipping away on their installed base either. )

Ingo

#54 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 35 lines · Ingo Molnar
Quoted 23 lines · Nick Piggin
Quoted 17 lines · Linus Torvalds
Quoted 6 lines · Rusty Russell
Quoted 3 lines

Turn off HIGHMEM64G, please (and HIGHMEM4G too, for that matter - you can't compare it to a no-highmem case).

Thanks, your point is demonstrated below. I don't think HIGHMEM4G is unreasonable for a distro tho, so I turned that on instead.

Well, I agree that HIGHMEM4G is a _reasonable_ thing to turn on.

The thing I disagree with is that it's at all valid to then compare to some all-software feature thing. HIGHMEM doesn't expand any esoteric capability that some people might use - it's about regular RAM for regular users.

Quoted 4 lines

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

Quoted 8 lines

I was looking forward to be able to get rid of it... unfortunately other 32-bit architectures are starting to use it again :(

I guess it is not incredibly intrusive for generic mm code. A bit of kmap sprinkled around which is actually quite a useful delimiter of where pagecache is addressed via its kernel mapping.

Do you hate more the x86 code? Maybe that can be removed?

Quoted 7 lines

IMHO what hurts most about highmem isnt even its direct source code overhead, but three factors:

- The buddy allocator allocates top down, with highmem pages first. So a lot of critical apps (the first ones started) will have highmem footprint, and that shows up every time they use it for file IO or other ops. kmap() overhead and more.

Yeah this really sucks about it. OTOH, we have basically the same thing today with NUMA allocations and task placement.

Quoted 4 lines

- Highmem is not really a 'solvable' problem in terms of good VM balancing. It gives conflicting constraints and there's no single 'good VM' that can really work - just a handful of bad solutions that differ in their level and area of suckiness.

But we have other zones too. And you also run into similar (and in some senses harder) choices with NUMA as well.

Quoted 1 line

- The kmap() cache itself can be depleted,

Yeah, the rule is not allowed to do 2 nested ones.

Quoted 3 lines

and using atomic kmaps
is fragile and error-prone. I think we still have a FIXME of a possibly triggerable deadlock somewhere in the core MM code ...

Not that I know of. I fixed the last long standing known one with the write_begin/write_end changes a year or two ago. It wasn't exactly related to kmap of the pagecache (but page fault of the user address in copy_from_user).

Quoted 4 lines

OTOH, highmem is clearly a useful hardware enablement feature with a slowly receding upside and a constant downside. The outcome is
clear: when a critical threshold is reached distros will stop enabling it. (or more likely, there will be pure 64-bit x86 distros)

Well now lots of embedded type archs are enabling it... So the upside is slowly increasing again I think.

Quoted 4 lines

Highmem simply enables a sucky piece of hardware so the code itself has an intrinsic level of suckage, so to speak. There's not much to be done about it but it's not a _big_ problem either: this type of hw is moving fast out of the distro attention span.

Yes but Linus really hated the code. I wonder whether it is generic code or x86 specific. OTOH with x86 you'd probably still have to support different page table formats, at least, so you couldn't rip it all out.

#55 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 9 lines · Nick Piggin
Quoted 3 lines

and using atomic kmaps
is fragile and error-prone. I think we still have a FIXME of a possibly triggerable deadlock somewhere in the core MM code ...

Not that I know of. I fixed the last long standing known one with the write_begin/write_end changes a year or two ago. It wasn't exactly related to kmap of the pagecache (but page fault of the user address in copy_from_user).

Quoted 8 lines
Quoted 5 lines

OTOH, highmem is clearly a useful hardware enablement feature with a slowly receding upside and a constant downside. The outcome is clear: when a critical threshold is reached distros will stop enabling it. (or more likely, there will be pure 64-bit x86 distros)

Well now lots of embedded type archs are enabling it... So the upside is slowly increasing again I think.

Sure - but the question is always how often does it show up on lkml? Less and less. There might be a lot of embedded Linux products sold, but their users are not reporting bugs to us and are not sending patches to us in the proportion of their apparent usage.

And on lkml there's a clear downtick in highmem relevance.

Quoted 10 lines
Quoted 5 lines

Highmem simply enables a sucky piece of hardware so the code itself has an intrinsic level of suckage, so to speak. There's not much to be done about it but it's not a _big_ problem
either: this type of hw is moving fast out of the distro attention span.

Yes but Linus really hated the code. I wonder whether it is generic code or x86 specific. OTOH with x86 you'd probably still have to support different page table formats, at least, so you couldn't rip it all out.

In practice the pte format hurts the VM more than just highmem. (the two are inseparably connected of course)

I did this fork overhead measurement some time ago, using perfcounters and 'perf':

Performance counter stats for './fork':

        32-bit  32-bit-PAE     64-bit
     ---------  ----------  ---------
     27.367537   30.660090  31.542003  task clock ticks     (msecs)

          5785        5810       5751  pagefaults           (events)
           389         388        388  context switches     (events)
             4           4          4  CPU migrations       (events)
     ---------  ----------  ---------
                    +12.0%     +15.2%  overhead

So PAE is 12.0% slower (the overhead of double the pte size and three page table levels), and 64-bit is 15.2% slower (the extra overhead of having four page table levels added to the overhead of double the pte size). [the pagefault count noise is well below the systematic performance difference.]

Fork is pretty much the worst-case measurement for larger pte overhead, as it has to copy around a lot of pagetables.

Larger ptes do not come for free and the 64-bit instructions do not mitigate the cachemiss overhead and memory bandwidth cost.

Ingo

#56 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 21 lines · Ingo Molnar
Quoted 9 lines · Nick Piggin
Quoted 3 lines

and using atomic kmaps
is fragile and error-prone. I think we still have a FIXME of a possibly triggerable deadlock somewhere in the core MM code ...

Not that I know of. I fixed the last long standing known one with the write_begin/write_end changes a year or two ago. It wasn't exactly related to kmap of the pagecache (but page fault of the user address in copy_from_user).

Quoted 8 lines
Quoted 5 lines

OTOH, highmem is clearly a useful hardware enablement feature with a slowly receding upside and a constant downside. The outcome is clear: when a critical threshold is reached distros will stop enabling it. (or more likely, there will be pure 64-bit x86 distros)

Well now lots of embedded type archs are enabling it... So the upside is slowly increasing again I think.

Quoted 6 lines

Sure - but the question is always how often does it show up on lkml? Less and less. There might be a lot of embedded Linux products sold, but their users are not reporting bugs to us and are not sending patches to us in the proportion of their apparent usage.

And on lkml there's a clear downtick in highmem relevance.

Definitely. Probably it works *reasonably* well enough in the end that embedded systems with reasonable highmem:lowmem ratio probably will work OK. Sadly for them in a year or two they probably get the full burden of carrying the crap ;)

Quoted 35 lines
Quoted 10 lines
Quoted 5 lines

Highmem simply enables a sucky piece of hardware so the code itself has an intrinsic level of suckage, so to speak. There's not much to be done about it but it's not a _big_ problem
either: this type of hw is moving fast out of the distro attention span.

Yes but Linus really hated the code. I wonder whether it is generic code or x86 specific. OTOH with x86 you'd probably still have to support different page table formats, at least, so you couldn't rip it all out.

In practice the pte format hurts the VM more than just highmem. (the two are inseparably connected of course)

I did this fork overhead measurement some time ago, using perfcounters and 'perf':

Performance counter stats for './fork':

        32-bit  32-bit-PAE     64-bit
     ---------  ----------  ---------
     27.367537   30.660090  31.542003  task clock ticks     (msecs)

          5785        5810       5751  pagefaults           (events)
           389         388        388  context switches     (events)
             4           4          4  CPU migrations       (events)
     ---------  ----------  ---------
                    +12.0%     +15.2%  overhead

So PAE is 12.0% slower (the overhead of double the pte size and three page table levels), and 64-bit is 15.2% slower (the extra overhead of having four page table levels added to the overhead of double the pte size). [the pagefault count noise is well below the systematic performance difference.]

Quoted 5 lines

Fork is pretty much the worst-case measurement for larger pte overhead, as it has to copy around a lot of pagetables.

Larger ptes do not come for free and the 64-bit instructions do not mitigate the cachemiss overhead and memory bandwidth cost.

No question about that... but you probably can't get rid of that because somebody will cry about NX bit, won't they?

#57 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

Fork is pretty much the worst-case measurement for larger pte overhead, as it has to copy around a lot of pagetables.

We could eliminate that if we use the R/W bit on pgd entries. fork() would be 256 clear_bit()s (1536 and 768 on i386 pae and nonpae).

copy_one_pte() disagrees though:

       if (unlikely(!pte_present(pte))) {
               if (!pte_file(pte)) {
                       swp_entry_t entry = pte_to_swp_entry(pte);

                       swap_duplicate(entry);
                       /* make sure dst_mm is on swapoff's mmlist. */
                       if (unlikely(list_empty(&dst_mm->mmlist))) {
                               spin_lock(&mmlist_lock);
                               if (list_empty(&dst_mm->mmlist))
                                       list_add(&dst_mm->mmlist,
                                                &src_mm->mmlist);
                               spin_unlock(&mmlist_lock);
                       }
                       if (is_write_migration_entry(entry) &&
                                       is_cow_mapping(vm_flags)) {
                               /*
                                * COW mappings require pages in both parent
                                * and child to be set to read.
                                */
                               make_migration_entry_read(&entry);
                               pte = swp_entry_to_pte(entry);
                               set_pte_at(src_mm, addr, src_pte, pte);
                       }
               }
               goto out_set_pte;
       }

Not sure how we can enlaze this thing.

Signature · 1 line
error compiling committee.c: too many arguments to function
#67 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Nick Piggin
Quoted 8 lines · Nick Piggin
Quoted 5 lines

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

I was looking forward to be able to get rid of it... unfortunately other 32-bit architectures are starting to use it again :(

.. and 32-bit x86 is still not dead, and there are still people who use it with more than 1G of RAM (ie it's not like it's just purely a "small embedded cell-phones with Atom" kind of thing that Intel seems to be pushing for eventually).

Quoted 5 lines

I guess it is not incredibly intrusive for generic mm code. A bit of kmap sprinkled around which is actually quite a useful delimiter of where pagecache is addressed via its kernel mapping.

Do you hate more the x86 code? Maybe that can be removed?

No, we can't remove the x86 code, and quite frankly, I don't even mind that. The part I mind is actually the sprinkling of kmap all over. Do a "git grep kmap fs", and you'll see that there are four times as many kmap's in filesystem code than there are in mm/.

I was benchmarking btrfs on my little EeePC. There, kmap overhead was 25% of file access time. Part of it is that people have been taught to use "kmap_atomic()", which is usable under spinlocks and people have been told that it's "fast". It's not fast. The whole TLB thing is slow as hell.

Oh well. It's sad. But we can't get rid of it.

Linus

#68 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 5 lines · Linus Torvalds

I was benchmarking btrfs on my little EeePC. There, kmap overhead was 25% of file access time. Part of it is that people have been taught to use "kmap_atomic()", which is usable under spinlocks and people have been told that it's "fast". It's not fast. The whole TLB thing is slow as hell.

yeah. I noticed it some time ago that INVLPG is unreasonably slow.

My theory is that in the CPU it's perhaps a loop (in microcode?) over _all_ TLBs - so as TLB caches get larger, INVLPG gets slower and slower ...

Ingo

#61 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Nick Piggin
Quoted 9 lines · Nick Piggin
Quoted 5 lines · Ingo Molnar

- The buddy allocator allocates top down, with highmem pages first. So a lot of critical apps (the first ones started) will have highmem footprint, and that shows up every time they use it for file IO or other ops. kmap() overhead and more.

Yeah this really sucks about it. OTOH, we have basically the same thing today with NUMA allocations and task placement.

It's not the buddy allocator. Each zone has it's own buddy list.

It's that we do the zones in order, and always start with the HIGHMEM zone.

Which is quite reasonablefor most loads (if the page is only used as a user mapping, we won't kmap it all that often), but it's bad for things where we will actually want to touch it over and over again. Notably filesystem caches that aren't just for user mappings.

Quoted 9 lines
Quoted 4 lines

Highmem simply enables a sucky piece of hardware so the code itself has an intrinsic level of suckage, so to speak. There's not much to be done about it but it's not a _big_ problem either: this type of hw is moving fast out of the distro attention span.

Yes but Linus really hated the code. I wonder whether it is generic code or x86 specific. OTOH with x86 you'd probably still have to support different page table formats, at least, so you couldn't rip it all out.

The arch-specific code really isn't that nasty. We have some silly workarouds for doing 8-byte-at-a-time operations on x86-32 with cmpxchg8b etc, but those are just odd small details.

If highmem was just a matter of arch details, I wouldn't mind it at all.

It's the generic code pollution I find annoying. It really does pollute a lot of crap. Not just fs/ and mm/, but even drivers.

Linus

#58 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 3 lines · Ingo Molnar

In practice the pte format hurts the VM more than just highmem. (the two are inseparably connected of course)

I think PAE is a separate issue (ie I think HIGHMEM4G and HIGHMEM64G are about different issues).

I do think we could probably drop PAE some day - very few 32-bit x86's have more than 4GB of memory, and the ones that did buy lots of memory back when it was a big deal for them have hopefully upgraded long since.

Of course, PAE also adds the NX flag etc, so there are probably other reasons to have it. And qutie frankly, PAE is just a small x86-specific detail that doesn't hurt anybody else.

So I have no reason to really dislike PAE per se - the real dislike is for HIGHMEM itself, and that gets enabled already for HIGHMEM4G without any PAE.

Of course, I'd also not ever enable it on any machine I have. PAE does add overhead, and the NX bit isn't _that_ important to me.

Linus

#70 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 21 lines · Linus Torvalds
Quoted 8 lines · Nick Piggin
Quoted 5 lines

And don't get me wrong - I don't like HIGHMEM. I detest the damn thing. I hated having to merge it, and I still hate it. It's a stupid, ugly, and very invasive config option. It's just that it's there to support a stupid, ugly and very annoying fundamental hardware problem.

I was looking forward to be able to get rid of it... unfortunately other 32-bit architectures are starting to use it again :(

.. and 32-bit x86 is still not dead, and there are still people who use it with more than 1G of RAM (ie it's not like it's just purely a "small embedded cell-phones with Atom" kind of thing that Intel seems to be pushing for eventually).

Quoted 4 lines

I guess it is not incredibly intrusive for generic mm code. A bit of kmap sprinkled around which is actually quite a useful delimiter of where pagecache is addressed via its kernel mapping.

Quoted 6 lines
Quoted 1 line

Do you hate more the x86 code? Maybe that can be removed?

No, we can't remove the x86 code, and quite frankly, I don't even mind that. The part I mind is actually the sprinkling of kmap all over. Do a "git grep kmap fs", and you'll see that there are four times as many kmap's in filesystem code than there are in mm/.

Yeah, I guess I just don't see it as such a bad thing. As I said, it's nice to have something to grep for and not have pointers into pagecache stored around the place (although filesystems do that with buffercache).

If code has to jump through particular nasty hoops to use atomic kmaps, that's not such a good thing...

Quoted 6 lines

I was benchmarking btrfs on my little EeePC. There, kmap overhead was 25% of file access time. Part of it is that people have been taught to use "kmap_atomic()", which is usable under spinlocks and people have been told that it's "fast". It's not fast. The whole TLB thing is slow as hell.

Oh well. It's sad. But we can't get rid of it.

If it's such a problem, it could be made a lot faster without too much problem. You could just introduce a FIFO of ptes behind it and flush them all in one go. 4K worth of ptes per CPU might hopefully bring your overhead down to < 1%.

#69 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 2 lines · Linus Torvalds

I was benchmarking btrfs on my little EeePC. There, kmap overhead was 25% of file access time. Part of it is that people have been taught to use "kmap_atomic()", which is usable under spinlocks and people have been told that it's "fast". It's not fast. The whole TLB thing is slow as hell.

yeah. I noticed it some time ago that INVLPG is unreasonably slow.

My theory is that in the CPU it's perhaps a loop (in microcode?) over _all_ TLBs - so as TLB caches get larger, INVLPG gets slower and slower ...

The tlb already content-addresses entries when looking up translations, so it shouldn't be that bad.

invlpg does have to invalidate all the intermediate entries ("paging-structure caches"), and it does (obviously) force a tlb reload.

I seem to recall 50 cycles for invlpg, what do you characterize as unreasonably slow?

Signature · 1 line
error compiling committee.c: too many arguments to function
#71 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Nick Piggin
Quoted 5 lines · Nick Piggin

If it's such a problem, it could be made a lot faster without too much problem. You could just introduce a FIFO of ptes behind it and flush them all in one go. 4K worth of ptes per CPU might hopefully bring your overhead down to < 1%.

We already have that. The regular kmap() does that. It's just not usable in atomic context.

We'd need to fix the locking: right now kmap_high() uses non-irq-safe locks, and it does that whole cross-cpu flushing thing (which is why those locks _have_ to be non-irq-safe.

The way to fix that, though, would be to never do any cross-cpu calls, and instead just have a cpumask saying "you need to flush before you do anything with kmap". So you'd just set that cpumask inside the lock, and if/when some other CPU does a kmap, they'd flush their local TLB at _that_ point instead of having to have an IPI call.

If we can get rid of kmap_atomic(), I'd already like HIGHMEM more. Right now I absolutely _hate_ all the different "levels" of kmap_atomic() and having to be careful about crazy nesting rules etc.

Linus

#72 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Nick Piggin

If it's such a problem, it could be made a lot faster without too much problem. You could just introduce a FIFO of ptes behind it and flush them all in one go. 4K worth of ptes per CPU might hopefully bring your overhead down to < 1%.

We already have that. The regular kmap() does that. It's just not usable in atomic context.

Well this would be more like the kmap cache idea rather than the kmap_atomic FIFO (which would remain per-cpu and look much like the existing kmap_atomic).

Quoted 9 lines

We'd need to fix the locking: right now kmap_high() uses non-irq-safe locks, and it does that whole cross-cpu flushing thing (which is why those locks _have_ to be non-irq-safe.

The way to fix that, though, would be to never do any cross-cpu calls, and instead just have a cpumask saying "you need to flush before you do anything with kmap". So you'd just set that cpumask inside the lock, and if/when some other CPU does a kmap, they'd flush their local TLB at _that_ point instead of having to have an IPI call.

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

#73 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 4 lines · Nick Piggin

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

No, everybody has to use kmap()/kunmap().

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

Linus

#74 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 9 lines · Linus Torvalds
Quoted 4 lines · Nick Piggin

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

No, everybody has to use kmap()/kunmap().

So it is strictly a bug to expose a pointer returned by kmap to another CPU? That would make it easier, although it would need to remove the global bit I think so when one task migrates CPUs then the entry will be flushed and reloaded properly.

Quoted 5 lines

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

This could be avoided with an anti-deadlock pool. If a task attempts a nested kmap and already holds a kmap, then give it exclusive access to this pool until it releases its last nested kmap.

#75 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 5 lines · Nick Piggin
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Nick Piggin
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

No, everybody has to use kmap()/kunmap().

So it is strictly a bug to expose a pointer returned by kmap to another CPU?

No, not at all. The pointers are all global. They have to be, since the original kmap() user may well be scheduled away.

Quoted 10 lines
Quoted 5 lines

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

This could be avoided with an anti-deadlock pool. If a task attempts a nested kmap and already holds a kmap, then give it exclusive access to this pool until it releases its last nested kmap.

We just sleep, waiting for somebody to release their. Again, that obviously won't work in atomic context, but it's easy enough to just have a "we need to have a few entries free" for the atomic case, and make it busy-loop if it runs out (which is not going to happen in practice anyway).

Linus

#62 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

OTOH, highmem is clearly a useful hardware enablement feature with a slowly receding upside and a constant downside. The outcome is
clear: when a critical threshold is reached distros will stop enabling it. (or more likely, there will be pure 64-bit x86 distros)

A major problem is that distros don't seem to be willing to push 64-bit kernels for 32-bit distros. There are a number of good (and not-so-good) reasons why users may want to run a 32-bit userspace, but not running a 64-bit kernel on capable hardware is just problematic.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#63 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 5 lines · H. Peter Anvin

A major problem is that distros don't seem to be willing to push 64-bit kernels for 32-bit distros. There are a number of good (and not-so-good) reasons why users may want to run a 32-bit userspace, but not running a 64-bit kernel on capable hardware is just problematic.

Yeah, that's just stupid. A 64-bit kernel should work well with 32-bit tools, and while we've occasionally had compat issues (the intel gfx people used to claim that they needed to work with a 32-bit kernel because they cared about 32-bit tools), they aren't unfixable or even all _that_ common.

And they'd be even less common if the whole "64-bit kernel even if you do a 32-bit distro" was more common.

The nice thing about a 64-bit kernel is that you should be able to build one even if you don't in general have all the 64-bit libraries. So you don't need a full 64-bit development environment, you just need a compiler that can generate code for both (and that should be the default on x86 these days).

Linus

#64 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 3 lines · Linus Torvalds

And they'd be even less common if the whole "64-bit kernel even if you do a 32-bit distro" was more common.

Side note: intel is to blame too. I think several Atom versions were shipped with 64-bit mode disabled. So even "modern" CPU's are sometimes artifically crippled to just 32-bit mode.

Linus

#65 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 3 lines · Linus Torvalds

Side note: intel is to blame too. I think several Atom versions were shipped with 64-bit mode disabled. So even "modern" CPU's are sometimes artifically crippled to just 32-bit mode.

And some people still want to run dosemu so they can drive their godforsaken 80s era PIO driven data analyzer. It'd be nice to think that nobody used vm86, but they always seem to pop out of the woodwork whenever someone suggests 64-bit kernels by default.

Signature · 1 line
Matthew Garrett | [email protected]
#66 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 11 lines · Matthew Garrett
Quoted 3 lines · Linus Torvalds

Side note: intel is to blame too. I think several Atom versions were shipped with 64-bit mode disabled. So even "modern" CPU's are sometimes artifically crippled to just 32-bit mode.

And some people still want to run dosemu so they can drive their godforsaken 80s era PIO driven data analyzer. It'd be nice to think that nobody used vm86, but they always seem to pop out of the woodwork whenever someone suggests 64-bit kernels by default.

There is both KVM and Qemu as alternatives, though. The godforsaken 80s-era PIO driven data analyzer will run fine in Qemu even on non-HVM-capable hardware if it's 64-bit capable. Most of the time it'll spend sitting in PIO no matter what you do.

-hpa

#59 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

I did this fork overhead measurement some time ago, using perfcounters and 'perf':

Could you post the program? I'd like to try it on some systems here.

Paul.

#60 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 7 lines · Paul Mackerras
Quoted 2 lines · Ingo Molnar

I did this fork overhead measurement some time ago, using perfcounters and 'perf':

Could you post the program? I'd like to try it on some systems here.

I still have it, it was something really, really simple and silly:

int main(void)

{
	int i;

	for (i = 0; i < 8; i++)
		if (!fork())
			wait(0);
}

Ingo

#76 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 13 lines · Linus Torvalds
Quoted 9 lines · Nick Piggin
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Nick Piggin

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

No, everybody has to use kmap()/kunmap().

So it is strictly a bug to expose a pointer returned by kmap to another CPU?

No, not at all. The pointers are all global. They have to be, since the original kmap() user may well be scheduled away.

Sorry, I meant another task.

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

This could be avoided with an anti-deadlock pool. If a task attempts a nested kmap and already holds a kmap, then give it exclusive access to this pool until it releases its last nested kmap.

Quoted 5 lines

We just sleep, waiting for somebody to release their. Again, that obviously won't work in atomic context, but it's easy enough to just have a "we need to have a few entries free" for the atomic case, and make it busy-loop if it runs out (which is not going to happen in practice anyway).

The really theoretical one (which Andrew likes complaining about) is when *everybody* is holding a kmap and asking for another one ;) But I think it isn't too hard to make a pool for that. And yes we'd also need a pool for atomic kmaps as you point out.

#79 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 4 lines · Nick Piggin

The idea seems nice but isn't the problem that kmap gives back a basically 1st class kernel virtual memory? (ie. it can then be used by any other CPU at any point without it having to use kmap?).

Quoted 7 lines

No, everybody has to use kmap()/kunmap().

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

One of the things I did for -rt back when I rewrote mm/highmem.c for it was to reserve multiple slots per kmap() user so that if you did 1 you could always do another.

With everything in task context like rt does 2 seemed enough, but you cuold ways extend that scheme and reserve enough for the worst case nesting depth and be done with it.

#77 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

In reply to Nick Piggin

Hi!

Quoted 17 lines
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

This could be avoided with an anti-deadlock pool. If a task attempts a nested kmap and already holds a kmap, then give it exclusive access to this pool until it releases its last nested kmap.

Quoted 5 lines

We just sleep, waiting for somebody to release their. Again, that obviously won't work in atomic context, but it's easy enough to just have a "we need to have a few entries free" for the atomic case, and make it busy-loop if it runs out (which is not going to happen in practice anyway).

Quoted 3 lines

The really theoretical one (which Andrew likes complaining about) is when *everybody* is holding a kmap and asking for another one ;) But I think it isn't too hard to make a pool for that. And yes we'd

Does one pool help?

Now you can have '*everyone* is holding the kmaps and is asking for another one'.

You could add as many pools as maximum nesting level... Is there maximum nesting level?

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#78 lore raw

Re: [benchmark] 1% performance overhead of paravirt_ops on native kernels

Quoted 19 lines · Pavel Machek

Hi!

Quoted 17 lines
Quoted 16 lines
Quoted 10 lines
Quoted 5 lines

The "problem" is that you could in theory run out of kmap frames, since if everybody does a kmap() in an interruptible context and you have lots and lots of threads doing different pages, you'd run out. But that has nothing to do with kmap_atomic(), which is basically limited to just the number of CPU's and a (very small) level of nesting.

This could be avoided with an anti-deadlock pool. If a task attempts a nested kmap and already holds a kmap, then give it exclusive access to this pool until it releases its last nested kmap.

We just sleep, waiting for somebody to release their. Again, that obviously won't work in atomic context, but it's easy enough to just have a "we need to have a few entries free" for the atomic case, and make it busy-loop if it runs out (which is not going to happen in practice anyway).

Quoted 5 lines
Quoted 3 lines

The really theoretical one (which Andrew likes complaining about) is when *everybody* is holding a kmap and asking for another one ;) But I think it isn't too hard to make a pool for that. And yes we'd

Does one pool help?

So long as only one process is allowed access to the pool at one time, yes I think it solves it. It would probably never even hit in practice, so synchronization overhead would not matter.

Quoted 5 lines

Now you can have '*everyone* is holding the kmaps and is asking for another one'.

You could add as many pools as maximum nesting level... Is there maximum nesting level?

Yes there are only a set number of kmap_atomic nesting levels, so if you converted them all to kmap then it would be that + 1.