x86_64: Double the per cpu area size

With the support for more than 256 irqs kernel_stat.irqs has grown noticeably in size. When asked about it no one wants to keep less information, so we need a larger per cpu area.

According to my rough calculations the per cpu usage has grown by about 15K (in the worst case) while the per cpu area size has grown by 32K so we should have enough room for a while.

---
 include/asm-x86_64/percpu.h |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

index 08dd9f9..56e3bdb 100644
@@ -11,6 +11,8 @@ #ifdef CONFIG_SMP
 
 #include <asm/pda.h>
 
+#define PERCPU_ENOUGH_ROOM 65536
+
 #define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
 #define __my_cpu_offset() read_pda(data_offset)
 

1.4.2.rc3.g7e18e

I would prefer if you didn't do that unconditionally. Can you make it dependent on NR_IRQS or so? Can you add a test for CONFIG_TINY to make it smaller?

Also longer term it should really properly fixed

-Andi

We already ignore this variable for the per cpu allocation if we build a kernel wihtout module support. I guess a good fix could entail changing the concept to be how much per cpu room to reserve for modules.

I'm a big believer in stupid and simple solutions for as long as you can get away with it. When people trip over this then it will be clear what the real problem is and we can fix it.

Agreed. But we need a solution that works now so we have a solution for when the 2.6.19 window opens up. There is no agreement on even what a proper fix is, or even what it looks like. Keeping the data per cpu seems about as good as anything else for memory size savings, as most systems don't have many cpus.

Throwing a few more bytes at the problem solves it today and for all systems currently built. This buys us time to look at and discuss the problem. With MSI starting to be useful I have no expectation that we will stop here.

There are two fundamental problems that need to be fixed.
- Small size and static allocate of the per cpu area.
- Data structures that don't scale to large numbers of possible irqs.

Solving either of these two fundamental problems involves reexamining some of our current trade offs in the kernel.

The proper fix for irqs is a refactoring of the data structures so we can handle a 16 or better yet a 24 bit irq number, and only allocate the pieces we need. A proper fix needs to find someway not to keep a counter for every cpu and irq pair, which no one has the will to seriously consider right now.

The proper fix for the per cpu area size is much trickier. Especially if we every reach the point of hotplug NUMA nodes. One odd observation is that the amount of per cpu data we want grows with the size of the system.

Eric

Is it possible to put this into -mm untill things are sorted? 2.6.18-rc3-mm2 x86_64 is still without any per-cpu space for modules.

Thanks,
Keith

I think we are sorted (see the later patch to auto size the per cpu area). But you should be ok with current -mm if you compile for 64 or fewer cpus.

Eric

Yep it looks like the config file I tote around on this box has CONFIG_NR_CPUS=128. I thought this this was the default NR_CPUS but it is not.

Thanks,
Keith