Skip to contentSkip to conversation

larger per cpu data

DISCUSSION Andi Kleen 4 participants 10 messages Apr 20, 2007 – Apr 24, 2007 Generic include/asm header files

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

Hi,

I've been looking at optimizing the network per CPU statistics and the best way to do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too?
Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?

-Andi

#1 lore raw
Quoted 1 line

I heard there were some problems with ia64 have those been fixed too?

IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the arch/ia64/configs/* variations.

Quoted 2 lines

Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?

How many is "a few" ... and much more importantly is your few KB a fixed amount, or does it scale up with the number of network cards (or anything else that there might be a lot of on a large system)?

A fixed extra 8-10K should be fine. #NIC * 1K would cause problems.

-Tony

author
#2 lore raw
Quoted 4 lines · Tony
Quoted 1 line

I heard there were some problems with ia64 have those been fixed too?

IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the arch/ia64/configs/* variations.

Could that be enlarged without too much trouble?

Quoted 8 lines
Quoted 2 lines

Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?

How many is "a few" ... and much more importantly is your few KB a fixed amount, or does it scale up with the number of network cards (or anything else that there might be a lot of on a large system)?

A fixed extra 8-10K should be fine. #NIC * 1K would cause problems.

There are scalable parts, but those would likely stay allocated. ==

-Andi

#3 lore raw
Quoted 4 lines
Quoted 2 lines

IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the arch/ia64/configs/* variations.

Could that be enlarged without too much trouble?

If we stick to mapping it with a single TLB entry, then the next size up is 256K ... which is a pretty big jump. But the code that does the mapping is just about to be changed from a single locked TLB entry to a map-on-tlb-miss version (thanks to Ken Chen). Some small-ish additions to that code would allow an expansion to 128K by mapping each half with a 64K TLB entry on demand.

-Tony

#7 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen

Hi,

I've been looking at optimizing the network per CPU statistics and the best way to
do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too? Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?

Andi which areas are you looking at modifying. I'm trying to get a feel to see if your changes will have an effect on the network namespace work.

Eric

author
#8 lore raw
Quoted 13 lines · On Saturday 21 April 2007 22:23:21 Eric W. Biederman wrote:
Quoted 10 lines · Andi Kleen

Hi,

I've been looking at optimizing the network per CPU statistics and the best way to
do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too? Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?

Quoted 2 lines

Andi which areas are you looking at modifying. I'm trying to get a feel to see if your changes will have an effect on the network namespace work.

Just the internal implementation of the statistics. Not very many changes outside those include files

-Andi

#9 lore raw
Quoted 2 lines · Andi Kleen

Just the internal implementation of the statistics. Not very many changes outside those include files

Ok. I guess I'm dense at the moment. Which include files?

The only statistics I can think of at the moment are per network device, and that doesn't sound like something that should be made per cpu.

Eric

#4 lore raw
In reply to Luck, Tony
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

"Tony" == Luck, Tony <[email protected]> writes:

IA64 has a 64KB page allocated for each cpu for the percpu

data. Right > now we are using 28K-32K for the arch/ia64/configs/* variations.

Could that be enlarged without too much trouble?

Tony> If we stick to mapping it with a single TLB entry, then the next Tony> size up is 256K ... which is a pretty big jump. But the code Tony> that does the mapping is just about to be changed from a single Tony> locked TLB entry to a map-on-tlb-miss version (thanks to Ken Tony> Chen). Some small-ish additions to that code would allow an Tony> expansion to 128K by mapping each half with a 64K TLB entry on Tony> demand.

256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.

Cheers,
Jes

#5 lore raw
Quoted 2 lines

256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.

"Only" 256M :-) That 1024 cpu system probably has 1TB of memory or more ... so this isn't quite as terrible as it sounds (at least in terms of percentage of memory used).

Are there customers buying high cpu count, low memory systems?

But unless these are some fabulously useful counters that Andi wants to add ... I'd prefer to keep the ia64 percpu area at 64K for a while longer.

-Tony

#6 lore raw
Quoted 8 lines · Tony
Quoted 2 lines

256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.

"Only" 256M :-) That 1024 cpu system probably has 1TB of memory or more ... so this isn't quite as terrible as it sounds (at least in terms of percentage of memory used).

Are there customers buying high cpu count, low memory systems?

I doubt it, but I don't have numbers. However 256M is still a good chunk of memory that could be put to better use.

Quoted 3 lines

But unless these are some fabulously useful counters that Andi wants to add ... I'd prefer to keep the ia64 percpu area at 64K for a while longer.

Nod, my thoughts exactly.

Cheers,
Jes