larger per cpu data
Hi,
I've been looking at optimizing the network per CPU statistics and the best way to do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too?
Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?
-Andi
Quoted 1 line
I heard there were some problems with ia64 have those been fixed too?
IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the arch/ia64/configs/* variations.
Quoted 2 lines
Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?
How many is "a few" ... and much more importantly is your few KB a fixed amount, or does it scale up with the number of network cards (or anything else that there might be a lot of on a large system)?
A fixed extra 8-10K should be fine. #NIC * 1K would cause problems.
-Tony
Quoted 4 lines · Tony
Quoted 1 line
I heard there were some problems with ia64 have those been fixed too?
IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the
arch/ia64/configs/*variations.
Could that be enlarged without too much trouble?
Quoted 8 lines
Quoted 2 lines
Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?
How many is "a few" ... and much more importantly is your few KB a fixed amount, or does it scale up with the number of network cards (or anything else that there might be a lot of on a large system)?
A fixed extra 8-10K should be fine. #NIC * 1K would cause problems.
There are scalable parts, but those would likely stay allocated. ==
-Andi
Quoted 4 lines
Quoted 2 lines
IA64 has a 64KB page allocated for each cpu for the percpu data. Right now we are using 28K-32K for the
arch/ia64/configs/*variations.Could that be enlarged without too much trouble?
If we stick to mapping it with a single TLB entry, then the next size up is 256K ... which is a pretty big jump. But the code that does the mapping is just about to be changed from a single locked TLB entry to a map-on-tlb-miss version (thanks to Ken Chen). Some small-ish additions to that code would allow an expansion to 128K by mapping each half with a 64K TLB entry on demand.
-Tony
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
"Tony" == Luck, Tony <[email protected]> writes:
IA64 has a 64KB page allocated for each cpu for the percpu
data. Right > now we are using 28K-32K for the
arch/ia64/configs/*variations.Could that be enlarged without too much trouble?
Tony> If we stick to mapping it with a single TLB entry, then the next Tony> size up is 256K ... which is a pretty big jump. But the code Tony> that does the mapping is just about to be changed from a single Tony> locked TLB entry to a map-on-tlb-miss version (thanks to Ken Tony> Chen). Some small-ish additions to that code would allow an Tony> expansion to 128K by mapping each half with a 64K TLB entry on Tony> demand.
256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.
Cheers,
Jes
Quoted 2 lines
256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.
"Only" 256M :-) That 1024 cpu system probably has 1TB of memory or more ... so this isn't quite as terrible as it sounds (at least in terms of percentage of memory used).
Are there customers buying high cpu count, low memory systems?
But unless these are some fabulously useful counters that Andi wants to add ... I'd prefer to keep the ia64 percpu area at 64K for a while longer.
-Tony
Quoted 8 lines · Tony
Quoted 2 lines
256KB * 1024 CPUs (or more) == verymightyhuge ..... I certainly wouldn't like to see this.
"Only" 256M :-) That 1024 cpu system probably has 1TB of memory or more ... so this isn't quite as terrible as it sounds (at least in terms of percentage of memory used).
Are there customers buying high cpu count, low memory systems?
I doubt it, but I don't have numbers. However 256M is still a good chunk of memory that could be put to better use.
Quoted 3 lines
But unless these are some fabulously useful counters that Andi wants to add ... I'd prefer to keep the ia64 percpu area at 64K for a while longer.
Nod, my thoughts exactly.
Cheers,
Jes
Quoted 10 lines · Andi Kleen
Hi,
I've been looking at optimizing the network per CPU statistics and the best way to
do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too? Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?
Andi which areas are you looking at modifying. I'm trying to get a feel to see if your changes will have an effect on the network namespace work.
Eric
Quoted 13 lines · On Saturday 21 April 2007 22:23:21 Eric W. Biederman wrote:
Quoted 10 lines · Andi Kleen
Hi,
I've been looking at optimizing the network per CPU statistics and the best way to
do that would be to move them back to direct per CPU data. Currently they are allocated separately because there used to be some trouble on x86 (and ia64?) with limited per CPU areas. At least the x86 side has been fixed now. I heard there were some problems with ia64 have those been fixed too? Anybody would have a problem with increasing per CPU data in a standard kernel by a few KB?
Quoted 2 lines
Andi which areas are you looking at modifying. I'm trying to get a feel to see if your changes will have an effect on the network namespace work.
Just the internal implementation of the statistics. Not very many changes outside those include files
-Andi
Quoted 2 lines · Andi Kleen
Just the internal implementation of the statistics. Not very many changes outside those include files
Ok. I guess I'm dense at the moment. Which include files?
The only statistics I can think of at the moment are per network device, and that doesn't sound like something that should be made per cpu.
Eric