Skip to contentSkip to conversation

More problems in setup_pcpu_remap()

DISCUSSION David Miller 2 participants 6 messages Apr 2, 2009

lore
Share & tools
lore
#0 lore raw

The way this code is currently designed, it can exhaust all of the VMALLOC address space on both x86 and x86_64, and then some.

It allocates PMD_SIZE * num_possible_cpus() of vmalloc space.

PMD_SIZE is 2MB, num_possible_cpus() can be up to 4096.... which can easily exceed (VMALLOC_END - VMALLOC_START)

Initially I had set out to implement sparc64 support for the new per-cpu stuff, but it looks like I'm stuck finding bugs in the x86 implementation instead :-)

#1 lore raw

Hello,

Quoted 11 lines · David Miller

The way this code is currently designed, it can exhaust all of the VMALLOC address space on both x86 and x86_64, and then some.

It allocates PMD_SIZE * num_possible_cpus() of vmalloc space.

PMD_SIZE is 2MB, num_possible_cpus() can be up to 4096.... which can easily exceed (VMALLOC_END - VMALLOC_START)

Initially I had set out to implement sparc64 support for the new per-cpu stuff, but it looks like I'm stuck finding bugs in the x86 implementation instead :-)

Eh... sorry about that. :-)

I guess we'll have to put a cap on how high possible cpus can be for remap allocator. e.g. if single chunk size is over 20% of the whole vmalloc area, don't use remap. Does anyone have a good random % number on mind?

Thanks.

Signature · 1 line
tejun
#2 lore raw

From: Tejun Heo <[email protected]>
Date: Thu, 02 Apr 2009 13:42:57 +0900

Quoted 4 lines

I guess we'll have to put a cap on how high possible cpus can be for remap allocator. e.g. if single chunk size is over 20% of the whole vmalloc area, don't use remap. Does anyone have a good random % number on mind?

I would suggest instead to rethink what this code is doing.

It would make more sense to carve up 2MB chunks into some-power-of-2 pieces and use that as the unit size.

You could retain the NUMA goals of this function, as well as the ability to be using 2MB pages in the TLBs.

And consider that if the dynamic allocation part of this code triggers even once, you'll end up eating twice as much VMALLOC space.

Using 2MB per cpu is just rediculious, and really not even necessary.

#3 lore raw

Hello, David.

Quoted 6 lines · David Miller
Quoted 4 lines

I guess we'll have to put a cap on how high possible cpus can be for remap allocator. e.g. if single chunk size is over 20% of the whole vmalloc area, don't use remap. Does anyone have a good random % number on mind?

I would suggest instead to rethink what this code is doing.

Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 2^45, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

If it's a problem for other archs or extreme x86_32 configurations, we

can add some safety measures but in general I don't think it is a problem.

Quoted 5 lines

It would make more sense to carve up 2MB chunks into some-power-of-2 pieces and use that as the unit size.

You could retain the NUMA goals of this function, as well as the ability to be using 2MB pages in the TLBs.

Can you please elaborate a bit?

Quoted 4 lines

And consider that if the dynamic allocation part of this code triggers even once, you'll end up eating twice as much VMALLOC space.

Using 2MB per cpu is just rediculious, and really not even necessary.

The focus at the moment is using large page for the first chunk to reduce pressure on TLB, not necessarily actually using 2MB for each unit.

Thanks.

Signature · 1 line
tejun
#4 lore raw

From: Tejun Heo <[email protected]>
Date: Thu, 02 Apr 2009 14:55:48 +0900

Quoted 9 lines
Quoted 5 lines · David Miller

It would make more sense to carve up 2MB chunks into some-power-of-2 pieces and use that as the unit size.

You could retain the NUMA goals of this function, as well as the ability to be using 2MB pages in the TLBs.

Can you please elaborate a bit?

Quoted 8 lines
Quoted 4 lines

And consider that if the dynamic allocation part of this code triggers even once, you'll end up eating twice as much VMALLOC space.

Using 2MB per cpu is just rediculious, and really not even necessary.

The focus at the moment is using large page for the first chunk to reduce pressure on TLB, not necessarily actually using 2MB for each unit.

You'll get better TLB hit rates with my suggestion.

The idea is to carve up a 2MB page amongst consequetive cpus in the same NUMA node.

With hyperthreading these cpus will be sharing the TLB, so you'll get better TLB hit rates than the current code.

I'm going to do something like this on sparc64. This x86 code is severely demotivating me, so I'll stop looking at it now :-)

#5 lore raw

Hello, David.

Quoted 5 lines · David Miller
Quoted 4 lines
Quoted 1 line

Using 2MB per cpu is just rediculious, and really not even necessary.

The focus at the moment is using large page for the first chunk to reduce pressure on TLB, not necessarily actually using 2MB for each unit.

Quoted 7 lines

You'll get better TLB hit rates with my suggestion.

The idea is to carve up a 2MB page amongst consequetive cpus in the same NUMA node.

With hyperthreading these cpus will be sharing the TLB, so you'll get better TLB hit rates than the current code.

That sounds great.

Quoted 2 lines

I'm going to do something like this on sparc64. This x86 code is severely demotivating me, so I'll stop looking at it now :-)

Yeah, please go ahead and write severely motivating code. I'll be very motivated to apply it to x86. :-)

Thanks.

Signature · 1 line
tejun