numa: mempolicy: dynamic interleave map for system init.

This is an alternative approach to the MPOL_INTERLEAVE across online nodes as the system init policy. Andi suggested it might be worthwhile trying to do this dynamically rather than as a command line option, so that's what this tries to do.

With this, the online nodes are sized and packed in to an interleave map if they're large enough for interleave to be worthwhile. I arbitrarily chose 16MB as the node size to enable interleaving, but perhaps someone has a better figure in mind?

In the case where all of the nodes are smaller than that, the largest node is selected and placed in to the map by itself (if they're all the same size, the first online node gets used).

If people prefer this approach, the previous patch adding mpolinit can be dropped.

Signed-off-by Paul Mundt <[email protected]>

Well I took silence as assent.

Quoted 1 line · Andrew Morton

Well I took silence as assent.

Well, grudgingly. How far are we willing to go to support these asymmetric setups? The NUMA code initially was designed for mostly symmetric systems with roughly the same amount of memory on each node. The farther we go from this the more options we will have to add special casing to deal with these imbalances.

With memoryless nodes we already have one issue that will ripple through the kernel likely requiring numerous modifications and special casing. Then we now have the ZONE_DMA issues reording the zonelists. Now we will support systems with 1MB size nodes? We will need to modify the slab allocators to only allocate on special processors?

Failing to support memoryless nodes was a bug, and we should continue to take bugfixes for that.

Quoted 1 line · Andrew Morton

Well I took silence as assent.

Well, grudgingly. How far are we willing to go to support these asymmetric setups? The NUMA code initially was designed for mostly symmetric systems with roughly the same amount of memory on each node. The farther we go from this the more options we will have to add special casing to deal with these imbalances.

With memoryless nodes we already have one issue that will ripple through the kernel likely requiring numerous modifications and special casing. Then we now have the ZONE_DMA issues reording the zonelists. Now we will support systems with 1MB size nodes? We will need to modify the slab allocators to only allocate on special processors?

obviously possible to try to work that in to SLOB or something similar, if making SLUB or SLAB lighterweight and more tunable for these cases ends up being a real barrier.

Quoted 3 lines · Paul Mundt

obviously possible to try to work that in to SLOB or something similar, if making SLUB or SLAB lighterweight and more tunable for these cases ends up being a real barrier.

Its obviously possible and as far as I can tell the architecture you have there requires it to operate. But the question is how much special casing we will have to add to the core VM.

We would likely have to add a

slub_nodes=

parameter that allows the specification of a nodelist that is allowed for the slab allocator. Then modify slub to use its own nodemap instead of the node online map. Modify get_partial_node to not try a node not in the nodemap and go to get_any_partial immediately. In addition to checking cpuset_zone_allowed we would need to check the slab node list.

Hmm.... That would also help to create isolated nodes that have no memory on them.

See what evil things you drive me to...

Could you try this patch (untested)? Set the allowed nodes on boot with

slub_nodes=0

if you have only node 0 for SLUB.

Node 1 SUnreclaim: 8 kB

So at least that gets back the couple of slab pages!

Quoted 3 lines · Paul Mundt

Node 1 SUnreclaim: 8 kB

So at least that gets back the couple of slab pages!

Hmmmm.. is that worth it? The patch is not right btw. There is still the case that new_slab can acquire a page on the wrong node and since we are not setup to allow that node in SLUB we will crash.

This now gets a bit ugly. In order to avoid that situation we check first if the node is allowed. If not then we simply ask for an alloc on the first node.

But that may still make the page allocator fall back. If that happens then we redo the allocation with GFP_THISNODE to force an allocation on the first node or fail.

I think we could do better by constructing a custom zonelist but that will be even more special casing.

Quoted 3 lines

I think we could do better by constructing a custom zonelist but that will be even more special casing.

I don't know if a custom zonelist is worth the trouble. For the common asymmetric case, you could at least infer that ZONE_NORMAL is the only thing populated per node (well, small nodes other than node 0). If you mean just creating the zonelist from the range of allowable SLUB nodes, that could work.

Quoted 8 lines · Paul Mundt

Quoted 3 lines

I think we could do better by constructing a custom zonelist but that will be even more special casing.

I don't know if a custom zonelist is worth the trouble. For the common asymmetric case, you could at least infer that ZONE_NORMAL is the only thing populated per node (well, small nodes other than node 0). If you mean just creating the zonelist from the range of allowable SLUB nodes, that could work.

Well that is quit difficult because of the other constraints on the alloc. The allocation must consider the cpuset context and the memory policies of the task (which may need special casing already there for interleave). Maybe we can determine from those restrictions a zonelist. Then we need to kick out the zones belonging to the illegal nodes from that zonelist. Then pass that to __alloc_pages to perform the alloc.

Looks like we are heading for a new alloc function

alloc_pages_node_not_nodes(order, gfpmask, node, forbidden-nodes)

But may be the hack of just going to node 0 on a problem is enough???

Incidentally, the interleave map created for mempol sysinit is something that could also be picked up by SLUB for the allowable node map (at least as a starting point, exlucding cpuset constraints).

Quoted 10 lines · Christoph Lameter

Quoted 1 line · Andrew Morton

Well I took silence as assent.

Well, grudgingly. How far are we willing to go to support these asymmetric setups? The NUMA code initially was designed for mostly symmetric systems with roughly the same amount of memory on each node. The farther we go from this the more options we will have to add special casing to deal with these imbalances.

Well, this doesn't all have to be dynamic either. I opted for the mpolinit= approach first so we wouldn't make the accounting for the common case heavier, but certainly having it dynamic is less hassle. The asymmetric case will likely be the common case for embedded, but it's obviously possible to try to work that in to SLOB or something similar, if making SLUB or SLAB lighterweight and more tunable for these cases ends up being a real barrier.

On the other hand, as we start having machines with multiple gigs of RAM that are stashed in node 0 (with many smaller memories in other nodes), SLOB isn't going to be a long-term option either.

Quoted 8 lines · Paul Mundt

Well, this doesn't all have to be dynamic either. I opted for the mpolinit= approach first so we wouldn't make the accounting for the common case heavier, but certainly having it dynamic is less hassle. The asymmetric case will likely be the common case for embedded, but it's obviously possible to try to work that in to SLOB or something similar, if making SLUB or SLAB lighterweight and more tunable for these cases ends up being a real barrier.

Quoted 3 lines

On the other hand, as we start having machines with multiple gigs of RAM that are stashed in node 0 (with many smaller memories in other nodes), SLOB isn't going to be a long-term option either.

SLOB in -mm should scale to this size reasonably well now, and Nick and I have another tweak planned that should make it quite fast here.

SLOB's big scalability problem at this point is number of CPUs. Throwing some fine-grained locking at it or the like may be able to help with that too.

Why would you even want to bother making it scale that large? For starters, it's less affected by things like dcache fragmentation. The majority of pages pinned by long-lived dcache entries will still be available to other allocations.

Haven't given any thought to NUMA yet though..

Quoted 11 lines · Matt Mackall

SLOB's big scalability problem at this point is number of CPUs. Throwing some fine-grained locking at it or the like may be able to help with that too.

Why would you even want to bother making it scale that large? For starters, it's less affected by things like dcache fragmentation. The majority of pages pinned by long-lived dcache entries will still be available to other allocations.

Haven't given any thought to NUMA yet though..

This is what I've hacked together and tested with my small nodes. It's not terribly intelligent, and it pushes off most of the logic to the page allocator. Obviously it's not terribly scalable, and I haven't tested it with page migration, either. Still, it works for me with my simple tmpfs
+ mpol policy tests.

Tested on a UP + SPARSEMEM (static, not extreme) + NUMA (2 nodes) + SLOB configuration.

Flame away!

static void slob_timer_cbk(void)

Quoted 20 lines · Paul Mundt

Quoted 11 lines · Matt Mackall

SLOB's big scalability problem at this point is number of CPUs. Throwing some fine-grained locking at it or the like may be able to help with that too.

Why would you even want to bother making it scale that large? For starters, it's less affected by things like dcache fragmentation. The majority of pages pinned by long-lived dcache entries will still be available to other allocations.

Haven't given any thought to NUMA yet though..

This is what I've hacked together and tested with my small nodes. It's not terribly intelligent, and it pushes off most of the logic to the page allocator. Obviously it's not terribly scalable, and I haven't tested it with page migration, either. Still, it works for me with my simple tmpfs
+ mpol policy tests.

Quoted 4 lines

Tested on a UP + SPARSEMEM (static, not extreme) + NUMA (2 nodes) + SLOB configuration.

Flame away!

For starters, it's not against the current SLOB, which no longer has the bigblock list.

mm/slob.c static void slob_free(void *block, int size) · line 171

Quoted 2 lines

mm/slob.c +1 -1

-void *__kmalloc(size_t size, gfp_t gfp)
+static void *__kmalloc_alloc(size_t size, gfp_t gfp, int node)

That's a ridiculous name. So, uh.. more underbars!

Though really, I think you can just name it __kmalloc_node?

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 359

Quoted 5 lines

mm/slob.c +5 -0

+		if (node == -1)
+			pages = alloc_pages(flags, get_order(c->size));
+		else
+			pages = alloc_pages_node(node, flags,
+						get_order(c->size));

This fragment appears a few times. Looks like it ought to get its own function. And that function can reduce to a trivial inline in the !NUMA case.

Quoted 41 lines · Matt Mackall

Quoted 20 lines · Paul Mundt

Quoted 11 lines · Matt Mackall

SLOB's big scalability problem at this point is number of CPUs. Throwing some fine-grained locking at it or the like may be able to help with that too.

Why would you even want to bother making it scale that large? For starters, it's less affected by things like dcache fragmentation. The majority of pages pinned by long-lived dcache entries will still be available to other allocations.

Haven't given any thought to NUMA yet though..

This is what I've hacked together and tested with my small nodes. It's not terribly intelligent, and it pushes off most of the logic to the page allocator. Obviously it's not terribly scalable, and I haven't tested it with page migration, either. Still, it works for me with my simple tmpfs
+ mpol policy tests.

Quoted 4 lines

Tested on a UP + SPARSEMEM (static, not extreme) + NUMA (2 nodes) + SLOB configuration.

Flame away!

For starters, it's not against the current SLOB, which no longer has the bigblock list.

mm/slob.c static void slob_free(void *block, int size) · line 171

Quoted 2 lines

mm/slob.c +1 -1

-void *__kmalloc(size_t size, gfp_t gfp)
+static void *__kmalloc_alloc(size_t size, gfp_t gfp, int node)

That's a ridiculous name. So, uh.. more underbars!

Though really, I think you can just name it __kmalloc_node?

Quoted 10 lines

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 359

Quoted 5 lines

mm/slob.c +5 -0

+		if (node == -1)
+			pages = alloc_pages(flags, get_order(c->size));
+		else
+			pages = alloc_pages_node(node, flags,
+						get_order(c->size));

This fragment appears a few times. Looks like it ought to get its own function. And that function can reduce to a trivial inline in the !NUMA case.

BTW. what I would like to see tried initially -- which may give reasonable scalability and NUMAness -- is perhaps a percpu or per-node free pages lists. However these lists would not be exclusively per-cpu, because that would result in worse memory consumption (we should always try to put memory consumption above all else with SLOB).

So each list would have its own lock and can be accessed by any CPU, but they would default to their own list first (or in the case of a kmalloc_node, they could default to some other list).

Then we'd probably like to introduce a *little* bit of slack, so that we will allocate a new page on our local list even if there is a small amount of memory free on another list. I think this might be enough to get a reasonable number of list-local allocations without blowing out the memory usage much. The slack ratio could be configurable so at one extreme we could always allocate from our local lists for best NUMA placement I guess.

I haven't given it a great deal of thought, so this strategy might go horribly wrong in some cases... but I have a feeling something reasonably simple like that might go a long way to improving locking scalability and NUMAness.

Quoted 9 lines · Paul Mundt

Quoted 2 lines · Matt Mackall

Haven't given any thought to NUMA yet though..

This is what I've hacked together and tested with my small nodes. It's not terribly intelligent, and it pushes off most of the logic to the page allocator. Obviously it's not terribly scalable, and I haven't tested it with page migration, either. Still, it works for me with my simple tmpfs
+ mpol policy tests.

Quoted 4 lines

Tested on a UP + SPARSEMEM (static, not extreme) + NUMA (2 nodes) + SLOB configuration.

Flame away!

For starters, it's not against the current SLOB, which no longer has the bigblock list.

mm/slob.c static void slob_free(void *block, int size) · line 171

Quoted 2 lines

mm/slob.c +1 -1

-void *__kmalloc(size_t size, gfp_t gfp)
+static void *__kmalloc_alloc(size_t size, gfp_t gfp, int node)

That's a ridiculous name. So, uh.. more underbars!

Though really, I think you can just name it __kmalloc_node?

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 359

Quoted 5 lines

mm/slob.c +5 -0

+		if (node == -1)
+			pages = alloc_pages(flags, get_order(c->size));
+		else
+			pages = alloc_pages_node(node, flags,
+						get_order(c->size));

This fragment appears a few times. Looks like it ought to get its own function. And that function can reduce to a trivial inline in the !NUMA case.

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 380

Quoted 4 lines

mm/slob.c +4 -0

+void *kmem_cache_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
+{
+	return __kmem_cache_alloc(c, flags, node);
+}

If we make the underlying functions all take a node, this stuff all gets simpler.

Quoted 10 lines · Matt Mackall

Quoted 9 lines · Paul Mundt

Quoted 2 lines · Matt Mackall

Haven't given any thought to NUMA yet though..

This is what I've hacked together and tested with my small nodes. It's not terribly intelligent, and it pushes off most of the logic to the page allocator. Obviously it's not terribly scalable, and I haven't tested it with page migration, either. Still, it works for me with my simple tmpfs
+ mpol policy tests.

Quoted 8 lines

Quoted 4 lines

Tested on a UP + SPARSEMEM (static, not extreme) + NUMA (2 nodes) + SLOB configuration.

Flame away!

For starters, it's not against the current SLOB, which no longer has the bigblock list.

Sorry about that, seems I used the wrong tree.

Quoted 5 lines

mm/slob.c static void slob_free(void *block, int size) · line 171

Quoted 2 lines

mm/slob.c +1 -1

-void *__kmalloc(size_t size, gfp_t gfp)
+static void *__kmalloc_alloc(size_t size, gfp_t gfp, int node)

That's a ridiculous name. So, uh.. more underbars!

Agreed, though I couldn't think of a better one.

Quoted 2 lines

Though really, I think you can just name it __kmalloc_node?

No, kmalloc_node and __kmalloc_node are both required by CONFIG_NUMA, otherwise that would have been the logical choice.

Quoted 10 lines

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 359

Quoted 5 lines

mm/slob.c +5 -0

+		if (node == -1)
+			pages = alloc_pages(flags, get_order(c->size));
+		else
+			pages = alloc_pages_node(node, flags,
+						get_order(c->size));

This fragment appears a few times. Looks like it ought to get its own function. And that function can reduce to a trivial inline in the !NUMA case.

Ok.

Quoted 8 lines

mm/slob.c void kmem_cache_destroy(struct kmem_cache *c) · line 380

Quoted 4 lines

mm/slob.c +4 -0

+void *kmem_cache_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
+{
+	return __kmem_cache_alloc(c, flags, node);
+}

If we make the underlying functions all take a node, this stuff all gets simpler.

Could you elaborate on that?