Memory Binding API v0.3 2.5.41

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding!

Quoted 3 lines · Matthew Dobson

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding!

Seems reasonable to me.

Could you tell us a bit about the operator's view of this?

I assume that a typical usage scenario would be to bind a process to a bunch of CPUs and to then bind that process to a bunch of memblks as well?

If so, then how does the operator know how to identify those memblks? To perform the (cpu list) <-> (memblk list) mapping?

Also, what advantage does this provide over the current node-local
allocation policy? I'd have thought that once you'd bound a process to a CPU (or to a node's CPUs) that as long as the zone fallback list was right, that process would be getting local memory
pretty much all the time anyway?

Last but not least: you got some benchmark numbers for this?

Thanks.

include/linux/mmzone.h line 271

Quoted 3 lines

include/linux/mmzone.h +3 -0

+#define for_each_valid_zone(zone, zonelist) 		\
+	for (zone = *zonelist->zones; zone; zone++)	\
+		if (current->memblk_binding.bitmask & (1 << zone->zone_pgdat->memblk_id))

Does the compiler optimise the last bit away on non-NUMA?

Want to wrap it in #ifdef CONFIG_NUMA_MEMBIND or something?

Not sure what the speed impact of this would be, but I'd rather it was optional, even on NUMA boxen.

Other than that, looks pretty good.

M.

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding! It works just like CPU Affinity (binding) except that it binds a processes memory allocations (just buddy allocator for now) to specific memory blocks.

Quoted 3 lines · Matthew Dobson

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding! It works just like CPU Affinity (binding) except that it binds a processes memory allocations (just buddy allocator for now) to specific memory blocks.

If the VM works right just doing CPU binding ought to be enough, surely?

kernel/sys.c line 1314

Quoted 10 lines

kernel/sys.c +9 -0

+/**
+ * sys_mem_setbinding - set the memory binding of a process
+ * @pid: pid of the process
+ * @memblks: new bitmask of memory blocks
+ * @behavior: new behavior
+ */
+asmlinkage long sys_mem_setbinding(pid_t pid, unsigned long memblks, 
+				    unsigned int behavior)
+{

Do you really think exposing low level internals as memory layout / zone split up to userspace is a good idea ? (and worth it given that the VM already has a cpu locality preference?)

Quoted 2 lines

Quoted 2 lines

kernel/sys.c +2 -0

+asmlinkage long sys_mem_setbinding(pid_t pid, unsigned long memblks, 
+				    unsigned int behavior)

Quoted 3 lines

Do you really think exposing low level internals as memory layout / zone split up to userspace is a good idea ? (and worth it given that the VM already has a cpu locality preference?)

At least in the embedded world that level is a good idea. I'm not sure about the syscall interface. An "unsigned long" mask of blocks sounds like a good way to ensure a broken syscall in the future

Quoted 3 lines · Matthew Dobson

Quoted 3 lines

Quoted 2 lines

kernel/sys.c +2 -0

+asmlinkage long sys_mem_setbinding(pid_t pid, unsigned long memblks, 
+				    unsigned int behavior)

Quoted 4 lines · Arjan van de Ven

Quoted 3 lines

Do you really think exposing low level internals as memory layout / zone split up to userspace is a good idea ? (and worth it given that the VM already has a cpu locality preference?)

Quoted 3 lines · Alan Cox

At least in the embedded world that level is a good idea. I'm not sure about the syscall interface. An "unsigned long" mask of blocks sounds like a good way to ensure a broken syscall in the future

Seconded wrt. memblk bitmask interface.

Also, I've already privately replied with some of my stylistic concerns, including things like the separability of the for_each_in_zonelist() cleanup bundled into the patch and a typedef or so.

Quoted 5 lines · Arjan van de Ven

Quoted 5 lines

+/**
+ * sys_mem_setbinding - set the memory binding of a process
+ * @pid: pid of the process
+ * @memblks: new bitmask of memory blocks
+ * @behavior: new behavior

Quoted 8 lines

kernel/sys.c line 1314

Quoted 4 lines

+ */

kernel/sys.c +3 -0

+asmlinkage long sys_mem_setbinding(pid_t pid, unsigned long memblks, 
+				    unsigned int behavior)
+{

Do you really think exposing low level internals as memory layout / zone split up to userspace is a good idea ? (and worth it given that the VM already has a cpu locality preference?)

At least in the embedded world that level is a good idea. I'm not sure about the syscall interface. An "unsigned long" mask of blocks sounds like a good way to ensure a broken syscall in the future

Quoted 5 lines

+/**
+ * sys_mem_setbinding - set the memory binding of a process
+ * @pid: pid of the process
+ * @memblks: new bitmask of memory blocks
+ * @behavior: new behavior

kernel/sys.c line 1314

Quoted 4 lines

+ */

kernel/sys.c +3 -0

+asmlinkage long sys_mem_setbinding(pid_t pid, unsigned long memblks, 
+				    unsigned int behavior)
+{

Do you really think exposing low level internals as memory layout / zone split up to userspace is a good idea ? (and worth it given that the VM already has a cpu locality preference?)

I'd much rather see the VM have an arch-specified "cost" for getting memory from not-the-prefered zones than exposing all this stuff to userspace and depending on userspace to do the right thing.... it's the kernel's task to abstract the low level details of the hardware after all.

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding! It works just like CPU Affinity (binding) except that it binds a processes memory allocations (just buddy allocator for now) to specific memory blocks.
I've sent this out in the past, but haven't touched it in months. Since

the feature freeze is rapidly approaching, I want to get this out there again and see if anyone has any interest in it.
It's a fairly large patch, mostly because it includes a few odds and
ends that are topology related, and don't strictly belong in this patch, but are

pre-requisites for it (ie: the [memblk|node]_online_map stuff, and some of the cleanups to page_alloc). I'll probably try and break it up into more discrete parts very soon.

Questions, comments, flames, and indifferent shrugs are all welcome.

btw, It applies (mostly) cleanly to mm1 as well. The mm/page_alloc.c changes fail, but if anyone is interested, they'll clean up easily, and I'll send you a patch.

Quoted 5 lines · Matthew Dobson

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding! It works just like CPU Affinity (binding) except that it binds a processes memory allocations (just buddy allocator for now) to specific memory blocks.

Due we want this per numa area or simply per zone? My suspicion is that internally at least we want this per zone.

The API doesn't make much sense at the moment.

1) You are operating on tasks and not mm's, or preferably vmas.

2) sys_mem_setbinding does not move the mm to the new binding.

3) You specify a pid and then change current task instead of the specified one.

4) An ordered zone list is probably the more natural mapping.

5) mprotect is the more natural model rather than set_cpu_affinity.

6) The code belongs in mm/* not kernel/*

Quoted 2 lines

4) An ordered zone list is probably the more natural mapping.

See my comments above about per zone/memblk. And you reemphasize my point, how do we order the zone lists in such a way that a user of the API can easily know/find out what zone #5 is?

Quoted 3 lines

Quoted 1 line

4) An ordered zone list is probably the more natural mapping.

See my comments above about per zone/memblk. And you reemphasize my point, how do we order the zone lists in such a way that a user of the API can easily know/find out what zone #5 is?

Could you explain how that problem is different from finding out what memblk #5 is ... I don't see the difference?

Quoted 5 lines

Quoted 3 lines

Quoted 1 line

4) An ordered zone list is probably the more natural mapping.

See my comments above about per zone/memblk. And you reemphasize my point, how do we order the zone lists in such a way that a user of the API can easily know/find out what zone #5 is?

Could you explain how that problem is different from finding out what memblk #5 is ... I don't see the difference?

Errm... __memblk_to_node(5)

I"m not saying that we couldn't add a similar interface for zones... something along the lines of:

	__memblk_and_zone_to_flat_zone_number(5, DMA)

or some such. It just isn't there now...

Quoted 8 lines

Quoted 6 lines

Quoted 3 lines

Quoted 1 line

4) An ordered zone list is probably the more natural mapping.

See my comments above about per zone/memblk. And you reemphasize my point, how do we order the zone lists in such a way that a user of the API can easily know/find out what zone #5 is?

Could you explain how that problem is different from finding out what memblk #5 is ... I don't see the difference?

Errm... __memblk_to_node(5)

As opposed to creating __zone_to_node(5) ?

Quoted 3 lines

I"m not saying that we couldn't add a similar interface for zones... something along the lines of:

	__memblk_and_zone_to_flat_zone_number(5, DMA)

or some such. It just isn't there now...

Surely this would dispose of the need for memblks? If not, then I'd agree it's probably just adding more complication.

Well, since each node's memory (or memblk in the parlance of my head ;) has several 'zones' in it (DMA, HIGHMEM, etc), this conversion function will need 2 parameters. It may well be called __node_and_zone_type_to_flat_zone_number(node, DMA|NORMAL|HIGHMEM). Or, we could have:
__zone_to_node(5) = node #
and
__zone_to_zone_type(5) = DMA|NORMAL|HIGHMEM.
But either way, we would need to specify both pieces. Cheers!
-Matt

Also, right now, memblks map to nodes in a straightforward manner (1-1 on NUMA-Q, the only architecture that has defined them). It will likely look the same on most architectures, too.

Quoted 3 lines

Also, right now, memblks map to nodes in a straightforward manner (1-1 on NUMA-Q, the only architecture that has defined them). It will likely look the same on most architectures, too.

Just an FYI: I believe the x440 breaks this assumption. There are 2 chunks on the first CEC. The current discontig patch for it has to drop the second chunk (anything over 3.5G on the first CEC) in order to work w/ the existing code. However, that will probably need to be addressed at some point, so be aware that this might affect you as well.

MAP_NR_DENSE()-based zone-relative pfn to zone->zone_mem_map index remapping is designed to handle this (and actually more severe situations). The only constraint is that pfn's must be monotonically increasing with ->zone_mem_map index. Some non-i386 architectures virtually remap physical memory to provide the illusion of contiguity of kernel virtual memory, but in a mature port (e.g. i386) there's high risk of breaking numerous preexisting drivers.

Quoted 3 lines

Also, right now, memblks map to nodes in a straightforward manner (1-1 on NUMA-Q, the only architecture that has defined them). It will likely look the same on most architectures, too.

Just an FYI: I believe the x440 breaks this assumption.

There are 2 chunks on the first CEC. The current discontig patch for it has to drop the second chunk (anything over 3.5G on the first CEC) in order to work w/ the existing code. However, that will probably need to be addressed at some point, so be aware that this might affect you as well.

Quoted 6 lines

Just an FYI: I believe the x440 breaks this assumption. There are 2 chunks on the first CEC. The current discontig patch for it has to drop the second chunk (anything over 3.5G on the first CEC) in order to work w/ the existing code. However, that will probably need to be addressed at some point, so be aware that this might affect you as well.

No, the NUMA code in the kernel doesn't support that anyway. You have to use zholes_size, and waste some struct pages, or config_nonlinear. Either way you get 1 memblk.

Quoted 3 lines

No, the NUMA code in the kernel doesn't support that anyway. You have to use zholes_size, and waste some struct pages, or config_nonlinear. Either way you get 1 memblk.

I thought zholes stuff freed the struct pages. Maybe that was done by hand.

Quoted 2 lines

I thought zholes stuff freed the struct pages. Maybe that was done by hand.

The only place I see that used in generic code is calculate_zone_totalpages, free_area_init_core, free_area_init_node, none of which seem to do that. But cscope might be borked again, I guess. Must be done in each arch if at all ... which arch did you think did it?

Quoted 5 lines

The only place I see that used in generic code is calculate_zone_totalpages, free_area_init_core, free_area_init_node, none of which seem to do that. But cscope might be borked again, I guess. Must be done in each arch if at all ... which arch did you think did it?

Not sure, ISTR something about this going on but I don't see any extant examples. At any rate, it should be easy to do it by hand, just make sure there are struct pages tracking the holes in mem_map and free the space in mem_map that would track the holes.

Quoted 9 lines · Eric W. Biederman

Quoted 6 lines · Matthew Dobson

Greetings & Salutations,
Here's a wonderful patch that I know you're all dying for... Memory
Binding! It works just like CPU Affinity (binding) except that it binds a processes memory allocations (just buddy allocator for now) to specific memory

blocks.

Due we want this per numa area or simply per zone? My suspicion is that internally at least we want this per zone.

I think that per memory block is better.

I'm not fanatically
opposed to per zone binding, though, and if there is a general agreement that it would be better that way, I don't think it would be unreasonably difficult to change it.

Quoted 1 line

The API doesn't make much sense at the moment.

Hmm.. That is unfortunate, I'd aimed to make it as simple as possible.

Quoted 1 line

1) You are operating on tasks and not mm's, or preferably vmas.

Correct. There are plans (somewhere inside my cranium) to allow binding at that

granularity. For now, per task seemed an appropriate level.

Quoted 1 line

2) sys_mem_setbinding does not move the mm to the new binding.

Also correct. A task may wish to allocate several large data structures from one memory area, rebind, do more allocations, rebind, ad nauseum. There are plans to have a flag that, if set, would force relocation of all currently allocated memory.

Quoted 1 line

5) mprotect is the more natural model rather than set_cpu_affinity.

Well, I think that may be true for the API you are imagining (per zone, per mm/vma, etc), not the one that I've written.