: Containers(V2)- Introduction

Containers:

This is based on lot of discussions over last month or so. I hope this patch set is something that we can agree and more support can be added on top of this. Please provide feedback and add other extensions that are useful in the TODO list.

I'm not sure about containers & workload management people, but from a core mm/ perspective I see no reason why this couldn't get in, given review and testing. Great!

Quoted 3 lines · Nick Piggin

I'm not sure about containers & workload management people, but from a core mm/ perspective I see no reason why this couldn't get in, given review and testing. Great!

Nack. We already have the ability to manage workloads. We may want to extend the existing functionality but this is duplicating what is already available through cpusets.

If it wasn't clear was talking specifically about the hooks for page tracking rather than the whole patchset. If anybody wants such page tracking infrastructure in the kernel, then this (as opposed to the huge beancounters stuff) is what it should look like.

Quoted 4 lines · Nick Piggin

If it wasn't clear was talking specifically about the hooks for page tracking rather than the whole patchset. If anybody wants such page tracking infrastructure in the kernel, then this (as opposed to the huge beancounters stuff) is what it should look like.

Could you point to the patch and a description for what is meant here by page tracking (did not see that in the patch, maybe I could not find it)? If these are just statistics then we likely already have them.

Patch 2/5 in this series provides hooks, and they are pretty unintrusive.

Quoted 1 line · Nick Piggin

Patch 2/5 in this series provides hooks, and they are pretty unintrusive.

Ok. We shadow existing vm counters add stuff to the adress_space structure. The task add / remove is duplicating what some of the cpuset hooks do. That clearly shows that we are just duplicating functionality.

But as I said above, I don't know what the containers and workload management people want exactly... The recent discussions about using nodes and cpusets for memory workload management does seem like a promising idea, and if it would avoid the need for this kind of per-page tracking entirely, then that would probably be even better.

Anyway I don't think I have much to say other than: this is almost exactly as I had imagined the memory resource tracking should look like. Just a small number of hooks and a very simple set of rules for tracking allocations. Also, the possibility to track kernel allocations as a whole rather than at individual callsites (which shouldn't be too difficult to implement).

If anything I would perhaps even argue for further cutting down the number of hooks and add them back as they prove to be needed.

I'm not sure about containers & workload management people, but from a core mm/ perspective I see no reason why this couldn't get in, given review and testing. Great!

I'd much rather containterize the whole reclaim code, which should not be too hard since he already adds a container pointer to struct page.

Esp. when we get some of my page reclaim abstractions merged, moving the reclaim from struct zone to a container is not a lot of work. (this is basically what one of the ckrm mm policies did too)

I still have to reread what Rohit does for file backed pages, that gave my head a spin.
I've been thinking a bit on that problem, and it would be possible to share all address_space pages equally between attached containers, this would lose some accuracy, since one container could read 10% of the file and another 90%, but I don't think that is a common scenario.

Quoted 4 lines

I've been thinking a bit on that problem, and it would be possible to share all address_space pages equally between attached containers, this would lose some accuracy, since one container could read 10% of the file and another 90%, but I don't think that is a common scenario.

Yeah, I'm not sure about that. I don't think really complex schemes are needed... but again I might need more knowledge of their workloads and problems.

Ar Iau, 2006-09-21 am 03:00 +1000, ysgrifennodd Nick Piggin:

Quoted 6 lines

Quoted 4 lines

I've been thinking a bit on that problem, and it would be possible to share all address_space pages equally between attached containers, this would lose some accuracy, since one container could read 10% of the file and another 90%, but I don't think that is a common scenario.

Quoted 3 lines

Yeah, I'm not sure about that. I don't think really complex schemes are needed... but again I might need more knowledge of their workloads and problems.

Any scenario which permits "cheating" will be a scenario that happens because people will try and cheat.

Yeah, I'm not sure about that. I don't think really complex schemes are needed... but again I might need more knowledge of their workloads and problems.

(this time to the lists as well)

Quoted 2 lines · Peter Zijlstra

I'd much rather containterize the whole reclaim code, which should not be too hard since he already adds a container pointer to struct page.

Quoted 2 lines

I still have to reread what Rohit does for file backed pages, that gave my head a spin.

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Quoted 4 lines · Rohit Seth

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Which comes naturally with cpusets.

Quoted 7 lines · Christoph Lameter

Quoted 4 lines · Rohit Seth

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Quoted 1 line

Which comes naturally with cpusets.

How are shared mappings dealt with, are pages charged to the set that first faults them in?

Quoted 8 lines · Peter Zijlstra

Quoted 7 lines · Christoph Lameter

Quoted 4 lines · Rohit Seth

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Quoted 5 lines

Quoted 1 line

Which comes naturally with cpusets.

How are shared mappings dealt with, are pages charged to the set that first faults them in?

For anonymous pages (simpler case), they get charged to the faulting task's container.

For filesystem pages (could be shared across tasks running different containers): Every time a new file mapping is created, it is bound to a container of the process creating that mapping. All subsequent pages belonging to this mapping will belong to this container, irrespective of different tasks running in different containers accessing these pages. Currently, I've not implemented a mechanism to allow a file to be specifically moved into or out of container. But when that gets implemented then all pages belonging to a mapping will also move out of container (or into a new container).

Yes, I read that in your patches, I was wondering how the cpuset approach would handle this.

Yes, I read that in your patches, I was wondering how the cpuset approach would handle this.

Neither are really satisfactory for shared mappings.

Quoted 1 line

Which comes naturally with cpusets.

How are shared mappings dealt with, are pages charged to the set that first faults them in?

Quoted 1 line

Which comes naturally with cpusets.

How are shared mappings dealt with, are pages charged to the set that first faults them in?

Quoted 4 lines · Peter Zijlstra

Quoted 1 line

Which comes naturally with cpusets.

How are shared mappings dealt with, are pages charged to the set that first faults them in?

They are charged to the node from which they were allocated. If the process is restricted to the node (container) then all pages allocated are are charged to the container regardless if they are shared or not.

Quoted 8 lines · Nick Piggin

(this time to the lists as well)

Quoted 2 lines · Peter Zijlstra

I'd much rather containterize the whole reclaim code, which should not be too hard since he already adds a container pointer to struct page.

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Quoted 2 lines

Quoted 2 lines

I still have to reread what Rohit does for file backed pages, that gave my head a spin.

Please let me know if there is any specific part that isn't making much sense.

Quoted 10 lines · Rohit Seth

Quoted 8 lines · Nick Piggin

(this time to the lists as well)

Quoted 2 lines · Peter Zijlstra

I'd much rather containterize the whole reclaim code, which should not be too hard since he already adds a container pointer to struct page.

Quoted 4 lines

Right now the memory handler in this container subsystem is written in such a way that when existing kernel reclaimer kicks in, it will first operate on those (container with pages over the limit) pages first. But in general I like the notion of containerizing the whole reclaim code.

Patch 5/5 seems to have a horrid deactivation scheme.

Quoted 5 lines

Quoted 2 lines

Quoted 2 lines

I still have to reread what Rohit does for file backed pages, that gave my head a spin.

Please let me know if there is any specific part that isn't making much sense.

Well, the whole over the limit handler is quite painfull, having taken a second reading it isn't all that complex after all, just odd.

You just start invalidating whole files for file backed pages. Granted, this will get you below the threshold. but you might just have destroyed your working set.

Pretty much the same for you anonymous memory handler, you scan through the pages in linear fashion and demote the first that you encounter.

Both things pretty thoroughly destroy the existing kernel reclaim.

This is based on lot of discussions over last month or so. I hope this patch set is something that we can agree and more support can be added on top of this. Please provide feedback and add other extensions that are useful in the TODO list.

Quoted 5 lines · Rohit Seth

This is based on lot of discussions over last month or so. I hope this patch set is something that we can agree and more support can be added on top of this. Please provide feedback and add other extensions that are useful in the TODO list.

thanks rohit for this patchset.

I applied it on rc7-mm1, compiles smoothly and boots. Here's a edge issue, I got this oops while rmdir a container with running tasks.

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

Quoted 5 lines · Rohit Seth

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

I just saw this for the first time since linux-mm was not cced. We have discussed a similar mechanism on linux-mm.

We already have such a functionality in the kernel its called a cpuset. A container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

Quoted 8 lines

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

Quoted 3 lines

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

What I like about Rohit's patches is the page tracking stuff which seems quite simple but capable.

I suspect cpusets don't quite provide enough features for non-exclusive use of memory (eg. page tracking for directed reclaim).

Quoted 9 lines · Nick Piggin

Quoted 3 lines

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

What I like about Rohit's patches is the page tracking stuff which seems quite simple but capable.

I suspect cpusets don't quite provide enough features for non-exclusive use of memory (eg. page tracking for directed reclaim).

Look at the VM statistics please. We have detailed page statistics per zone these days. If there is anything missing then this would best be put into general functionality. When I looked at it, I saw page statistics that were replicating things that we already track per zone. All these would become available if a container is realized via a node and we would be using proven VM code.

Look at what the patches do. These are not only for hard partitioning of memory per container but also those that share memory (eg. you might want each to share 100MB of memory, up to a max of 80MB for an individual container).

The nodes+cpusets stuff doesn't seem to help with that because you with that because you fundamentally need to track pages on a per container basis otherwise you don't know who's got what.

Now if, in practice, it turns out that nobody really needed these features then of course I would prefer the cpuset+nodes approach. My point is that I am not in a position to know who wants what, so I hope people will come out and discuss some of these issues.

Quoted 4 lines

Look at what the patches do. These are not only for hard partitioning of memory per container but also those that share memory (eg. you might want each to share 100MB of memory, up to a max of 80MB for an individual container).

So far I have not been able to find the hooks to the VM. The sharing would also work with nodes. Just create a couple of nodes with the sizes you want and then put the node with the shared memory into the cpusets for the apps sharing them.

Quoted 3 lines

The nodes+cpusets stuff doesn't seem to help with that because you with that because you fundamentally need to track pages on a per container basis otherwise you don't know who's got what.

Hmmm... That gets into issues of knowing how many pages are in use by an application and that is fundamentally difficult to do due to pages being shared between processes.

Quoted 4 lines

Now if, in practice, it turns out that nobody really needed these features then of course I would prefer the cpuset+nodes approach. My point is that I am not in a position to know who wants what, so I hope people will come out and discuss some of these issues.

Quoted 3 lines

Hmmm... That gets into issues of knowing how many pages are in use by an application and that is fundamentally difficult to do due to pages being shared between processes.

Fundamentally difficult or not, it seems to be required for what Nick describes, and for sure cpusets doesn't do it (track memory usage per container.)

We already have such a functionality in the kernel its called a cpuset. A container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

Ar Mer, 2006-09-20 am 09:25 -0700, ysgrifennodd Christoph Lameter:

Quoted 5 lines

We already have such a functionality in the kernel its called a cpuset. A container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

CPUsets don't appear to scale to large numbers of containers (say 5000, with 200-500 doing stuff at a time). They also don't appear to do any tracking of kernel side resource objects, which is critical to containment. Indeed for some of us the CPU management and user memory management angle is mostly uninteresting.

I'm also not clear how you handle shared pages correctly under the fake node system, can you perhaps explain that further how this works for say a single apache/php/glibc shared page set across 5000 containers each a web site.

The scalability issues can certainly be managed. See the discussions on linux-mm.

Kernel side resource objects? slab pages? Those are tracked.

Cpusets can share nodes. I am not sure what the problem would be? Paul may be able to give you more details.

If it can do it in a human understandable way, configured at runtime with dynamic sharing, overcommit and reconfiguration of sizes then great. Lets see what Paul has to say.

Quoted 2 lines

Cpusets can share nodes. I am not sure what the problem would be? Paul may be able to give you more details.

If it can do it in a human understandable way, configured at runtime with dynamic sharing, overcommit and reconfiguration of sizes then great. Lets see what Paul has to say.

Quoted 4 lines

I'm also not clear how you handle shared pages correctly under the fake node system, can you perhaps explain that further how this works for say a single apache/php/glibc shared page set across 5000 containers each a web site.

Cpusets can share nodes. I am not sure what the problem would be? Paul may be able to give you more details.

I'm also not clear how you handle shared pages correctly under the fake node system, can you perhaps explain that further how this works for say a single apache/php/glibc shared page set across 5000 containers each a web site.

Quoted 4 lines

I'm also not clear how you handle shared pages correctly under the fake node system, can you perhaps explain that further how this works for say a single apache/php/glibc shared page set across 5000 containers each a web site.

If you can associate files with containers, you can have a "shared libraries" container that the libraries/binaries for apache/php/glibc are associated with - all pages from those files are then accounted to the shared container.

The way you "associate" a file with a cpuset is to have some task in that cpuset open that file and touch its pages -- where that task does so before any other would be user of the file.

Such pre-touching of files is common occurrence on the HPC (High Perf Comp.) apps that run on the big honkin NUMA iron where cpusets were born. I'm guessing that someone hosting 5000 web servers would rather not deal with that particular hastle.

An alternative would be a way of binding files (or directory hierarchies) to a particular set of memory nodes. Then you wouldn't need to pre-fault the data. Extended attributes might be one way of doing it.

Some of the file system folks have considered such use of extended attributes, yes.

I remain unaware that any relation between that work and cpusets exists or should exist.

It doesn't have to be linked to cpusets - but userspace could use it in conjunction with cpusets to control/account pagecache memory sharing.

Quoted 5 lines · Rohit Seth

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

I just saw this for the first time since linux-mm was not cced. We have discussed a similar mechanism on linux-mm.

We already have such a functionality in the kernel its called a cpuset. A container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

Quoted 8 lines

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

cpusets provides cpu and memory NODES binding to tasks. And I think it works great for NUMA machines where you have different nodes with its own set of CPUs and memory. The number of those nodes on a commodity HW is still 1. And they can have 8-16 CPUs and in access of 100G of memory. You may start using fake nodes (untested territory) to

Containers also provide a mechanism to move files to containers. Any further references to this file come from the same container rather than the container which is bringing in a new page.

In future there will be more handlers like CPU and disk that can be easily embeded into this container infrastructure.

cpusets provides cpu and memory NODES binding to tasks. And I think it works great for NUMA machines where you have different nodes with its own set of CPUs and memory. The number of those nodes on a commodity HW is still 1. And they can have 8-16 CPUs and in access of 100G of memory. You may start using fake nodes (untested territory) to

translate a single node machine into N different nodes. But am not sure if this number of nodes can change dynamically on the running machine or a reboot is required to change the number of nodes.

Though when you want to have in access of 100 containers then the cpuset function starts popping up on the oprofile chart very aggressively. And this is the cost that shouldn't have to be paid (particularly) for a single node machine.

And what happens when you want to have cpuset with memory that needs to be even further fine grained than each node.

Containers also provide a mechanism to move files to containers. Any further references to this file come from the same container rather than the container which is bringing in a new page.

In future there will be more handlers like CPU and disk that can be easily embeded into this container infrastructure.

I think we should have one container mechanism instead of multiple. Maybe merge the two? The cpuset functionality is well established and working right.

Quoted 5 lines · Rohit Seth

cpusets provides cpu and memory NODES binding to tasks. And I think it works great for NUMA machines where you have different nodes with its own set of CPUs and memory. The number of those nodes on a commodity HW is still 1. And they can have 8-16 CPUs and in access of 100G of memory. You may start using fake nodes (untested territory) to

See linux-mm. We just went through a series of tests and functionality wise it worked just fine.

Quoted 3 lines

translate a single node machine into N different nodes. But am not sure if this number of nodes can change dynamically on the running machine or a reboot is required to change the number of nodes.

This is commonly discussed under the subject of memory hotplug.

Quoted 4 lines

Though when you want to have in access of 100 containers then the cpuset function starts popping up on the oprofile chart very aggressively. And this is the cost that shouldn't have to be paid (particularly) for a single node machine.

Yes this is a new way of using cpusets but it works and we could fix the scalability issues rather than adding new subsystems.

Quoted 2 lines

And what happens when you want to have cpuset with memory that needs to be even further fine grained than each node.

New node?

Quoted 3 lines

Containers also provide a mechanism to move files to containers. Any further references to this file come from the same container rather than the container which is bringing in a new page.

Hmmmm... Thats is interesting.

Quoted 2 lines

In future there will be more handlers like CPU and disk that can be easily embeded into this container infrastructure.

I think we should have one container mechanism instead of multiple. Maybe merge the two? The cpuset functionality is well established and working right.

I thought the fake NUMA support still does not work on x86_64 baseline kernel. Though Paul and Andrew have patches to make it work.

Quoted 1 line

This is commonly discussed under the subject of memory hotplug.

So now we depend on getting memory hot-plug to work for faking up these nodes ...for the memory that is already present in the system. It just does not sound logical.

I thought the fake NUMA support still does not work on x86_64 baseline kernel. Though Paul and Andrew have patches to make it work.

So now we depend on getting memory hot-plug to work for faking up these nodes ...for the memory that is already present in the system. It just does not sound logical.

the kernel. A useful memory containerization should (IMHO) allow for both adding and removing such containers.

Quoted 2 lines · Paul Jackson

the kernel. A useful memory containerization should (IMHO) allow for both adding and removing such containers.

How does the containers implementation under discussion behave if a process is part of a container and the container is removed?

Quoted 2 lines

How does the containers implementation under discussion behave if a process is part of a container and the container is removed?

It first removes all the tasks belonging to this container (which means resetting the container pointers in task_struct and then per page container pointer belonging to anonymous pages). It then clears the container pointers in the mapping structure and also in the pages belonging to these files.

Quoted 2 lines · Rohit Seth

Quoted 2 lines

How does the containers implementation under discussion behave if a process is part of a container and the container is removed?

Quoted 5 lines

It first removes all the tasks belonging to this container (which means resetting the container pointers in task_struct and then per page container pointer belonging to anonymous pages). It then clears the container pointers in the mapping structure and also in the pages belonging to these files.

So the application continues to run unharmed?

Could we equip containers with restrictions on processors and nodes for NUMA?

Quoted 2 lines

Could we equip containers with restrictions on processors and nodes for NUMA?

Yes. That is something we will have to do (I think part of CPU handler-TBD).

Quoted 4 lines

Quoted 2 lines

Could we equip containers with restrictions on processors and nodes for NUMA?

Yes. That is something we will have to do (I think part of CPU handler-TBD).

Paul: Will we still need cpusets if that is there?

Yes. There's quite a bit more to cpusets than just some form, any form, of CPU and Memory restriction. I can't imagine that Containers, in any form, are going to replicate that API.

Quoted 3 lines · Seth

So now we depend on getting memory hot-plug to work for faking up these nodes ...for the memory that is already present in the system. It just does not sound logical.

It's logical to me. Part of memory hotplug is adding physial memory, which is not an issue here. Part of it is adding another logical memory node (turning on another bit in node_online_map) and fixing up any code that thought a systems memory nodes were baked in at boottime. Perhaps the hardest part is the memory hot-un-plug, which would become more urgently needed with such use of fake numa nodes. The assumption that memory doesn't just up and vanish is non-trivial to remove from the kernel. A useful memory containerization should (IMHO) allow for both adding and removing such containers.

Absolutely. Since these containers are not (hard) partitioning the memory in any way so it is easy to change the limits (effectively reducing and increasing the memory limits for tasks belonging to containers). As you said, memory hot-un-plug is important and it is non-trivial amount of work.

Quoted 5 lines · Rohit Seth

Absolutely. Since these containers are not (hard) partitioning the memory in any way so it is easy to change the limits (effectively reducing and increasing the memory limits for tasks belonging to containers). As you said, memory hot-un-plug is important and it is non-trivial amount of work.

Maybe the hotplug guys want to contribute to the discussion?

Quoted 8 lines · On Wed, 20 Sep 2006 16:31:22 -0700 (PDT) Christoph Lameter <[email protected]>…

Quoted 5 lines · Rohit Seth

Absolutely. Since these containers are not (hard) partitioning the memory in any way so it is easy to change the limits (effectively reducing and increasing the memory limits for tasks belonging to containers). As you said, memory hot-un-plug is important and it is non-trivial amount of work.

Quoted 2 lines

Maybe the hotplug guys want to contribute to the discussion?

Ah, I'm reading threads with interest.

For example

In following scenario,
==
(1). add <pid> > /mnt/configfs/containers/my_container/add_task (2). <pid> does some work.
(3). echo <pid> > /mnt/configfs/containers/my_container/rm_task (4). echo <pid> > /mnt/configfs/containers/my_container2/add_task ==
(if fake-pgdat/memory-hotplug is used)
The pages used by <pid> in (2) will be accounted in 'my_container' after (3). Is this user's wrong use of system ?

But am not sure
if this number of nodes can change dynamically on the running machine or a reboot is required to change the number of nodes.

Though when you want to have in access of 100 containers then the cpuset function starts popping up on the oprofile chart very aggressively.

Containers also provide a mechanism to move files to containers. Any further references to this file come from the same container rather than the container which is bringing in a new page.

In future there will be more handlers like CPU and disk that can be easily embeded into this container infrastructure.

All in all, it could avoid anything more than trivial changes to the existing memory allocation code hot paths. But the infrastructure needed for managing this mechanism needs some non-trivial work.

Quoted 3 lines · Seth

But am not sure
if this number of nodes can change dynamically on the running machine or a reboot is required to change the number of nodes.

The current numa=fake=N kernel command line option is just boottime, and just x86_64.

I presume we'd have to remove these two constraints for this to be generally usable to containerize memory.

We also, in my current opinion, need to fix up the node_distance between such fake numa sibling nodes, to correctly reflect that they are on the same real node (LOCAL_DISTANCE).

And some non-trivial, arch-specific, zonelist sorting and reconstruction work will be needed.

And an API devised for the above mentioned dynamic changing.

And this will push on the memory hotplug/unplug technology.

All in all, it could avoid anything more than trivial changes to the existing memory allocation code hot paths. But the infrastructure needed for managing this mechanism needs some non-trivial work.

Quoted 2 lines

Though when you want to have in access of 100 containers then the cpuset function starts popping up on the oprofile chart very aggressively.

As the linux-mm discussion last weekend examined in detail, we can eliminate this performance speed bump, probably by caching the last zone on which we found some memory. The linear search that was implicit in __alloc_pages()'s use of zonelists for many years finally become explicit with this new usage pattern.

Quoted 4 lines

Containers also provide a mechanism to move files to containers. Any further references to this file come from the same container rather than the container which is bringing in a new page.

I haven't read these patches enough to quite make sense of this, but I suspect that this is not a distinction between cpusets and these containers, for the basic reason that cpusets doesn't need to 'move' a file's references because it has no clue what such are.

Quoted 3 lines

In future there will be more handlers like CPU and disk that can be easily embeded into this container infrastructure.

This may be a deciding point.

But container support will allow the certain files pages to come from the same container irrespective of who is using them. Something useful for shared libs etc.

Quoted 5 lines · Rohit Seth

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

I just saw this for the first time since linux-mm was not cced. We have discussed a similar mechanism on linux-mm.

We already have such a functionality in the kernel its called a cpuset. A

container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

Quoted 8 lines

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Quoted 1 line

We already have such a functionality in the kernel its called a cpuset. A

Christoph,

There had been multiple discussions in the past (as recent as Aug 18, 2006), where we (Paul and CKRM/RG folks) have concluded that cpuset and resource management are orthogonal features.

cpuset provides "resource isolation", and what we, the resource management guys want is work-conserving resource control.

Quoted 10 lines · Chandra Seetharaman

Quoted 1 line

We already have such a functionality in the kernel its called a cpuset. A

Christoph,

There had been multiple discussions in the past (as recent as Aug 18, 2006), where we (Paul and CKRM/RG folks) have concluded that cpuset and resource management are orthogonal features.

cpuset provides "resource isolation", and what we, the resource management guys want is work-conserving resource control.

CPUset provides two things:

- a generic process container abstraction

- "resource controllers" for CPU masks and memory nodes.

Rather than adding a new process container abstraction, wouldn't it make more sense to change cpuset to make it more extensible (more separation between resource controllers), possibly rename it to "containers", and let the various resource controllers fight it out (e.g. zone/node-based memory controller vs multiple LRU controller, CPU masks vs a properly QoS-based CPU scheduler, etc)

Or more specifically, what would need to be added to cpusets to make it possible to bolt the CKRM/RG resource controllers on to it?

Paul

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

We had this discussion more than 18 months back and concluded that it is not the right thing to do. Here is the link to the thread:

Quoted 3 lines · Chandra Seetharaman

We had this discussion more than 18 months back and concluded that it is not the right thing to do. Here is the link to the thread:

Even if the resource control portions aren't totally compatible, having two separate process container abstractions in the kernel is sub-optimal, both in terms of efficiency and userspace management. How about splitting out the container portions of cpuset from the actual resource control, so that CKRM/RG can hang off of it too? Creation of a cpuset or a resource group would be driven by creation of a container; at fork time, a task inherits its parent's container, and hence its cpuset and/or resource groups.

At its most crude, this could be something like:

struct container {
#ifdef CONFIG_CPUSETS
struct cpuset cs;
#endif
#ifdef CONFIG_RES_GROUPS
struct resource_group rg;
#endif
};

but at least it would be sharing some of the abstractions.

Paul

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Quoted 10 lines

At its most crude, this could be something like:

struct container {
#ifdef CONFIG_CPUSETS
struct cpuset cs;
#endif
#ifdef CONFIG_RES_GROUPS
struct resource_group rg;
#endif
};

Won't it restrict the user to choose one of these, and not both.

Quoted 11 lines · Chandra Seetharaman

Quoted 10 lines

At its most crude, this could be something like:

struct container {
#ifdef CONFIG_CPUSETS
struct cpuset cs;
#endif
#ifdef CONFIG_RES_GROUPS
struct resource_group rg;
#endif
};

Quoted 1 line

Won't it restrict the user to choose one of these, and not both.

Not necessarily - you could have both compiled in, and each would only worry about the resource management that they cared about - e.g. you could use the memory node isolation portion of cpusets (in conjunction with fake numa nodes/zones) for memory containment, but give every cpuset access to all CPUs and control CPU usage via the resource groups CPU controller.

The generic code would take care of details like container creation/destruction (with appropriate callbacks into cpuset and/or res_group code, tracking task membership of containers, etc.

Paul

What I am wondering is that whether the tight coupling of rg and cpuset (into a container data structure) is ok.

Quoted 2 lines · Chandra

What I am wondering is that whether the tight coupling of rg and cpuset (into a container data structure) is ok.

Just guessing wildly here, but I'd anticipate that at best we (resource groups and cpusets) would share container mechanisms, but not share the same container instances.

What I am wondering is that whether the tight coupling of rg and cpuset (into a container data structure) is ok.

Even if the resource control portions aren't totally compatible, having two separate process container abstractions in the kernel is sub-optimal

It seems that cpusets can mimic memory resource groups. I don't see how cpusets could mimic other resource groups. But maybe I'm just being a dimm bulb.

I'm not saying that they can - but they could be parallel types of resource controller for a generic container abstraction,

idea of separation between the container aspects of cpusets (which are almost identical to the container aspects of resource groups) and the resource control portions. Essentially, I've

- ripped out the cpusetfs portions of cpusets and moved them to container.c

these various mechanisms, including cpusets. It would be good to persue this common technology infrastructure -before- the kernel to user API of these other thingies gets frozen, as the details of just what this common code can do conveniently might impact details of the kernel API presented to users.

Indeed - although one of the advantages of the patch I sent is that it can continue to present exactly the same interface, and functionality, to userspace as the current cpusets code does.

Quoted 8 lines · Paul Jackson

Quoted 2 lines · Paul M.

I'm not saying that they can - but they could be parallel types of resource controller for a generic container abstraction,

When there are a sufficiently large number of sufficiently similar types of objects, such as for example file systems, then a 'generic container abstraction' such as vfs in the file system case becomes well worth it, even essential.

Well, cpusets, resource groups and bean counters are all process container abstractions of a sort, all with different management interfaces to userspace. But they all have the notion of wanting to be able to group tasks together.

Here's the kind of approach I'm suggesting. Note that the extremely incomplete patch attached (not included inline since it's 87K) hasn't even been compiled, let alone tested - it's just to illustrate the idea of separation between the container aspects of cpusets (which are almost identical to the container aspects of resource groups) and the resource control portions. Essentially, I've

- ripped out the cpusetfs portions of cpusets and moved them to container.c

- added some hooks from the container system into cpusets, to e.g. populate a containerfs dir with the appropriate cpusets files, or set the right memory/cpu masks for a task that moves into a container.

- adjusted places that would need to be updated to call container_xxx() rather than cpuset_xxx().

It would be fairly easy to hook resource groups into the same container system.

Paul

Quoted 3 lines · Paul M

Even if the resource control portions aren't totally compatible, having two separate process container abstractions in the kernel is sub-optimal

At heart, CKRM (ne Resource Groups) are (well, have been until now) different than cpusets.

Cpusets answers the question 'where', and Resource Groups 'how much'.

The fundamental motivation behind cpusets was to be able to enforce job isolation. A job can get dedicated use of specified resources, -even- if it means those resources are severely underutilized by that job.

The fundamental motivation (Chandra or others correct me if I'm wrong) of Resource Groups is to improve capacity utilization while limiting starvation due to greedy, competing users for the same resources.

Cpusets seeks maximum isolation. Resource Groups seeks maximum capacity utilization while preserving guaranteed levels of quality of service.

Cpusets are that wall between you and the neighbor you might not trust. Resource groups are a large family of modest wealth sitting down to share a meal.

It seems that cpusets can mimic memory resource groups. I don't

see how cpusets could mimic other resource groups. But maybe I'm just being a dimm bulb.

Quoted 1 line

It seems that cpusets can mimic memory resource groups. I don't

I am little confused w.r.t how cpuset can mimic memory resource groups. How can cpuset provide support for over commit.

I have some patches locally that basically let you give out a small set of nodes initially to a cpuset, and if memory pressure in try_to_free_pages() passes a specified threshold, automatically allocate one of the parent cpuset's unused memory nodes to the child cpuset, up to specified limit.

We had this discussion more than 18 months back and concluded that it is not the right thing to do. Here is the link to the thread:

We had this discussion more than 18 months back and concluded that it is not the right thing to do. Here is the link to the thread:

For other resources, such as CPU cycles and network bandwidth, unless another bright spark comes up with an insight, I don't see how to express the "percentage used" semantics provided by something such as CKRM, using anything resembling cpusets.

... Can one imagine having the scheduler subdivide each second of time available on a CPU into several fake-CPUs, each one of which speaks for one of those sub-second fake-CPU slices? Sounds too weird to me, and a bit too rigid to be a servicable CKRM substitute.

How abt metered cpusets? Each child cpuset of a metered cpuset represents a fraction of CPU time alloted to the tasks of the child cpuset.

Rather than adding a new process container abstraction, wouldn't it make more sense to change cpuset to make it more extensible (more separation between resource controllers), possibly rename it to "containers",

Quoted 4 lines · Paul M.

Rather than adding a new process container abstraction, wouldn't it make more sense to change cpuset to make it more extensible (more separation between resource controllers), possibly rename it to "containers",

Without commenting one way or the other on the overall advisability of this (for lack of sufficient clues), if we did this and renamed "cpusets" to "containers", we would still want to export the /dev/cpuset interface to just the CPU/Memory controllers. Perhaps the "container" pseudo-filesystem could optionally be mounted with a "cpuset" option, that just exposed the cpuset relevant interface, or some such thing.

cpuset partitions resource and hence the resource that are assigned to a node is not available for other cpuset, which is not good for "resource management".

Quoted 3 lines · Chandra Seetharaman

cpuset partitions resource and hence the resource that are assigned to a node is not available for other cpuset, which is not good for "resource management".

cpusets can have one node in multiple cpusets.

AFAICS, That doesn't help me in over committing resources.

Quoted 1 line · Chandra

AFAICS, That doesn't help me in over committing resources.

I agree - I don't think cpusets plus fake numa ... handles over commit. You might could hack up a cheap substitute, but it wouldn't do the job.

Quoted 5 lines · Paul Jackson

Quoted 1 line · Chandra

AFAICS, That doesn't help me in over committing resources.

I agree - I don't think cpusets plus fake numa ... handles over commit. You might could hack up a cheap substitute, but it wouldn't do the job.

I have some patches locally that basically let you give out a small set of nodes initially to a cpuset, and if memory pressure in try_to_free_pages() passes a specified threshold, automatically allocate one of the parent cpuset's unused memory nodes to the child cpuset, up to specified limit. It's a bit ugly, but lets you trade of performance vs memory footprint on a per-job basis (when combined with fake numa to give lots of small nodes).

Paul

Interesting. So you could set up the fake node with "guarantee" and let it grow till "limit" ?

BTW, can you do these with fake nodes:
- dynamic creation
- dynamic removal
- dynamic change of size

Also, How could we account when a process moves from one node to another ?

Quoted 3 lines · Chandra Seetharaman

Interesting. So you could set up the fake node with "guarantee" and let it grow till "limit" ?

Sure - that works great. (Theoretically you could do this all in userspace - start by assigning "guarantee" nodes to a container/cpuset and when it gets close to its memory limit assign more nodes to it. But in practice userspace can't keep up with rapid memory allocators.

Quoted 5 lines

BTW, can you do these with fake nodes:
- dynamic creation
- dynamic removal
- dynamic change of size

The current fake numa support requires you to choose your node layout at boot time - I've been working with 64 fake nodes of 128M each, which gives a reasonable granularity for dividing a machine between multiple different sized jobs.

Quoted 4 lines

Also, How could we account when a process moves from one node to another ?

If you want to do that (the systems I'm working on don't really) you could probably do it with the migrate_pages() syscall. It might not be that efficient though.

Paul

Quoted 4 lines

The current fake numa support requires you to choose your node layout at boot time - I've been working with 64 fake nodes of 128M each, which gives a reasonable granularity for dividing a machine between multiple different sized jobs.

It still will not satisfy what OpenVZ/Container folks are looking for:
100s of containers.

Quoted 5 lines · Chandra Seetharaman

Quoted 4 lines

The current fake numa support requires you to choose your node layout at boot time - I've been working with 64 fake nodes of 128M each, which gives a reasonable granularity for dividing a machine between multiple different sized jobs.

Quoted 2 lines

It still will not satisfy what OpenVZ/Container folks are looking for:
100s of containers.

Right - so fake-numa is not the right solution for everyone, and I never suggested that it is. (Having said that, there are discussions underway to make the zone-based approach more practical - if you could have dynamically-resizable nodes, this would be more applicable to openvz).

But, there's no reason that the OpenVZ resource control mechanisms couldn't be hooked into a generic process container mechanism along with cpusets and RG.

Paul

Quoted 5 lines

But, there's no reason that the OpenVZ resource control mechanisms couldn't be hooked into a generic process container mechanism along with cpusets and RG.

Isn't that one of the things we are trying to avoid (each one having their own solution, especially when we _can_ have a common solution).

Quoted 8 lines · Chandra Seetharaman

Quoted 5 lines

But, there's no reason that the OpenVZ resource control mechanisms couldn't be hooked into a generic process container mechanism along with cpusets and RG.

Isn't that one of the things we are trying to avoid (each one having their own solution, especially when we _can_ have a common solution).

Can we actually have a single common solution that works for everyone, no matter what their needs? It's already apparent that there are multiple different and subtly incompatible definitions of what "memory controller" means and needs to do. Maybe these can be resolved - but maybe it's better to have, say, two simple but very different memory controllers that the user can pick between, rather than one big and complicated one that tries to please everyone.

Paul

Think about what will be available to customer through a distro.

There are two (competing) memory controllers in the kernel. But, distro can turn only one ON. Which in turn mean

So, IMHO, it is better to sort out the differences before we get things in mainline kernel.

Quoted 4 lines · Chandra Seetharaman

Think about what will be available to customer through a distro.

There are two (competing) memory controllers in the kernel. But, distro can turn only one ON. Which in turn mean

Why's that? I don't see why cpuset memory nodemasks can't coexist with, say, the RG memory controller. They're attempting to solve different problems, and I can see situations where you might want to use both at once.

Quoted 4 lines

So, IMHO, it is better to sort out the differences before we get things in mainline kernel.

Agreed, if we can come up with a definition of e.g. memory controller that everyone agrees is suitable for their needs. You're assuming that's so a priori, I'm not yet convinced.

And I'm not trying to get another memory controller into the kernel, I'm just trying to get a standard process aggregation into the kernel (or rather, take the one that's already in the kernel and make it possible to hook other controller frameworks into it), so that the various memory controllers can become less intrusive patches in their own right.

Paul

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

There are two (competing) memory controllers in the kernel. But, distro can turn only one ON.

Quoted 2 lines · Chandra

There are two (competing) memory controllers in the kernel. But, distro can turn only one ON.

Huh - time for me to play the dummy again ...

My (fog shrouded) vision of the future has:
1) mempolicy - provides fine grained memory placement for task on self
2) cpuset - provides system wide cpu and memory placement for unrelated tasks
3) some form of resource groups - measures and limits proportion of various resources used, including cpu cycles, memory pages and network bandwidth, by collections of tasks.k

Both (2) and (3) need to group tasks in flexible ways distinct from the existing task groupings supported by the kernel.

I thought that Paul M suggested (2) and (3) use common underlying grouping or 'bucket' technology - the infrastructure that separates tasks into buckets and can be used to associate various resource metrics and limits with each bucket.

I can't quite figure out whether you have in mind above:
* a conflict between two competing memory controllers for (3),

* or a conflict between cpusets and one memory controller for (3).

And either way, I don't see what that has to do with the underling bucket technology - how we group tasks generically.

Guess I am missing something ...

I clarified it in the reply to Paul M.

But, there's no reason that the OpenVZ resource control mechanisms couldn't be hooked into a generic process container mechanism along with cpusets and RG.

Can the generic container avoid performance bottlenecks due to locks or other hot cache lines on the main code paths for fork, exit, page allocation and task scheduling?

Page allocation and task scheduling are resource controller issues, not generic process container issues.

Quoted 6 lines

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

I just saw this for the first time since linux-mm was not cced. We have discussed a similar mechanism on linux-mm.

We already have such a functionality in the kernel its called a cpuset. A

container could be created simply by creating a fake node that then allows constraining applications to this node. We already track the types of pages per node. The statistics you want are already existing. See /proc/zoneinfo and /sys/devices/system/node/node*/*.

Quoted 8 lines

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

Right thats what cpusets do and it has been working fine for years. Maybe Paul can help you if you find anything missing in the existing means to control resources.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Containers:

Commodity HW is becoming more powerful. This is giving opportunity to run different workloads on the same platform for better HW resource utilization. To run different workloads efficiently on the same platform, it is critical that we have a notion of limits for each workload in Linux kernel. Current cpuset feature in Linux kernel provides grouping of CPU and memory support to some extent (for NUMA machines).

For example, a user can run a batch job like backup inside containers. This job if run unconstrained could step over most of the memory present in system thus impacting other workloads running on the system at that time. But when the same job is run inside containers then the backup job is run within container limits.

We use the term container to indicate a structure against which we track and charge utilization of system resources like memory, tasks etc for a workload. Containers will allow system admins to customize the underlying platform for different applications based on their performance and HW resource utilization needs. Containers contain enough infrastructure to allow optimal resource utilization without bogging down rest of the kernel. A system admin should be able to create, manage and free containers easily.

At the same time, changes in kernel are minimized so as this support can be easily integrated with mainline kernel.

The user interface for containers is through configfs. Appropriate file system privileges are required to do operations on each container. Currently implemented container resources are automatically visible to user space through /configfs/container/<container_name> after a container is created.

Signed-off-by Rohit Seth <[email protected]>

Diffstat for the patch set (against linux-2.6.18-rc6-mm2_:

 Documentation/containers.txt |   65 ++++
 fs/inode.c                   |    3
 include/linux/container.h    |  167 ++++++++++
 include/linux/fs.h           |    5
 include/linux/mm_inline.h    |    4
 include/linux/mm_types.h     |    4
 include/linux/sched.h        |    6
 init/Kconfig                 |    8
 kernel/Makefile              |    1
 kernel/container_configfs.c  |  440 ++++++++++++++++++++++++++++
 kernel/exit.c                |    2
 kernel/fork.c                |    9
 mm/Makefile                  |    2
 mm/container.c               |  658 +++++++++++++++++++++++++++++++++++++++++++
 mm/container_mm.c            |  512 +++++++++++++++++++++++++++++++++
 mm/filemap.c                 |    4
 mm/page_alloc.c              |    3
 mm/rmap.c                    |    8
 mm/swap.c                    |    1
 mm/vmscan.c                  |    1
 20 files changed, 1902 insertions(+), 1 deletion(-)

Changes from version 1:
Fixed the Documentation error
Fixed the corruption in container task list
Added the support for showing all the tasks belonging to a container through showtask attribute
moved the Kconfig changes to init directory (from mm)
Fixed the bug of unregistering container subsystem if we are not able to create workqueue
Better support for handling limits for file pages. This now includes support for flushing and invalidating page cache pages. Minor other changes.

*****************************************************************
This patch set has basic container support that includes:

- Create a container using mkdir command in configfs

- Free a container using rmdir command

- Dynamically adjust memory and task limits for container.

- Add/Remove a task to container (given a pid)

- Files are currently added as part of open from a task that already belongs to a container.

- Keep track of active, anonymous, mapped and pagecache usage of container memory

- Does not allow more than task_limit number of tasks to be created in the container.

- Over the limit memory handler is called when number of pages (anon + pagecache) exceed the limit. It is also called when number of active pages exceed the page limit. Currently, this memory handler scans the mappings and tasks belonging to container (file and anonymous) and tries to deactivate pages. If the number of page cache pages is also high then it also invalidate mappings. The thought behind this scheme is, it is okay for containers to go over limit as long they run in degraded manner when they are over their limit. Also, if there is any memory pressure then pages belonging to over the limit container(s) become prime candidates for kernel reclaimer. Container mutex is also held during the time this handler is working its way through to prevent any further addition of resources (like tasks or mappings) to this container. Though it also blocks removal of same resources from the container for the same time. It is possible that over the limit page handler takes lot of time if memory pressure on a container is continuously very high. The limits, like how long a task should schedule out when it hits memory limit, is also on the lower side at present (particularly when it is memory hogger). But should be easy to change if need be.

- Indicate the number of times the page limit and task limit is hit

- Indicate the tasks (pids) belonging to container.

Below is a one line description for patches that will follow:

[patch01]: Documentation on how to use containers
(Documentation/container.txt)

[patch02]: Changes in the generic part of kernel code

[patch03]: Container's interface with configfs

[patch04]: Core container support

[patch05]: Over the limit memory handler.

TODO:

- some code(like container_add_task) in mm/container.c should go elsewhere.
- Support adding/removing a file name to container through configfs
- /proc/pid/container to show the container id (or name)
- More testing for memory controller. Currently it is possible that limits are exceeded. See if a call to reclaim can be easily integrated.
- Kernel memory tracking (based on patches from BC)
- Limit on user locked memory
- Huge memory support
- Stress testing with containers
- One shot view of all containers
- CKRM folks are interested in seeing all processes belonging to a container. Add the attribute show_tasks to container.
- Add logic so that the sum of limits are not exceeding appropriate system requirements.
- Extend it with other controllers (CPU and Disk I/O)
- Add flags bits for supporting different actions (like in some cases provide a hard memory limit and in some cases it could be soft).
- Capability to kill processes for the extreme cases.
...

This is based on lot of discussions over last month or so. I hope this patch set is something that we can agree and more support can be added on top of this. Please provide feedback and add other extensions that are useful in the TODO list.

Thanks,
-rohit

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
ckrm-tech mailing list
https://lists.sourceforge.net/lists/listinfo/ckrm-tech

Rohit,

I finally looked into your memory controller patches. Here are some of the issues I see:
(All points below are in the context of page limit of containers being hit and the new code starts freeing up pages)

1. LRU is ignored totally thereby thrashing the working set (as pointed by Peter Zijlstra).

2. Frees up file pages first when hitting the page limit thereby making vm_swappiness ineffective.

3. Starts writing back pages when the # of file pages is close to the limit, thereby breaking the current writeback algorithm/logic.

4. MAPPED files are not counted against the page limit. why ?. This affects reclamation behavior and makes vm_swappiness ineffective.

5. Starts freeing up pages from the first task or the first file in the linked list. This logic unfairly penalizes the early members of the list.

6. Both active and inactive pages use physical pages. But, the controller only counts active pages and not inactive pages. why ?

7. Page limit is checked against the sum of (anon and file pages) in some places and against active pages at some other places. IMO, it should be always compared to the same value.

Quoted 9 lines

Rohit,

I finally looked into your memory controller patches. Here are some of the issues I see:
(All points below are in the context of page limit of containers being hit and the new code starts freeing up pages)

1. LRU is ignored totally thereby thrashing the working set (as pointed by Peter Zijlstra).

As the container goes over the limit, this algorithm deactivates some of the pages. I agree that the logic to find out the correct pages to deactivate needs to be improved. But the idea is that these pages go in front of inactive list so that if there is any memory pressure system wide then these pages can easily be reclaimed.

Quoted 2 lines

2. Frees up file pages first when hitting the page limit thereby making vm_swappiness ineffective.

Not sure if I understood this part correctly. But the choice when the container goes over its limit is between swap out some of the anonymous memory first or writeback some of the dirty file pages belonging to this container.

Quoted 2 lines

3. Starts writing back pages when the # of file pages is close to the limit, thereby breaking the current writeback algorithm/logic.

That is done so as to ensure processes belonging to container (Whose limit is hit) are the first ones getting penalized. For example, if you run a tar in a container with 100MB limit then the dirty file pages will be written back to disk when 100MB limit is hit). Though I will be adding a HARD_LIMIT on page cache flag and the strict limit will be only maintained if this container flag is set.

Quoted 2 lines

4. MAPPED files are not counted against the page limit. why ?. This affects reclamation behavior and makes vm_swappiness ineffective.

num_mapped_pages only indicates how many page cache pages are mapped in user page tables. More of an accounting variable.

Quoted 4 lines

5. Starts freeing up pages from the first task or the first file in the linked list. This logic unfairly penalizes the early members of the list.

This is the part that I've to fix. Some per container variables that remembers the last values will help here.

Quoted 3 lines

6. Both active and inactive pages use physical pages. But, the controller only counts active pages and not inactive pages. why ?

The thought is, it is okay for containers to go over its limit as long

as there is enough memory in the system. When there is any memory pressure then the inactive (+ dereferenced) pages get swapped out thus penalizing the container. I'm also thinking of having hard limit for

anonymous pages beyond which the container will not be able to grow its anonymous pages.

Quoted 5 lines

7. Page limit is checked against the sum of (anon and file pages) in some places and against active pages at some other places. IMO, it should be always compared to the same value.

It is checked against sum of anon+file pages at the time when new pages

is getting allocated. But as the reclaimer activate the pages, so it is also important to make sure the number of active pages is not going above its limit.

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

Quoted 11 lines · Chandra Seetharaman

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

<snip>

Hi, Rohit,

I have been playing around with the containers patch. I finally got around to reading the code.

1. Comments on reclaiming

You could try the following options to overcome some of the disadvantages of the current scheme.

(a) You could consider a reclaim path based on Dave Hansen's Challenged memory controller (see http://marc.theaimsgroup.com/?l=linux-mm&m=115566982532345&w=2).

(b) The other option is to do what the resource group memory controller does - build a per group LRU list of pages (active, inactive) and reclaim them using the existing code (by passing the correct container pointer, instead of the zone pointer). One disadvantage of this approach is that the global reclaim is impacted as the global LRU list is broken. At the expense of another list, we could maintain two lists, global LRU and container LRU lists. Depending on the context of the reclaim - (container over limit, memory pressure) we could update/manipulate both lists. This approach is definitely very expensive.

2. Comments on task migration support

(a) One of the issues I found while using the container code is that, one could add a task to a container say "a". "a" gets charged for the tasks usage, when the same task moves to a different container say "b", when the task exits, the credit goes to "b" and "a" remains indefinitely charged.

(b) For tasks addition and removal, I think it's probably better to move the entire process (thread group) rather than allow each individual thread to move across containers. Having threads belonging to the same process reside in different containers can be complex to handle, since they share the same VM. Do you have a scenario where the above condition would be useful?

Quoted 25 lines · Balbir Singh

Quoted 11 lines · Chandra Seetharaman

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

<snip>

Hi, Rohit,

I have been playing around with the containers patch. I finally got around to reading the code.

1. Comments on reclaiming

You could try the following options to overcome some of the disadvantages of the current scheme.

Quoted 3 lines

(a) You could consider a reclaim path based on Dave Hansen's Challenged memory controller (see http://marc.theaimsgroup.com/?l=linux-mm&m=115566982532345&w=2).

I will go through that. Did you get a chance to stress the system and found any short comings that should be resolved.

Quoted 11 lines

(b) The other option is to do what the resource group memory controller does - build a per group LRU list of pages (active, inactive) and reclaim them using the existing code (by passing the correct container pointer, instead of the zone pointer). One disadvantage of this approach is that the global reclaim is impacted as the global LRU list is broken. At the expense of another list, we could maintain two lists, global LRU and container LRU lists. Depending on the context of the reclaim - (container over limit, memory pressure) we could update/manipulate both lists. This approach is definitely very expensive.

Two LRUs is a nice idea. Though I don't think it will go too far. It will involve adding another list pointers in the page structure. I agree that the mem handler is not optimal at all but I don't want to make it mimic kernel reclaimer at the same time.

Quoted 8 lines

2. Comments on task migration support

(a) One of the issues I found while using the container code is that, one could add a task to a container say "a". "a" gets charged for the tasks usage, when the same task moves to a different container say "b", when the task exits, the credit goes to "b" and "a" remains indefinitely charged.

hmm, when the task is removed from "a" then "a" gets the credits for the amount of anon memory that is used by the task. Or do you mean something different.

Quoted 9 lines

(b) For tasks addition and removal, I think it's probably better to move the entire process (thread group) rather than allow each individual thread to move across containers. Having threads belonging to the same process reside in different containers can be complex to handle, since they share the same VM. Do you have a scenario where the above condition would be useful?

I don't have a scenario where a task actually gets to move out of container (except exit). That asynchronous removal of tasks has already got the code very complicated for locking etc. But if you think moving a thread group is useful then I will add that functionality.

Thanks,
-rohit

Quoted 14 lines · Rohit Seth

Quoted 10 lines · Balbir Singh

(b) The other option is to do what the resource group memory controller does - build a per group LRU list of pages (active, inactive) and reclaim them using the existing code (by passing the correct container pointer, instead of the zone pointer). One disadvantage of this approach is that the global reclaim is impacted as the global LRU list is broken. At the expense of another list, we could maintain two lists, global LRU and container LRU lists. Depending on the context of the reclaim - (container over limit, memory pressure) we could update/manipulate both lists. This approach is definitely very expensive.

Quoted 4 lines

Two LRUs is a nice idea. Though I don't think it will go too far. It will involve adding another list pointers in the page structure. I agree that the mem handler is not optimal at all but I don't want to make it mimic kernel reclaimer at the same time.

One possible solution is to move the container tracking out of the pages and into address_space and anon_vma. I guess this functionality will complicate task migration and accounting a bit though.

Quoted 8 lines

Quoted 8 lines

2. Comments on task migration support

(a) One of the issues I found while using the container code is that, one could add a task to a container say "a". "a" gets charged for the tasks usage, when the same task moves to a different container say "b", when the task exits, the credit goes to "b" and "a" remains indefinitely charged.

Quoted 3 lines

hmm, when the task is removed from "a" then "a" gets the credits for the amount of anon memory that is used by the task. Or do you mean something different.

Aah, I see. Once possible minor concern here is that a task could hope across several containers, it could map files in each container and allocate page cache pages, when it reaches the limit, it could hop to another container and carry on until it hits the limit there.

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

Quoted 35 lines · Chandra Seetharaman

Quoted 9 lines

Rohit,

I finally looked into your memory controller patches. Here are some of the issues I see:
(All points below are in the context of page limit of containers being hit and the new code starts freeing up pages)

1. LRU is ignored totally thereby thrashing the working set (as pointed by Peter Zijlstra).

As the container goes over the limit, this algorithm deactivates some of the pages. I agree that the logic to find out the correct pages to deactivate needs to be improved. But the idea is that these pages go in front of inactive list so that if there is any memory pressure system wide then these pages can easily be reclaimed.

Quoted 2 lines

2. Frees up file pages first when hitting the page limit thereby making vm_swappiness ineffective.

Not sure if I understood this part correctly. But the choice when the container goes over its limit is between swap out some of the anonymous memory first or writeback some of the dirty file pages belonging to this container.

Quoted 2 lines

3. Starts writing back pages when the # of file pages is close to the limit, thereby breaking the current writeback algorithm/logic.

That is done so as to ensure processes belonging to container (Whose limit is hit) are the first ones getting penalized. For example, if you run a tar in a container with 100MB limit then the dirty file pages will be written back to disk when 100MB limit is hit). Though I will be adding a HARD_LIMIT on page cache flag and the strict limit will be only maintained if this container flag is set.

Quoted 5 lines

Quoted 2 lines

4. MAPPED files are not counted against the page limit. why ?. This affects reclamation behavior and makes vm_swappiness ineffective.

num_mapped_pages only indicates how many page cache pages are mapped in user page tables. More of an accounting variable.

But, # of mapped pages is used in the reclamation path logic. These set of patches doesn't take them into account.

Quoted 5 lines

Quoted 4 lines

5. Starts freeing up pages from the first task or the first file in the linked list. This logic unfairly penalizes the early members of the list.

Quoted 2 lines

This is the part that I've to fix. Some per container variables that remembers the last values will help here.

Yes, that will help in fairness between the items in the list.

But, it will still suffer from (1) above, as we would have no idea of the current working set (LRU) (within an item or among the items).

Quoted 5 lines

Quoted 3 lines

6. Both active and inactive pages use physical pages. But, the controller only counts active pages and not inactive pages. why ?

The thought is, it is okay for containers to go over its limit as long

Real number of "physical pages" used by the container is the sum of active and inactive pages.

From the user pov, the real sum of pages that are used by container for

My question is, shouldn't that be used to check against page limit instead of active pages alone ?

How do we describe "page limit" as (to the user) ?

Quoted 3 lines

as there is enough memory in the system. When there is any memory pressure then the inactive (+ dereferenced) pages get swapped out thus penalizing the container. I'm also thinking of having hard limit for

Reclamation goes through active pages and page cache pages before it gets into inactive pages. So, this may not work as you are explaining.

Quoted 3 lines

anonymous pages beyond which the container will not be able to grow its anonymous pages.

You might break the current behavior (memory pressure must be very high before these starts failing) if you are going to be strict about it.

Quoted 6 lines

Quoted 5 lines

7. Page limit is checked against the sum of (anon and file pages) in some places and against active pages at some other places. IMO, it should be always compared to the same value.

It is checked against sum of anon+file pages at the time when new pages

why can't we check against active pages here ?

Quoted 3 lines

is getting allocated. But as the reclaimer activate the pages, so it is also important to make sure the number of active pages is not going above its limit.

My point is that they won't be same (ever) and hence the check is inconsistent.

Quoted 4 lines · Chandra Seetharaman

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

Quoted 4 lines

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

I agree that this implementation differs from existing VM subsystem. But the key point here is, it puts the pages that should be reclaimed.

And this part needs further refining.

Quoted 3 lines

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

How? I don't want to duplicate the whole logic for containers.

Quoted 4 lines

But, it will still suffer from (1) above, as we would have no idea of the current working set (LRU) (within an item or among the items).

Please let me know how do you propose to have another LRU for pages in containers. Though I can add some heuristics.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

6. Both active and inactive pages use physical pages. But, the controller only counts active pages and not inactive pages. why ?

Quoted 7 lines

Quoted 1 line

The thought is, it is okay for containers to go over its limit as long

Real number of "physical pages" used by the container is the sum of active and inactive pages.

From the user pov, the real sum of pages that are used by container for

user land is anon + file. Now some times it is possible that there are active pages that are neither in page cache nor in use as anon.

Quoted 2 lines

My question is, shouldn't that be used to check against page limit instead of active pages alone ?

I can use active+inactive as the test. Sure. But I will have to also still have a check to make sure that number of active pages themselves is not bigger than page_limit.

Quoted 2 lines

How do we describe "page limit" as (to the user) ?

Amount of memory below which no container throttling will happen. And

if the system is properly configured that it also ensures that this much memory will always be there to user. If a container goes over this limit then it will be throttled and it will suffer performance.

Quoted 7 lines

Quoted 4 lines

as there is enough memory in the system. When there is any memory pressure then the inactive (+ dereferenced) pages get swapped out thus penalizing the container. I'm also thinking of having hard limit for

Reclamation goes through active pages and page cache pages before it gets into inactive pages. So, this may not work as you are explaining.

That is a good point. I'll have to make a check in reclaim so that when the system is ready for swap or write back then containers are looked first.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

anonymous pages beyond which the container will not be able to grow its anonymous pages.

You might break the current behavior (memory pressure must be very high before these starts failing) if you are going to be strict about it.

That feature when implemented will be a container specific.

Quoted 8 lines

Quoted 7 lines

Quoted 6 lines

7. Page limit is checked against the sum of (anon and file pages) in some places and against active pages at some other places. IMO, it should be always compared to the same value.

It is checked against sum of anon+file pages at the time when new pages

Quoted 9 lines

why can't we check against active pages here ?

Quoted 3 lines

is getting allocated. But as the reclaimer activate the pages, so it is also important to make sure the number of active pages is not going above its limit.

My point is that they won't be same (ever) and hence the check is inconsistent.

The check ensures
1- when a new page is getting added then the total sum of pages is checked against the limit.
2- Number of active pages don't exceed the limit.

These two points combined together enforce the decision that once the container goes over the limit, we scan the pages again to deactivate the excess.

Thanks,
-rohit

Quoted 10 lines · Rohit Seth

Quoted 8 lines · Chandra Seetharaman

On Wed, 2006-09-27 at 14:28 -0700, Rohit Seth wrote:

Rohit,

For 1-4, I understand the rationale. But, your implementation deviates from the current behavior of the VM subsystem which could affect the ability of these patches getting into mainline.

Quoted 2 lines

I agree that this implementation differs from existing VM subsystem. But the key point here is, it puts the pages that should be reclaimed.

But, you are putting the pages up for reclamation without any consideration to the working set and system memory pressure.

Quoted 7 lines

And this part needs further refining.

Quoted 3 lines

IMO, the current behavior in terms of reclamation, LRU, vm_swappiness, and writeback logic should be maintained.

How? I don't want to duplicate the whole logic for containers.

We don't have to be duplicating the whole logic. Just make sure that the existing mechanisms are aware of containers, if they exist.

<snip>

Quoted 7 lines

Quoted 4 lines

But, it will still suffer from (1) above, as we would have no idea of the current working set (LRU) (within an item or among the items).

Please let me know how do you propose to have another LRU for pages in containers. Though I can add some heuristics.

There are multiple ways as Balbir pointed in his email:
- reclamation per container (as in current RG implementation) ( + do a system wide reclaim when the system pressure is high)
- reclaim with the knowledge of containers that are over limit (Dave Hansen's patches + avoid overhead of combing the list)
- have two lists one for the system and one per container

Quoted 22 lines

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

6. Both active and inactive pages use physical pages. But, the controller only counts active pages and not inactive pages. why ?

Quoted 7 lines

Quoted 1 line

The thought is, it is okay for containers to go over its limit as long

Real number of "physical pages" used by the container is the sum of active and inactive pages.

From the user pov, the real sum of pages that are used by container for

user land is anon + file. Now some times it is possible that there are active pages that are neither in page cache nor in use as anon.

Quoted 2 lines

My question is, shouldn't that be used to check against page limit instead of active pages alone ?

I can use active+inactive as the test. Sure. But I will have to also still have a check to make sure that number of active pages themselves is not bigger than page_limit.

Quoted 4 lines

Quoted 2 lines

How do we describe "page limit" as (to the user) ?

Amount of memory below which no container throttling will happen. And

But, from the implementation one cannot clearly derive what we mean by "memory" here (physical ?, file + anon ?; if we say physical, it is not correct).

Quoted 3 lines

if the system is properly configured that it also ensures that this much memory will always be there to user. If a container goes over this limit then it will be throttled and it will suffer performance.

But, the user's expectation would be that we would be throwing out pages based on LRU (within that container). But this implementation doesn't provide that behavior. It doesn't care about the working set.

Performance impact will be lesser if we consider the working set and throw out pages based on LRU (within a container).

Quoted 13 lines

Quoted 7 lines

Quoted 4 lines

as there is enough memory in the system. When there is any memory pressure then the inactive (+ dereferenced) pages get swapped out thus penalizing the container. I'm also thinking of having hard limit for

Reclamation goes through active pages and page cache pages before it gets into inactive pages. So, this may not work as you are explaining.

That is a good point. I'll have to make a check in reclaim so that when the system is ready for swap or write back then containers are looked first.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

Quoted 2 lines

anonymous pages beyond which the container will not be able to grow its anonymous pages.

You might break the current behavior (memory pressure must be very high before these starts failing) if you are going to be strict about it.

That feature when implemented will be a container specific.

My point is, even though it is container specific, the behavior (inside a container) should be same as what a user sees at the system level now.

For example, consider a workload that is run on a 1G system now, and user sees only occasional memory allocation failures and just a handful of oom kills. When the workload is moved to a container with 1G, failures the user see should be in the same order ( and similar with performance characteristics).

Do you agree that it will be the user's expectation ?