Container mini-summit notes
Held at De Vere Universty Arms Hotel, Cambridge, UK
* Monday, Sept 3, 9h00 to 16h00 :
Kir Kolyshkin <[email protected]>
Pavel Emelianov <[email protected]> Masahiko Takahashi <masahiko-de/[email protected]> Oren Laadan <[email protected]>
James Youngman <[email protected]> ??? (NTT)
Cedric Le Goater <clg-NmTC/[email protected]>
On the phone (skype with very high noise level)
Paul Menage <[email protected]> Srivatsa Vaddagiri <[email protected]> Dhaval Giani <[email protected]> Vaidyanathan Srinivasan <[email protected]>
* Tuesday, Sept 4, 15h00 to 18h00 :
Pavel Emelianov <[email protected]> Paul Menage <[email protected]> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]> Cedric Le Goater <clg-NmTC/[email protected]>
= Namespace status
======================================================================
* sysv ipc
extend to posix mqueue.
. check that /dev/mqueue can be mounted multiple times . mqueue sysctls will need a fix :
fs.mqueue.queues_max fs.mqueue.msg_max fs.mqueue.msgsize_max
* uname namespace
considered complete.
what about being able to set the kernel version ?
* user
useful today to current container technologies (openvz, vserver)
uid checks should be replaced by (uid, userns) to complete integration with filesystems
security needs to be looked at
so is signal delivery
* pid namespace
in dev
signal handling completion underway
pid_t cleanups
. the purpose is to remove any explicit reference to
task->pid . keep ->pid in task struct only for performance . complex cleanups ones: af_unix credentials file locks timer stat
kthread cleanup
. replace kernel_thread() by the kthread API . change core kthread API to support signals . then nfs needs extra love. is someone working on it ?
do we need hierarchical levels ?
* net
in dev
veth is in dmiller's tree
sysfs cleanups underway in greg's tree
eric is working on a mininal patchset acceptable for netdev. will ask dmiller advice on the topic
ip isolation could be done with netfilter or security hooks
* device namespace
to do
we don't want to get rid of mknod() but we also want to limit the view of the devices in a container. one way to do this is through a device namespace which would only expose a 'white list' of devices when unshared. a possible 'white list' is :
/dev/null
/dev/full
/dev/zero
/dev/rtc
/dev/random
/dev/pts/* do we require a extra namespace for /dev/pts/* to handle its virtualization or can this be done directly in the device namespace ?
check that /dev/pts can be mounted multiple times.
* time
to do
required for C/R
will only make sense in a "closed" environment
the purpose is to keep the monotonic timers from expiring when you restart
* other possible namespace ?
rtc ? which is an isolation issue and also a sysctl issue
comment from eric :
a redesign of lsm, a la netfilter, could cover all isolation needs.
* namespace management
. entering
no consensus on how this should be done.
probably because the need is related to a container and not just namespaces. it should be solved with a container object and probably a subsystem.
serge's proposal of sys_hijack() is interesting but will require more study because, in UNIX, it's not natural for a child process to have 2 parents !
. extending clone to support more flags
new syscall proposal for a clone2(struct clone2_arg_struct* args)
* tests
. ltp for unit
. keep the integration tests in each container framework.
* Filesystems
. unprivilege mounts (not addressed)
merged
. multiple /sys mounts (in dev)
missing some bits (eric working on it) to decouple sysfs and kobjects
. multiple /proc mounts (to complete)
multiple mount done
to limit access to /proc files, use the user namespace checks ? for the contents of each file, use the current context to identify namespace
* Console
. a running getty should be covered by tty namespace . printk will require some support to be isolated.
= Task Container (from container dev plan)
======================================================================
* base features
hierarchical/virtualized containers
support vserver mgmnt of sub-containers
locking cleanup
control file API simplification
unified container including namespaces
the "container"/"task container" name is ambiguous and it should change to "control group"
* userpace RBCE to provide controls for
users
groups
pgrp
executable
* specific containers targeted:
split cpusets into
cpuset
memset
network
connect/bind/accept controller using iptables
controllers :
memory controller (see detail below)
cpu controller
Status:
- Extensions required to CFS core for supporting group-scheduling aspects are mostly there (in mainline)
Todo:
- Better SMP group-fairness
- Hard-limit cpu usage
- SCHED_FIFO like policy for groups
- Group priorities (?)
io controller (see detail below)
network flow id control
per-container OOM handler (userspace)
per-container swap
per-container disk I/O scheduling
per container memory reclaim
per container dirty page (write throttling) limit.
network rate limiting (outbound) based on container
* misc
User level APIS to identify the resource limits that is allowed to a job, for example, how much physical memory a process can use. This should seamlessly integrated with non-container environment as well (may be with ulimit).
Per container stats, like pages on active list, cpus usage, etc
= Resource Management (from container dev plan)
======================================================================
* memory controller
users and requirements:
1. The containers solution would need resource management (including memory control and per container swap files). Paul Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have all shown interest in the memory controller patches.
2. The memory controller can account for page cache as well, all people interested in limiting page cahce control, can theoratically put move all page cache hungry applications under the same container.
Planned enhancements to the memory controller
1. Improved shared page accounting
2. Improved statistics
3. Soft-limit memory usage
generic infrastructure work:
1. Enhancing containerstats
a. Working on per controller statistics b. Integrating taskstats with containerstats
2. CPU accounting framework
a. Migrate the accounting to be more precis
* cpu controller
users and requirements:
1. Virtualization solutions like containers and KVM need CPU control. KVM for example would like to have both limits and guarantees supported by a CPU controller, to control CPU allocation to a particular instance.
2. Workload management products would like to exploit this for providing guaranteed cpu bandwidth and also (hard/soft) limiting cpu usage.
work items
1. Fine-grained proportional-share fair-group scheduling.
2. More accurate SMP fairness
3. Hard limit
4. SCHED_FIFO type policy for groups
5. Improved statistics and debug facility for group scheduler
* io controller
users and requirements:
1. At a talk presented to the Linux Foundation (OSDL), the attendees showed interest in an IO controller to control IO bandwidth of various filesystem operations (backup, journalling, etc)
work items:
1. Proof of concept IO controller and community discussion/feedback
2. Development and Integration of the IO controller with containers
open issues
1. Automatic tagging/resource classification engine
= Checkpoint/Restart
======================================================================
* need to unified the freezer to reach a quiescence point
* overall strategy :
. checkpoint: in kernel
. restart : first recreate process tree then let each
process restart itself
* possible direction for C/R user api
. checkpoint/restart syscalls
. C/R file systems
solves the set id issue elegant but exposes too much the ABI
example :
.
|-- 0x00003002 | |-- 0x00003002 | | |-- attr | | |-- signal | | |-- signal.altstack | | |-- signal.pending | | |-- thread | | |-- thread.frame | | |-- timers | | |-- tls | | `-- wait.zombies | |-- aio | |-- attr | |-- fds | |-- ldt | |-- mem.segments | |-- numa | |-- process | |-- signal.action | |-- signal.pending | |-- sysv.semadj | |-- sysv.shmcount | `-- thread.list |-- af_inet_listening |-- af_inet_orphan_count |-- af_inet_orphan_data |-- af_inet_orphan_info |-- files | |-- 0 | |-- 1 | |-- 10137663680 | |-- 1014250cdc0 | |-- 2 | `-- stdios |-- sysv.msq |-- sysv.sem
`-- sysv.shm
* memory C/R
critical for performance
per-container swapfile ?
* subsystem C/R API.
keep it on the side for the moment <subsys>_cr.c to identify the needs of each subsystem before asking the maintainer's comments
possible cr_ops in some objects (like for network protocols) but also ops 'a la' virt_ops to prepare for different C/R strategy :
brutal, incremental, live migration
* setting id back to what they where
possible global syscall to set ids of pid,ipc,pts. else use the C/R fs
* statefile format
no big issues. let's pick one.
* optimization
parallel C/R
A message that is not in the archive (sent off-list or not yet received): 46DEB80B.2040106-NmTC/[email protected]
Quoting Cedric Le Goater (clg-NmTC/[email protected]):
...
Quoted 21 lines
Container mini-summit notes v0.01
======================================================================
Held at Universty Arm's, Cambridge, UK
on 3rd of September, 9h00 to 16h00 :
Kir Kolyshkin <[email protected]>
Pavel Emelianov <[email protected]> Masahiko Takahashi <masahiko-de/[email protected]> Oren Laadan <[email protected]> Cedric Le Goater <clg-NmTC/[email protected]> James ??? (Google)
??? (NTT)On the phone (skype with very high noise level)
Srivatsa Vaddagiri <[email protected]> Paul Menage <[email protected]>
Quoted 6 lines
on 4th of September, 15h00 to 18h00 :
Pavel Emelianov <[email protected]> Cedric Le Goater <clg-NmTC/[email protected]> Paul Menage <[email protected]> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]>
Boy I really missed out didn't I...
Quoted 13 lines
= Namespace status
======================================================================* ipc
extend to posix mqueue.
check that/dev/mqueuecan be mounted multiple times mqueue sysctls need a fix
fs.mqueue.queues_max fs.mqueue.msg_max fs.mqueue.msgsize_max
Quoted 3 lines
* uname namespace
considered complete but what about setting the kernel version ?
I still say yuck. The kernel won't properly emulate the other kernel version, so lying seems a horrible thing to do. If you want a vm with another kernel, use qemu/kvm/xen.
Quoted 21 lines
* user (to complete)
useful today to current container technologies (userns, vserver) (uid, userns) checks must be completed to be integrate
with filesystems
security needs to be looked at
so is signal delivery* pid namespace (in dev)
stable.
Current tasks are :
. signal handling
.pid_tcleanupsremove any reference to task->pid keep ->pid in task struct only for performance complex ones: af_unix credentials file locks timer stat
Quoted 4 lines
. kthread cleanup
replacekernel_thread()by the kthread API change core kthread API to support signals nfs needs extra love. is someone working on it ?
REALLY would like to hear if anyone is.
Quoted 11 lines
do we need hierarchical levels ?
* net (in dev)
veth is in dmiller's tree
sysfs cleanups underway in greg's tree
eric is working on a mininal patchset acceptable for netdev. will ask dmiller advice on the topicip isolation could be done with netfilter or security hooks
And has by several people :)
Quoted 14 lines
* device namespace (to do)
we don't want to get rid of
mknod()but we also want to limit the view of the devices in a container. one way to do this is through a device namespace which would only expose a 'white list' of devices when unshared. a possible white list is :/dev/null /dev/full /dev/zero /dev/rtc /dev/random /dev/pts/*
Quoted 2 lines
do we require a extra namespace for
/dev/pts/*to handle its virtualization or can this be done directly in a device namespace ?
How the hell does a separate device ns protect you from mknod?
Quoted 1 line
check that
/dev/ptscan be mounted multiple times.
I previously sent a patch to implement CLONE_NEWPTS, which just intercepts /dev/pts/ access without multiple mounts. I think that, plus access to child containers' devpts mounts through the nsproxy container subsystem, would be a very simple and useful way to implement this.
Then there is also the array of console devices to turn into a namespace.
And finally there is the issue of actualy virtual devices. So I want to start up a virtual server, and I want it's serial port to be connected to a virtual tty which buffers the last page or few pages of data so I can connect to the server and see what's going on. I'm not too savvy here - can we do that now, maybe just through a fake serial device?
Quoted 5 lines
* time (to do)
keep the monotonic timers from expiring when you restart required for C/R but will only make sense in "close" environment
Quoted 7 lines
* other possible namespace ?
rtc ? which an isolation problem and also a sysctl issue
comment from eric :
redesign of lsm (a la netfilter) could cover all isolation needs.
I'd like to talk to Eric and find out more precisely what he is thinking.
Note that one issue with using lsm is that it will purely control access, not resource ids. So container 1 may not have access to /dev/pts/2, which at least is safe, but it'll see everyone's devices under devpts.
But yes, this is why I've been saying that userns is useful at the moment. You should be able to write simple selinux policies to fully container a userns.
Quoted 30 lines
* namespace management
. entering
no consensus on how this should be should be done.probably because the need is related to a container and not just namespaces. it should be solved with a container object and probably a subsystem.
serge's proposal of
sys_hijack()is interesting but will require more study because, in UNIX, it's not natural for a child process to have 2 parents !. extending clone to support more flags
new syscall proposal for a
clone2(struct clone2_arg_struct* args)* tests
. ltp for unit
. keep the integration tests in each container framework.* Filesystems
. unprivilege mounts (not addressed)
merged
. multiple /sys mounts (in dev)
Quoted 7 lines
missing some bits (eric working on it)
. multiple /proc mounts (to complete)
multiple mount done
to limit access to /proc files, use the user namespace checks ? for the contents of each file, use the current context
Just might work...
It kind of looks like having a 'initial' userns which is special-cased, but I assume the idea is that /proc/ files would be created at boot in the initial userns, so straightforward checks for userns equivalence between the writing process and the file would suffice.
Quoted 3 lines
* Console
. a running getty should be covered by tty namespace
I'm not understanding this line.
Is there a fully tty namespace design that was discussed, which this is summarizing?
Quoted 117 lines
. printk will require some support to be isolated.
= Task Container (from container dev plan)
======================================================================* base features
hierarchical/virtualized containers
support vserver mgmnt of sub-containers
locking cleanup
control file API simplification
unified container including namespaces* userpace RBCE to provide controls for
users
groups
pgrp
executable* specific containers targeted:
split cpusets into
cpuset
memset
network
connect/bind/accept controller using iptablescontrollers :
memory controller (see detail below)
cpu controller (see detailbelow)
io controller (see detail below)
network flow id control
per-container OOM handler (userspace) per-container swap
per-container disk I/O scheduling
per container memory reclaim
per container dirty page (write throttling) limit. network rate limiting (outbound) based on container* misc
User level APIS to identify the resource limits that is allowed to a job, for example, how much physical memory a process can use. This should seamlessly integrated with non-container environment as well (may be with ulimit).
Per container stats, like pages on active list, cpus usage, etc
= Resource Management (from container dev plan)
======================================================================* memory controller
users and requirements:
1. The containers solution would need resource management (including memory control and per container swap files). Paul Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have all shown interest in the memory controller patches.
2. The memory controller can account for page cache as well, all people interested in limiting page cahce control, can theoratically put move all page cache hungry applications under the same container.
Planned enhancements to the memory controller
1. Improved shared page accounting
2. Improved statistics
3. Soft-limit memory usagegeneric infrastructure work:
1. Enhancing containerstats
a. Working on per controller statistics b. Integrating taskstats with containerstats
2. CPU accounting framework
a. Migrate the accounting to be more precis* cpu controller
users and requirements:
1. Virtualization solutions like containers and KVM need CPU control. KVM for example would like to have both limits and guarantees supported by a CPU controller, to control CPU allocation to a particular instance.
2. Workload management products would like to exploit this for providing guaranteed cpu bandwidth and also (hard/soft) limiting cpu usage.work items
1. Fine-grained proportional-share fair-group scheduling.
2. More accurate SMP fairness
3. Hard limit
4.SCHED_FIFOtype policy for groups
5. Improved statistics and debug facility for group scheduler* io controller
users and requirements:
1. At a talk presented to the Linux Foundation (OSDL), the attendees showed interest in an IO controller to control IO bandwidth of various filesystem operations (backup, journalling, etc)
work items:
1. Proof of concept IO controller and community discussion/feedback
2. Development and Integration of the IO controller with containersopen issues
1. Automatic tagging/resource classification engine= Checkpoint/Restart
======================================================================
Quoted 6 lines
* need to unified the freezer to reach a quiescence point
* overall strategy :
. checkpoint: in kernel
. restart : first recreate process tree then let each
process restart itself
The data transfer method is pretty important to sort out, so that people can start writing patches to c/r various resources.
I know you've discussed having some sort of 'crfs'. I gather you would then do a checkpoint by doing something like
cp /mnt/crfs/vserver1 /opt/checkpoints/vserver1/`date +%F`/`date +%R`
I actually like that, so long as it's that, and not actually having the cr subsystem write the data straight to a regular file.
Another option would be to use relayfs for the checkpoint data dump, since checkpoint and restart will be assymetrical in nature anyway.
Quoted 45 lines
* possible direction for C/R user api
. checkpoint/restart syscalls
. C/R file systems
solves the set id issue elegant but exposes too much the ABIexample :
.
|-- 0x00003002 | |-- 0x00003002 | | |-- attr | | |-- signal | | |-- signal.altstack | | |-- signal.pending | | |-- thread | | |-- thread.frame | | |-- timers | | |-- tls | | `-- wait.zombies | |-- aio | |-- attr | |-- fds | |-- ldt | |-- mem.segments | |-- numa | |-- process | |-- signal.action | |-- signal.pending | |-- sysv.semadj | |-- sysv.shmcount | `-- thread.list |-- af_inet_listening |-- af_inet_orphan_count |-- af_inet_orphan_data |-- af_inet_orphan_info |-- files | |-- 0 | |-- 1 | |-- 10137663680 | |-- 1014250cdc0 | |-- 2 | `-- stdios |-- sysv.msq |-- sysv.sem
Quoted 6 lines
`-- sysv.shm
* memory C/R
critical for performance
per-container swapfile ?
Absolutely. per-container swapfile, and memory checkpoint forces write to swap and writes the pte data out to relayfs or a crfs file ready for copy.
One tricky issue then is that we really want to do a copy-on-write copy of the swapfile. We don't want to wait for the swapfile to be fully copied before restarting the process. Are any of the fancy new filesystems implementing that? Or are we gonna be using zfs over fuse for our per-container swapfiles? :)
Quoted 4 lines
* subsystem C/R API.
keep it on the side for the moment <subsys>_cr.c to identify the needs of each subsystem before asking the maintainer's comments
It would be great if we could provide a data write/read api so these subsystems can all be written without worrying about which method we'll use for dumping checkpoint data and pushing restart data.
Quoted 3 lines
possible
cr_opsin some objects (like for network protocols) but also ops 'a la'virt_opsto prepare for different C/R strategy :
brutal, incremental, live migration
Hmm.
Quoted 18 lines
* setting id back to what they where
possible global syscall to set ids of pid,ipc,pts. else use the C/R fs
* statefile format
no big issues. let's pick one.
* optimization
parallel C/R
A message that is not in the archive (sent off-list or not yet received): 20070905143849.GH1403-6s5zFf/[email protected]
Quoted 4 lines
Quoted 4 lines
* Console
. a running getty should be covered by tty namespace
to log onto a container, we could simply run a getty in the guest with the guest end of the tty in a pts namespace.
C.
kthread API cleanups (Re: Container mini-summit notes)
In reply to a message not in the archiveQuoted 3 lines
Quoted 3 lines
. kthread cleanup
replacekernel_thread()by the kthread API change core kthread API to support signals
no one on that task.
I'm pretty sure hch would be happy to help someone on it.
Quoted 3 lines
Quoted 1 line
nfs needs extra love. is someone working on it ?
REALLY would like to hear if anyone is.
No one apparently. Trond is not against it but he badly needs the above changes.
Cheers,
C.
Quoted 193 lines · Cedric Le Goater
Held at De Vere Universty Arms Hotel, Cambridge, UK
* Monday, Sept 3, 9h00 to 16h00 :
Kir Kolyshkin <[email protected]>
Pavel Emelianov <[email protected]> Masahiko Takahashi <masahiko-de/[email protected]> Oren Laadan <[email protected]>
James Youngman <[email protected]> ??? (NTT)
Cedric Le Goater <clg-NmTC/[email protected]>On the phone (skype with very high noise level)
Paul Menage <[email protected]> Srivatsa Vaddagiri <[email protected]> Dhaval Giani <[email protected]> Vaidyanathan Srinivasan <[email protected]>
* Tuesday, Sept 4, 15h00 to 18h00 :
Pavel Emelianov <[email protected]> Paul Menage <[email protected]> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]> Cedric Le Goater <clg-NmTC/[email protected]>
= Namespace status ======================================================================
* sysv ipc
extend to posix mqueue.
. check that/dev/mqueuecan be mounted multiple times . mqueue sysctls will need a fix :
fs.mqueue.queues_max fs.mqueue.msg_max fs.mqueue.msgsize_max* uname namespace
considered complete.
what about being able to set the kernel version ?
* user
useful today to current container technologies (openvz, vserver)
uid checks should be replaced by (uid, userns) to complete integration with filesystems
security needs to be looked at
so is signal delivery* pid namespace
in dev
signal handling completion underway
pid_tcleanups
. the purpose is to remove any explicit reference totask->pid . keep ->pid in task struct only for performance . complex cleanups ones: af_unix credentials file locks timer statkthread cleanup
. replacekernel_thread()by the kthread API . change core kthread API to support signals . then nfs needs extra love. is someone working on it ?do we need hierarchical levels ?
* net
in dev
veth is in dmiller's tree
sysfs cleanups underway in greg's tree
eric is working on a mininal patchset acceptable for netdev. will ask dmiller advice on the topicip isolation could be done with netfilter or security hooks
* device namespace
to do
we don't want to get rid of
mknod()but we also want to limit the view of the devices in a container. one way to do this is through a device namespace which would only expose a 'white list' of devices when unshared. a possible 'white list' is :/dev/null /dev/full /dev/zero /dev/rtc /dev/random /dev/pts/*do we require a extra namespace for
/dev/pts/*to handle its virtualization or can this be done directly in the device namespace ?check that
/dev/ptscan be mounted multiple times.* time
to do
required for C/R
will only make sense in a "closed" environment
the purpose is to keep the monotonic timers from expiring when you restart* other possible namespace ?
rtc ? which is an isolation issue and also a sysctl issue
comment from eric :
a redesign of lsm, a la netfilter, could cover all isolation needs.* namespace management
. entering
no consensus on how this should be done.
probably because the need is related to a container and not just namespaces. it should be solved with a container object and probably a subsystem.
serge's proposal of
sys_hijack()is interesting but will require more study because, in UNIX, it's not natural for a child process to have 2 parents !. extending clone to support more flags
new syscall proposal for a
clone2(struct clone2_arg_struct* args)* tests
. ltp for unit
. keep the integration tests in each container framework.* Filesystems
. unprivilege mounts (not addressed)
merged
. multiple /sys mounts (in dev)
missing some bits (eric working on it) to decouple sysfs and kobjects
. multiple /proc mounts (to complete)
multiple mount done
to limit access to /proc files, use the user namespace checks ? for the contents of each file, use the current context to identify namespace* Console
. a running getty should be covered by tty namespace . printk will require some support to be isolated.
= Task Container (from container dev plan)
======================================================================* base features
hierarchical/virtualized containers
support vserver mgmnt of sub-containers
locking cleanup
control file API simplification
unified container including namespacesthe "container"/"task container" name is ambiguous and it should change to "control group"
* userpace RBCE to provide controls for
users
groups
pgrp
executable
Quoted 7 lines
* specific containers targeted:
split cpusets into
cpuset
memset
network
connect/bind/accept controller using iptables
Just a comment here. The hooks needed for that are exactly the same as the security hooks, IMHO, iptables is not right subsystem to use to catch socket calls.
Quoted 18 lines
controllers :
memory controller (see detail below)
cpu controller
Status:
- Extensions required to CFS core for supporting group-scheduling aspects are mostly there (in mainline)Todo:
- Better SMP group-fairness
- Hard-limit cpu usage
-SCHED_FIFOlike policy for groups
- Group priorities (?)
Quoted 3 lines
io controller (see detail below)
network flow id control
I'm not sure I get that.
.
Quoted 5 lines
per-container OOM handler (userspace)
per-container swap
Quoted 7 lines
per-container disk I/O scheduling
per container memory reclaim
per container dirty page (write throttling) limit.
network rate limiting (outbound) based on container
I am not sure I get that too.
As far as I understand, if the container has its own network with a network namespace, then the rate limiting already exists with tc for outgoing __and__ incoming traffic. If the container has a simple IP isolation, the rate limiting can be set with tc again but only for outgoing traffic. If the container share the network with other container, I understand the reason for a network flow id. But in this case, how to handle incoming traffic ?
IHMO, rate limiting should be handled in conjunction with a network namespace.
Quoted 157 lines
* misc
User level APIS to identify the resource limits that is allowed to a job, for example, how much physical memory a process can use. This should seamlessly integrated with non-container environment as well (may be with ulimit).
Per container stats, like pages on active list, cpus usage, etc
= Resource Management (from container dev plan)
======================================================================* memory controller
users and requirements:
1. The containers solution would need resource management (including memory control and per container swap files). Paul Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have all shown interest in the memory controller patches.
2. The memory controller can account for page cache as well, all people interested in limiting page cahce control, can theoratically put move all page cache hungry applications under the same container.
Planned enhancements to the memory controller
1. Improved shared page accounting
2. Improved statistics
3. Soft-limit memory usagegeneric infrastructure work:
1. Enhancing containerstats
a. Working on per controller statistics b. Integrating taskstats with containerstats
2. CPU accounting framework
a. Migrate the accounting to be more precis* cpu controller
users and requirements:
1. Virtualization solutions like containers and KVM need CPU control. KVM for example would like to have both limits and guarantees supported by a CPU controller, to control CPU allocation to a particular instance.
2. Workload management products would like to exploit this for providing guaranteed cpu bandwidth and also (hard/soft) limiting cpu usage.work items
1. Fine-grained proportional-share fair-group scheduling.
2. More accurate SMP fairness
3. Hard limit
4.SCHED_FIFOtype policy for groups
5. Improved statistics and debug facility for group scheduler* io controller
users and requirements:
1. At a talk presented to the Linux Foundation (OSDL), the attendees showed interest in an IO controller to control IO bandwidth of various filesystem operations (backup, journalling, etc)
work items:
1. Proof of concept IO controller and community discussion/feedback
2. Development and Integration of the IO controller with containersopen issues
1. Automatic tagging/resource classification engine= Checkpoint/Restart
======================================================================* need to unified the freezer to reach a quiescence point
* overall strategy :
. checkpoint: in kernel
. restart : first recreate process tree then let each
process restart itself* possible direction for C/R user api
. checkpoint/restart syscalls
. C/R file systems
solves the set id issue elegant but exposes too much the ABIexample :
.
|-- 0x00003002 | |-- 0x00003002 | | |-- attr | | |-- signal | | |-- signal.altstack | | |-- signal.pending | | |-- thread | | |-- thread.frame | | |-- timers | | |-- tls | | `-- wait.zombies | |-- aio | |-- attr | |-- fds | |-- ldt | |-- mem.segments | |-- numa | |-- process | |-- signal.action | |-- signal.pending | |-- sysv.semadj | |-- sysv.shmcount | `-- thread.list |-- af_inet_listening |-- af_inet_orphan_count |-- af_inet_orphan_data |-- af_inet_orphan_info |-- files | |-- 0 | |-- 1 | |-- 10137663680 | |-- 1014250cdc0 | |-- 2 | `-- stdios |-- sysv.msq |-- sysv.sem`-- sysv.shm
* memory C/R
critical for performance
per-container swapfile ?* subsystem C/R API.
keep it on the side for the moment <subsys>_cr.c to identify the needs of each subsystem before asking the maintainer's comments
possible
cr_opsin some objects (like for network protocols) but also ops 'a la'virt_opsto prepare for different C/R strategy :
brutal, incremental, live migration* setting id back to what they where
possible global syscall to set ids of pid,ipc,pts. else use the C/R fs
* statefile format
no big issues. let's pick one.
* optimization
parallel C/R
Quoted 202 lines · Cedric Le Goater
Held at De Vere Universty Arms Hotel, Cambridge, UK
* Monday, Sept 3, 9h00 to 16h00 :
Kir Kolyshkin <[email protected]>
Pavel Emelianov <[email protected]> Masahiko Takahashi <masahiko-de/[email protected]> Oren Laadan <[email protected]>
James Youngman <[email protected]> ??? (NTT)
Cedric Le Goater <clg-NmTC/[email protected]>On the phone (skype with very high noise level)
Paul Menage <[email protected]> Srivatsa Vaddagiri <[email protected]> Dhaval Giani <[email protected]> Vaidyanathan Srinivasan <[email protected]>
* Tuesday, Sept 4, 15h00 to 18h00 :
Pavel Emelianov <[email protected]> Paul Menage <[email protected]> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/[email protected]> Cedric Le Goater <clg-NmTC/[email protected]>
= Namespace status
======================================================================* sysv ipc
extend to posix mqueue.
. check that/dev/mqueuecan be mounted multiple times . mqueue sysctls will need a fix :
fs.mqueue.queues_max fs.mqueue.msg_max fs.mqueue.msgsize_max* uname namespace
considered complete.
what about being able to set the kernel version ?
* user
useful today to current container technologies (openvz, vserver)
uid checks should be replaced by (uid, userns) to complete integration with filesystems
security needs to be looked at
so is signal delivery* pid namespace
in dev
signal handling completion underway
pid_tcleanups
. the purpose is to remove any explicit reference totask->pid . keep ->pid in task struct only for performance . complex cleanups ones: af_unix credentials file locks timer statkthread cleanup
. replacekernel_thread()by the kthread API . change core kthread API to support signals . then nfs needs extra love. is someone working on it ?do we need hierarchical levels ?
* net
in dev
veth is in dmiller's tree
sysfs cleanups underway in greg's tree
eric is working on a mininal patchset acceptable for netdev. will ask dmiller advice on the topicip isolation could be done with netfilter or security hooks
* device namespace
to do
we don't want to get rid of
mknod()but we also want to limit the view of the devices in a container. one way to do this is through a device namespace which would only expose a 'white list' of devices when unshared. a possible 'white list' is :/dev/null /dev/full /dev/zero /dev/rtc /dev/random /dev/pts/*do we require a extra namespace for
/dev/pts/*to handle its virtualization or can this be done directly in the device namespace ?check that
/dev/ptscan be mounted multiple times.* time
to do
required for C/R
will only make sense in a "closed" environment
the purpose is to keep the monotonic timers from expiring when you restart* other possible namespace ?
rtc ? which is an isolation issue and also a sysctl issue
comment from eric :
a redesign of lsm, a la netfilter, could cover all isolation needs.* namespace management
. entering
no consensus on how this should be done.
probably because the need is related to a container and not just namespaces. it should be solved with a container object and probably a subsystem.
serge's proposal of
sys_hijack()is interesting but will require more study because, in UNIX, it's not natural for a child process to have 2 parents !. extending clone to support more flags
new syscall proposal for a
clone2(struct clone2_arg_struct* args)* tests
. ltp for unit
. keep the integration tests in each container framework.* Filesystems
. unprivilege mounts (not addressed)
merged
. multiple /sys mounts (in dev)
missing some bits (eric working on it) to decouple sysfs and kobjects
. multiple /proc mounts (to complete)
multiple mount done
to limit access to /proc files, use the user namespace checks ? for the contents of each file, use the current context to identify namespace* Console
. a running getty should be covered by tty namespace . printk will require some support to be isolated.
= Task Container (from container dev plan)
======================================================================* base features
hierarchical/virtualized containers
support vserver mgmnt of sub-containers
locking cleanup
control file API simplification
unified container including namespacesthe "container"/"task container" name is ambiguous and it should change to "control group"
* userpace RBCE to provide controls for
users
groups
pgrp
executable* specific containers targeted:
split cpusets into
cpuset
memset
network
connect/bind/accept controller using iptables
Quoted 4 lines
controllers :
memory controller (see detail below)
The status is that the patches for memory control are in -mm. We need to stabilize the patches, look at reducing the overhead of double LRU for root containers. We need to explore the double LRU approach as a longer term solution.
Quoted 190 lines
cpu controller
Status:
- Extensions required to CFS core for supporting group-scheduling aspects are mostly there (in mainline)Todo:
- Better SMP group-fairness
- Hard-limit cpu usage
-SCHED_FIFOlike policy for groups
- Group priorities (?)io controller (see detail below)
network flow id control
per-container OOM handler (userspace)
per-container swap
per-container disk I/O scheduling
per container memory reclaim
per container dirty page (write throttling) limit.
network rate limiting (outbound) based on container
* misc
User level APIS to identify the resource limits that is allowed to a job, for example, how much physical memory a process can use. This should seamlessly integrated with non-container environment as well (may be with ulimit).
Per container stats, like pages on active list, cpus usage, etc
= Resource Management (from container dev plan)
======================================================================* memory controller
users and requirements:
1. The containers solution would need resource management (including memory control and per container swap files). Paul Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have all shown interest in the memory controller patches.
2. The memory controller can account for page cache as well, all people interested in limiting page cahce control, can theoratically put move all page cache hungry applications under the same container.
Planned enhancements to the memory controller
1. Improved shared page accounting
2. Improved statistics
3. Soft-limit memory usagegeneric infrastructure work:
1. Enhancing containerstats
a. Working on per controller statistics b. Integrating taskstats with containerstats
2. CPU accounting framework
a. Migrate the accounting to be more precis* cpu controller
users and requirements:
1. Virtualization solutions like containers and KVM need CPU control. KVM for example would like to have both limits and guarantees supported by a CPU controller, to control CPU allocation to a particular instance.
2. Workload management products would like to exploit this for providing guaranteed cpu bandwidth and also (hard/soft) limiting cpu usage.work items
1. Fine-grained proportional-share fair-group scheduling.
2. More accurate SMP fairness
3. Hard limit
4.SCHED_FIFOtype policy for groups
5. Improved statistics and debug facility for group scheduler* io controller
users and requirements:
1. At a talk presented to the Linux Foundation (OSDL), the attendees showed interest in an IO controller to control IO bandwidth of various filesystem operations (backup, journalling, etc)
work items:
1. Proof of concept IO controller and community discussion/feedback
2. Development and Integration of the IO controller with containersopen issues
1. Automatic tagging/resource classification engine= Checkpoint/Restart
======================================================================* need to unified the freezer to reach a quiescence point
* overall strategy :
. checkpoint: in kernel
. restart : first recreate process tree then let each
process restart itself* possible direction for C/R user api
. checkpoint/restart syscalls
. C/R file systems
solves the set id issue elegant but exposes too much the ABIexample :
.
|-- 0x00003002 | |-- 0x00003002 | | |-- attr | | |-- signal | | |-- signal.altstack | | |-- signal.pending | | |-- thread | | |-- thread.frame | | |-- timers | | |-- tls | | `-- wait.zombies | |-- aio | |-- attr | |-- fds | |-- ldt | |-- mem.segments | |-- numa | |-- process | |-- signal.action | |-- signal.pending | |-- sysv.semadj | |-- sysv.shmcount | `-- thread.list |-- af_inet_listening |-- af_inet_orphan_count |-- af_inet_orphan_data |-- af_inet_orphan_info |-- files | |-- 0 | |-- 1 | |-- 10137663680 | |-- 1014250cdc0 | |-- 2 | `-- stdios |-- sysv.msq |-- sysv.sem`-- sysv.shm
* memory C/R
critical for performance
per-container swapfile ?* subsystem C/R API.
keep it on the side for the moment <subsys>_cr.c to identify the needs of each subsystem before asking the maintainer's comments
possible
cr_opsin some objects (like for network protocols) but also ops 'a la'virt_opsto prepare for different C/R strategy :
brutal, incremental, live migration* setting id back to what they where
possible global syscall to set ids of pid,ipc,pts. else use the C/R fs
* statefile format
no big issues. let's pick one.
* optimization
parallel C/R
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers
Signature · 4 lines
Warm Regards, Balbir Singh Linux Technology Center IBM, ISTL
Quoted 10 lines · Cedric Le Goater
Held at De Vere Universty Arms Hotel, Cambridge, UK
* Monday, Sept 3, 9h00 to 16h00 :
Kir Kolyshkin <[email protected]>
Pavel Emelianov <[email protected]> Masahiko Takahashi <masahiko-de/[email protected]> Oren Laadan <[email protected]>
James Youngman <[email protected]> ??? (NTT)
Fernando Luis Vazquez Cao (NTT)
It took me a while. Sorry !
C.