container userspace tools
Subsystems (2)
Hi all,
I saw different emails from people asking for user space tools for the containers, so I made a library and a set of command line tools. This component is called "lxc" which stands for "LinuX Container".
The package can be found in the download area of:
http://sourceforge.net/projects/lxc
There is a cvs repository at:
http://lxc.cvs.sourceforge.net/lxc
The API is exported in the "lxc.h" header file.
It is not perfect, but it is very light and I hope that will make the life easier for the kernel hackers. It can be used to create virtual private server and to "boot" a minimal distro like "debian minimal".
It takes into account the control groups and the checkpoint/restart API ... in a very optimistic way :)
It should be used with the kernel coming from the git tree, git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.git
But the latest -mm should be ok, some commands will just fail (eg. checkpoint/restart).
How to use it ?
--------------------
(for checkpointr/restart see the end of the explanation for this very specific, experimental use case).
A container should be created before with (or without) a configuration file.
The configuration file has the following format (all fields are optional):
# set the hostname for the container lxc.utsname = myhostname lxc.network.type = veth lxc.network.link = br0 lxc.network.flags = up lxc.network.name = eth0 lxc.network.hwaddr = 4a:49:43:49:79:bf lxc.network.ipv4 = 1.2.3.5/24 lxc.network.ipv6 = 2003:db8:1:0:214:1234:fe0b:3597 lxc.mount = fstab lxc.rootfs = rootfs lxc.cgroup.cpuset.cpus = 1 lxc.cgroup.cpu.shares = 512
This is an example, but the configuration file can be much more simple or complex. In this example the configuration assume the host is configured with a bridge, the liblxc does not configure anything not related to the container, so it is up to the user/administrator to setup a bridge in this case. If the bridge is a problem, different network configurations exist to have a container to communicate with the outside world. The network can be specified with several ip addresses, or either if there is only one physical network device, it is possible to specify multiple network devices for the container. To do that just replicate the network sections as follow:
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.name = eth0
lxc.network.ipv4 = 192.168.9.100/24
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
lxc.network.name = eth1
lxc.network.ipv4 = 192.168.10.100/24
The more there are informations in the configuration file, the more the container is isolated. eg. if the network section is removed from the configuration file, the network won't be virtualized, the same for the ustname.
Step 1 : create the container named 'foo' with the configuration file 'lxc.conf'
lxc-create -n foo -f lxc.conf
If no configuration file is specified, the container will be created with default configuration, that is to say, pid namespace, mount namespace and ipc namespace. The other resources, network, file system, utsname, won't be virtualized.
Step 2 : launch a command inside
lxc-execute -n foo /bin/bash
or
lxc-start -n foo /bin/bash
The differences between lxc-execute and lxc-start are, lxc-execute creates an intermediate process for the pid namespace in order to support daemons and it mounts /proc and /sys.
Step 3 : Stop the container (from outside of the container)
lxc-stop -n foo
Step 4 : Destroy the container
lxc-destroy -n foo
ps: it is not mandatory to destroy the container, you can keep the configuration and execute/start the command again. That means after a reboot, the container configuration is still available.
These are the minimal steps to use a container, but there are some other useful commands:
lxc-freeze / lxc-unfreeze => stop / resume all processes running
inside the container
lxc-monitor => follow the different states of the
container
lxc-wait => wait for a specific state
before exiting (useful for scripting)
lxc-ps => show all non-virtualized pids
of a container
lxc-info => show the state of the container
lxc-cgroup => get/set cgroup info for this
container
Checkpoint/Restart:
--------------------------
The checkpoint / restart is currently *very* experimental, and takes
into account only one task. For this reason the **lxc-start** command *must* be used.
The easiest way is specifying a container with the minimal isolation
for the minimal CR :)
lxc-create -n ckpt
lxc-start -n ckpt ./myverysimpleprogram
lxc-checkpoint -n ckpt -s > myckptfile
lxc-restart -n ckpt < myckptfile
ps: you have just to create one time the container.
The lxc-checkpoint command will do freeze - checkpoint - unfreeze, if the '-s' option is set, the container is stopped after the checkpoint.
I hope that will make people to have fun with the containers and the new kernel features.
Thanks.
-- Daniel
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 48F5EBF3.1070204-NmTC/[email protected]
Dear Daniel,
I have tested the container tools in my VM. It runs very well. But I can not see any highlight show via container. I believe that two group of user's processes (with same uid) can be isolated via container when both of them access files or are enforced with different quota. But I don't know how to get the functionalities run.
Any help?
Thanks in advance.
Quoted 14 lines · Daniel Lezcano
Quoted 4 lines · Ian jonhson
Quoted 4 lines
The problem is related to the kernel version you have and a missing functionality.
Quoted 8 lines
Quoted 4 lines
You should use the kernel coming from:
git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.gitIt needs not to set "Container support" if I run with above kernel, right?
I have downloaded the source codes.
Quoted 6 lines
The container support will enable at the compile time the different features in the kernel. I added this kernel option because I was tired to activate the different features one by one as described in the README file. If you don't specify this option, the container code will be there in the kernel but the not active. So you have to set it.
OK, I have activated the "container support" step by step according to README, but I met a mismatch setting:
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810250406s4f1c9337ha2bccc10faed1948-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 5 lines · Ian jonhson
Dear Daniel,
I have tested the container tools in my VM. It runs very well. But I can not see any highlight show via container.
The container will be more or less isolated depending of what you specify in the configuration file.
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a new sshd server.
In the other side, the cgroup are tied with the container, so you can freeze/unfreeze all processes belonging to the container, change the priority or assign an amount of physical memory to be used by the container.
Quoted 5 lines
I believe that two group of
user's processes (with same uid) can be isolated via container when both of them access files or are enforced with different quota. But I don't know how to get the functionalities run.
Allowing to assign quota per container is a good idea, but I don't think it is supported by the kernel right now. Perhaps there is a trick to do that but I don't know it :)
The rootfs option allows you to specify the root file system to be used by the container, so if you specify it, your container will be chrooted inside. This feature is at a very early stage and will be improved in the future, allowing to example to specify a iso image of a file system tree and make use of it.
There are two contributions which are good examples on how to setup a container, I added them to:
http://sourceforge.net/projects/lxc/
The first one is a chroot of a sshd server and the second one is a minimalist debian showing a full distro booting.
Quoted 23 lines
Any help?
Thanks in advance.
Quoted 8 lines · Daniel Lezcano
Quoted 7 lines · Ian jonhson
Quoted 7 lines
The problem is related to the kernel version you have and a missing functionality.
You should use the kernel coming from:
git://git.kernel.org/pub/scm/linux/kernel/git/daveh/linux-2.6-lxc.gitQuoted 9 lines
Quoted 3 lines
It needs not to set "Container support" if I run with above kernel, right?
I have downloaded the source codes.The container support will enable at the compile time the different features in the kernel. I added this kernel option because I was tired to activate the different features one by one as described in the README file. If you don't specify this option, the container code will be there in the kernel but the not active. So you have to set it.
Quoted 2 lines
OK, I have activated the "container support" step by step according to README, but I met a mismatch setting:
It looks like there is a piece missing at the end of the email.
A message that is not in the archive (sent off-list or not yet received): 49031A54.70806-NmTC/[email protected]
Quoted 3 lines
The container will be more or less isolated depending of what you specify in the configuration file.
yes
Quoted 5 lines
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a new sshd server.
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points? I have looked through the whole configuration example in README and fond the following settings(which are not tried in my test because I don't understand
them very well)
* lxc.mount: is it to mount a file (or volume) so that all the
processes in this container can
access the content in file?
* lxc.utsname: I don't know what exact functionlity of this setting is.
* lxc.network.xxx: are there other options expect
"link","hwaddr","ipv4" and "ipv6"?
* lxc.rootfs:for application writing data in it?
I am now looking into the codes and find some information about what the container can do maximally. It seems all the configuration settings are binding with a container and all the schedules are based on container unit. Configuration settings only can be setup before container creation and can not be changed in container lifecycle, right?
Quoted 4 lines
In the other side, the cgroup are tied with the container, so you can freeze/unfreeze all processes belonging to the container, change the priority or assign an amount of physical memory to be used by the container.
In my brain, the cgroup is mainly used in multiple CPUs. In traditional single CPU machine, can the container separate or determine how much CPU cycle are used by its processes now? Also, administrator has to configure cgroup before it takes action. I have no idea whether both of cgroup and container are totally integrate with each other, or both of them have to be handle in some cases.
Quoted 4 lines
Allowing to assign quota per container is a good idea, but I don't think it is supported by the kernel right now. Perhaps there is a trick to do that but I don't know it :)
I would like to do this part of job. Also, I need to control several groups of processes (belonged to same user) in the same time, isolating them, enforced them with given resource quota, and freeze/unfreeze some of them.
BTW, as for checkpointing of container, is it easy to checkpoint/restart given group of processes in above example?
Quoted 6 lines
The rootfs option allows you to specify the root file system to be used by the container, so if you specify it, your container will be chrooted inside. This feature is at a very early stage and will be improved in the future, allowing to example to specify a iso image of a file system tree and make use of it.
Good. what kinds of rootfs are supported now? I fond there are a debian file in sourceforge, is it a rootfs image?
Quoted 8 lines
There are two contributions which are good examples on how to setup a container, I added them to:
http://sourceforge.net/projects/lxc/
The first one is a chroot of a sshd server and the second one is a minimalist debian showing a full distro booting.
In sourceforge, there are
* sshd.tar.gz
* debian.tar.gz
* utsname (actually, utstest.c)
I wonder what is the utstest.c?
Thanks again,
Ian
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810250739s77ad8271u93992c0bcf61345c-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 11 lines · Ian jonhson
Quoted 3 lines
The container will be more or less isolated depending of what you specify in the configuration file.
yes
Quoted 5 lines
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a new sshd server.
Quoted 3 lines
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points?
This is done automatically, with or without configuration.
For examples:
lxc-execute -n foo -- ps -ef --forest
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:55 ? 00:00:00 lxc-execute -n foo -- ps
-ef --forest
root 2 1 0 16:55 pts/6 00:00:00 ps -ef --forest
lxc-execute -n foo ls /proc
will only show process 1 and 2, showing the /proc fs has been remount inside the container without interfering with your own /proc.
You can do the same check by looking at the ipcs inside and outside the container (assuming they are different).
Quoted 7 lines
I have looked through the whole configuration example in README and fond the following settings(which are not tried in my test because I don't understand
them very well)* lxc.mount: is it to mount a file (or volume) so that all the
processes in this container can access the content in file?
It is a file location in the fstab format. All the mounts point specified in this file will be mounted when launching the container. You have a good example in the sshd contrib package.
Quoted 1 line
* lxc.utsname: I don't know what exact functionlity of this setting is.
This option will set the hostname inside the container, so the hostname command will be private to the container.
Quoted 2 lines
* lxc.network.xxx: are there other options expect
"link","hwaddr","ipv4" and "ipv6"?
This are the most complicated options:
lxc.network.type:
=================
That will specify the type of configuration, there are:
* empty : new network stack but only with the loopback
* veth : a bridge + veth pair device configuration, your system should
be configured with a bridge before this kind of configuration
* macvlan : virtualize using a macvlan
lxc.network.hwaddr:
===================
That will set a mac address for the virtualized network device (if not specified, a random value is used).
lxc.network.link:
=================
That specify the physical network device to be linked with the virtualized network device: eg. usually eth0 for macvlan configuration or br0 for bridge configuration.
lxc.network.ipv4:
=================
That sets the ipv4 address for the virtualized network device, it is in the form 1.2.3.4 and the prefix can be specified by adding "/24"
lxc.network.ipv6:
=================
Idem but for ipv6
There is a documentation about the network virtualization at
http://lxc.sourceforge.net/network/configuration.php
Please forget Method 1 , it is pointless.
Quoted 1 line
* lxc.rootfs:for application writing data in it?
This is the location of where the container will be chrooted, read the application will read/write inside.
Quoted 5 lines
I am now looking into the codes and find some information about what the container can do maximally. It seems all the configuration settings are binding with a container and all the schedules are based on container unit. Configuration settings only can be setup before container creation and can not be changed in container lifecycle, right?
Yes, that's right.
Quoted 11 lines
Quoted 4 lines
In the other side, the cgroup are tied with the container, so you can freeze/unfreeze all processes belonging to the container, change the priority or assign an amount of physical memory to be used by the container.
In my brain, the cgroup is mainly used in multiple CPUs. In traditional single CPU machine, can the container separate or determine how much CPU cycle are used by its processes now? Also, administrator has to configure cgroup before it takes action. I have no idea whether both of cgroup and container are totally integrate with each other, or both of them have to be handle in some cases.
Yes, the cpuset was integrated into the cgroup. But people is adding more subsystem to the cgroup. At present, there are the cpuset, the cpu accounting and the dev whitelist. There are the memory controller and cgroup fair scheduler too. Some other subsystems are not already in the mainline but -mm or in a specific patchset, this is the case of the freezer.
The lxc acts as a proxy for the cgroup. So if you mount the cgroup file system, you can see there are several subsystem. I have these ones for examples for my kernel:
cpuacct.usage cpuset.sched_relax_domain_level
cpu.rt_period_us cpu.shares
cpu.rt_runtime_us devices.allow
cpuset.cpu_exclusive devices.deny
cpuset.cpus devices.list
cpuset.mem_exclusive memory.failcnt
cpuset.mem_hardwall memory.force_empty
cpuset.memory_migrate memory.limit_in_bytes
cpuset.memory_pressure memory.max_usage_in_bytes
cpuset.memory_pressure_enabled memory.stat
cpuset.memory_spread_page memory.usage_in_bytes
cpuset.memory_spread_slab notify_on_release
cpuset.mems release_agent
cpuset.sched_load_balance tasks
For example, if I want to assign cpu 1 to my container 'foo', I will specify in the configuration file:
lxc.cgroup.cpuset.cpus = 1
If the container is already running, I can change this value by doing:
lxc-cgroup -n foo cpuset.cpu "0,1"
Quoted 8 lines
Quoted 4 lines
Allowing to assign quota per container is a good idea, but I don't think it is supported by the kernel right now. Perhaps there is a trick to do that but I don't know it :)
I would like to do this part of job. Also, I need to control several groups of processes (belonged to same user) in the same time, isolating them, enforced them with given resource quota, and freeze/unfreeze some of them.
Yeah, it is a good idea, I am not expert in fs/quota but perhaps someone in this mailing list can give some info.
Concerning the freeze, this is already part of lxc via lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem which should be in mainline soon.
Quoted 2 lines
BTW, as for checkpointing of container, is it easy to checkpoint/restart given group of processes in above example?
This is the objective. You should be able to checkpoint at any time the container. For example, you launched the container with the command lxc-execute -n foo, and later you want to checkpoint it. You can do lxc-checkpoint -n foo > my_checkpoint_file.
But the checkpoint / restart is actually under development. The lxc checkpoint/restart commands are experimental and the kernel code is at the beginning, just a single process can be checkpointed / restarted. Before being able to checkpoint multiple processes that will take awhile, especially to have it in the kernel mainline. I guess the quota do not need to be checkpointed as it is part of the file system, so it is always saved.
Quoted 9 lines
Quoted 6 lines
The rootfs option allows you to specify the root file system to be used by the container, so if you specify it, your container will be chrooted inside. This feature is at a very early stage and will be improved in the future, allowing to example to specify a iso image of a file system tree and make use of it.
Good. what kinds of rootfs are supported now? I fond there are a debian file in sourceforge, is it a rootfs image?
Right now, this is only a directory entry. I plan to change that to something more powerful, for example use the union mount, iso image and more.
The debian file is a rootfs image.
Quoted 9 lines
Quoted 8 lines
There are two contributions which are good examples on how to setup a container, I added them to:
http://sourceforge.net/projects/lxc/
The first one is a chroot of a sshd server and the second one is a minimalist debian showing a full distro booting.
Quoted 7 lines
In sourceforge, there are
* sshd.tar.gz
* debian.tar.gz
* utsname (actually,utstest.c)I wonder what is the
utstest.c?
This is a very old testcase, it checks the utsname is private.
Quoted 1 line
Thanks again,
Thanks.
-- Daniel
A message that is not in the archive (sent off-list or not yet received): 49033F90.8090201-NmTC/[email protected]
Quoted 12 lines · Daniel Lezcano
Quoted 11 lines · Ian jonhson
Quoted 3 lines
The container will be more or less isolated depending of what you specify in the configuration file.
yes
Quoted 5 lines
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a new sshd server.
Quoted 5 lines
Quoted 3 lines
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points?This is done automatically, with or without configuration.
I assume Ian has a kernel compiled with CONFIG_PID_NS=n?
-serge
A message that is not in the archive (sent off-list or not yet received): 20081027145058.GA9724-r/[email protected]
Quoted 2 lines
I assume Ian has a kernel compiled with
CONFIG_PID_NS=n?
The PID_NS is set with "yes", according to lxc's README.
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810270807k52fa15b8x6402ea262ae2ddc6-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 5 lines · Ian jonhson
Quoted 2 lines
I assume Ian has a kernel compiled with
CONFIG_PID_NS=n?The
PID_NSis set with "yes", according to lxc's README.
And you say that 'ps -ef' in the container shows all processes on the system?
If so, that's odd. What is the distro on your vm? You're starting the container as root I assume... Is there any output indicating that either the clone(CLONE_NEWPID) or the remount of /proc failed?
-serge
A message that is not in the archive (sent off-list or not yet received): 20081027160003.GB9724-r/[email protected]
Quoted 3 lines
And you say that 'ps -ef' in the container shows all processes on the system?
No, just two processes. similar as Denial's:
----------------------------------------------------------------------------------------------
$ sudo lxc-execute -n foo -- ps -ef --forest [sudo] password: UID PID PPID C STIME TTY TIME CMD root 1 0 0 23:24 ? 00:00:00 lxc-execute -n foo -- ps -ef --f root 2 1 0 23:24 pts/0 00:00:00 ps -ef --forest ---------------------------------------------------------------------------------------------
Quoted 1 line
What is the distro on your vm?
Ubuntu 7.10
Quoted 2 lines
You're
starting the container as root I assume...
I have to run above commands with root, or it doesn't work.
Best Regards,
Ian
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810270929g3ce890e5l91f21523247cf169-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 6 lines · Ian jonhson
Quoted 3 lines
And you say that 'ps -ef' in the container shows all processes on the system?
No, just two processes. similar as Denial's:
Quoted 7 lines
----------------------------------------------------------------------------------------------
$ sudo lxc-execute -n foo -- ps -ef --forest [sudo] password: UID PID PPID C STIME TTY TIME CMD root 1 0 0 23:24 ? 00:00:00 lxc-execute -n foo -- ps -ef --f root 2 1 0 23:24 pts/0 00:00:00 ps -ef --forest ---------------------------------------------------------------------------------------------
Oh, so in fact pids are being properly isolated.
-serge
Quoted 17 lines
Quoted 4 lines
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points?This is done automatically, with or without configuration.
For examples:
lxc-execute -n foo -- ps -ef --forestUID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:55 ? 00:00:00 lxc-execute -n foo -- ps -ef
--forest
root 2 1 0 16:55 pts/6 00:00:00 ps -ef --forest
Quoted 8 lines
lxc-execute -n foo ls /proc
will only show process 1 and 2, showing the /proc fs has been remount inside the container without interfering with your own /proc.
You can do the same check by looking at the ipcs inside and outside the container (assuming they are different).
Is it possible to isolate processes in two different containers when they access given local file. For example, I run a process_A in container_A to create a file named "shared_file". Then, another process (for example, process_B) in container_A can access the "shared_file", but another process, named process_C in container_B can not access the same file. The process_A, process_B, and process_C are run with same uid/gid. how to set the configurations of container_A and container_B to achieve the isolation? Is is possible to do this?
Quoted 11 lines
This are the most complicated options:
lxc.network.type:
=================
That will specify the type of configuration, there are:
* empty : new network stack but only with the loopback
* veth : a bridge + veth pair device configuration, your system
should be configured with a bridge before this kind of configuration
* macvlan : virtualize using a macvlan
Quoted 9 lines
lxc.network.hwaddr:
lxc.network.link:
lxc.network.ipv4:
lxc.network.ipv6:There is a documentation about the network virtualization at
http://lxc.sourceforge.net/network/configuration.php
Please forget Method 1 , it is pointless.
It seems that all the network settings just tell how the container uses network devices. Is it equipped with the functionalities in somewhere (or in kernel ) to limit which container can connect outside in a given time or how to schedule multiple containers access network via only one network device? Or, further, how much bandwidth can each container use?
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
In the other side, the cgroup are tied with the container, so you can freeze/unfreeze all processes belonging to the container, change the
Yes, the cpuset was integrated into the cgroup. But people is adding more subsystem to the cgroup. At present, there are the cpuset, the cpu accounting and the dev whitelist. There are the memory controller and cgroup fair scheduler too. Some other subsystems are not already in the mainline but -mm or in a specific patchset, this is the case of the freezer.
Quoted 4 lines
The lxc acts as a proxy for the cgroup. So if you mount the cgroup file system, you can see there are several subsystem. I have these ones for examples for my kernel:
I wonder here whether cgroup can efficiently isolate two containers access given memory spaces. In my previous experiment about cgroup, I can not achieve a ideal result.
Quoted 4 lines
Concerning the freeze, this is already part of lxc via lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem which should be in mainline soon.
The image of frozen container may be more easy to be migrated to other homogeneous system.
Quoted 8 lines
Quoted 2 lines
BTW, as for checkpointing of container, is it easy to checkpoint/restart given group of processes in above example?
This is the objective. You should be able to checkpoint at any time the container. For example, you launched the container with the command lxc-execute -n foo, and later you want to checkpoint it. You can do lxc-checkpoint -n foo >
my_checkpoint_file.
Quoted 5 lines
But the checkpoint / restart is actually under development. The lxc checkpoint/restart commands are experimental and the kernel code is at the beginning, just a single process can be checkpointed / restarted. Before being able to checkpoint multiple processes that will take awhile, especially to have it in the kernel mainline.
good
Quoted 3 lines
I guess the quota do not need
to be checkpointed as it is part of the file system, so it is always saved.
Not just concerning with file system, the quota I meant also includes how many CPU cycle, memory, bandwidth are allowed to use for a running container. Ideally, it should also be changed dynamically.
Quoted 4 lines
Right now, this is only a directory entry. I plan to change that to something more powerful, for example use the union mount, iso image and more.
I agree.
Best Regards,
Ian
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810270923s40ee505ay214bc9dd8ebae26d-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 26 lines · Ian jonhson
Quoted 17 lines
Quoted 4 lines
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points?This is done automatically, with or without configuration.
For examples:
lxc-execute -n foo -- ps -ef --forestUID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:55 ? 00:00:00 lxc-execute -n foo -- ps -ef
--forest
root 2 1 0 16:55 pts/6 00:00:00 ps -ef --forestQuoted 8 lines
lxc-execute -n foo ls /proc
will only show process 1 and 2, showing the /proc fs has been remount inside the container without interfering with your own /proc.
You can do the same check by looking at the ipcs inside and outside the container (assuming they are different).
Quoted 8 lines
Is it possible to isolate processes in two different containers when they access given local file. For example, I run a
process_Aincontainer_Ato create a file named "shared_file". Then, another process (for example,process_B) incontainer_Acan access the "shared_file", but another process, namedprocess_Cincontainer_Bcan not access the same file. Theprocess_A,process_B, andprocess_Care run with same uid/gid. how to set the configurations ofcontainer_Aandcontainer_Bto achieve the isolation? Is is possible to do this?
1. You can put them in each a different chroot. For instance, extract the debian.tar.gz which Daniel has pointed you to before into both /vs1 and /vs2. Put container_A in /vs1 and container_B in /vs2.
2. You can use SELinux or Smack. Details for now left as an excercise to the reader.
3. One day you'll be able to use user namespaces. In that case, even though process_{A,B,C} are all running with the same integer uid/gid, process_C will be in a different different user namespace than process_{A,B}. So the file created by process_A will be accessed by process_C as though they were different userids. Which means if process_A set permissions on the file as rwxr-xr-x, then process_c will get r-x permissions. If process_A set the permissions to rwxr-xr-x, then process_c will have no permissions.
-serge
Quoted 24 lines · Ian jonhson
Quoted 15 lines
Quoted 3 lines
hmm.... then, how to configure the container to get the isolation of pid, ipc and
mount points?This is done automatically, with or without configuration.
For examples:
lxc-execute -n foo -- ps -ef --forestUID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:55 ? 00:00:00 lxc-execute -n foo -- ps -ef
--forest
root 2 1 0 16:55 pts/6 00:00:00 ps -ef --forestQuoted 8 lines
lxc-execute -n foo ls /proc
will only show process 1 and 2, showing the /proc fs has been remount inside the container without interfering with your own /proc.
You can do the same check by looking at the ipcs inside and outside the container (assuming they are different).
Quoted 8 lines
Is it possible to isolate processes in two different containers when they access given local file. For example, I run a
process_Aincontainer_Ato create a file named "shared_file". Then, another process (for example,process_B) incontainer_Acan access the "shared_file", but another process, namedprocess_Cincontainer_Bcan not access the same file. Theprocess_A,process_B, andprocess_Care run with same uid/gid. how to set the configurations ofcontainer_Aandcontainer_Bto achieve the isolation? Is is possible to do this?
Yes, you can do this in different ways. The first one is as Serge suggested with chroot, sshd contrib is a good example on how a directory tree with a set of mount --bind can be chrooted to ensure file isolation (cf the fstab file for this configuration).
This first solution provides isolation and security. Isolation because in a container A, the processes running inside can access a file at the same location than the processes running inside a container B without interacting. Security because processes from container A can not access the file in container B.
But, if someone wants to run several instances of the same application and wants to avoid the file to be overwritten, the security is pointless in this case and a simpler configuration can be used with just a bind mount specified in the fstab file lxc option.
For example, we have a computing program writing its results in the directory /tmp/results and we want to launch multiple instances of this program in parallel. We can do:
mkdir -p /tmp/container_A/results
mkdir -p /tmp/container_B/results
One fstab per container
for container_A:/tmp/container_A/results /tmp/results none ro,bind 0 0
for container_B:/tmp/container_B/results /tmp/results none ro,bind 0 0
And one configuration per container with:
for container_A:
lxc.mount = fstab.container_A
for container_B:
lxc.mount = fstab.container_B
And finally:
lxc-execute -n container_A -f ./container_A.conf
lxc-execute -n container_B -f ./configure_B.conf
(if the container was not created before lxc-execute will automatically create it and autodestroy it after at exit).
So, the application is unmodified and will continue writing the results in /tmp/results which is in fact a private location for the container.
Quoted 20 lines
Quoted 10 lines
This are the most complicated options:
lxc.network.type:
=================
That will specify the type of configuration, there are:
* empty : new network stack but only with the loopback
* veth : a bridge + veth pair device configuration, your system
should be configured with a bridge before this kind of configuration
* macvlan : virtualize using a macvlanQuoted 9 lines
lxc.network.hwaddr:
lxc.network.link:
lxc.network.ipv4:
lxc.network.ipv6:There is a documentation about the network virtualization at
http://lxc.sourceforge.net/network/configuration.php
Please forget Method 1 , it is pointless.
Quoted 6 lines
It seems that all the network settings just tell how the container uses network devices. Is it equipped with the functionalities in somewhere (or in kernel ) to limit which container can connect outside in a given time or how to schedule multiple containers access network via only one network device? Or, further, how much bandwidth can each container use?
I didn't think about the scheduling. I guess this is certainly a job for an upper layer build on top of the liblxc. I wrote a small network namespace freezer patch I will post soon. It blocks all the network traffic for a given network namespace, so having some kind of resource manager making use of this feature can do easily the trick.
Concerning the bandwidth, this is part of the traffic control job and it can be plugged with the container. This is something I have in mind, that is specify the bandwidth per container. We have all the tools to do that now, network isolation and traffic control :)
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
In the other side, the cgroup are tied with the container, so you can freeze/unfreeze all processes belonging to the container, change the
Yes, the cpuset was integrated into the cgroup. But people is adding more subsystem to the cgroup. At present, there are the cpuset, the cpu accounting and the dev whitelist. There are the memory controller and cgroup fair scheduler too. Some other subsystems are not already in the mainline but -mm or in a specific patchset, this is the case of the freezer.
Quoted 8 lines
Quoted 4 lines
The lxc acts as a proxy for the cgroup. So if you mount the cgroup file system, you can see there are several subsystem. I have these ones for examples for my kernel:
I wonder here whether cgroup can efficiently isolate two containers access given memory spaces. In my previous experiment about cgroup, I can not achieve a ideal result.
The cgroup does not aim to do isolation, it only provides a mechanism to group the processes under the same identifier. Each subsystem, written as plugins, make use of this to assign cpu, identify cpu consumption...
But in the other hand, if we use both cgroup and namespaces to provide a container with resource management, some isolation should be provided. For example, on a 8 cpus system, we assign 2 cpus to a container via cgroup/cpuset, because the cpus are not isolated, inside the container browsing /proc/cpuinfo will show all the cpus of the system and that can, in some cases, lead to an error (eg. sched_setaffinity), IMO.
Quoted 23 lines
Quoted 4 lines
Concerning the freeze, this is already part of lxc via lxc-freeze/lxc-unfreeze but that relies on the freezer cgroup subsystem which should be in mainline soon.
The image of frozen container may be more easy to be migrated to other homogeneous system.
Quoted 7 lines
Quoted 2 lines
BTW, as for checkpointing of container, is it easy to checkpoint/restart given group of processes in above example?
This is the objective. You should be able to checkpoint at any time the container. For example, you launched the container with the command lxc-execute -n foo, and later you want to checkpoint it. You can do lxc-checkpoint -n foo >
my_checkpoint_file.Quoted 5 lines
But the checkpoint / restart is actually under development. The lxc checkpoint/restart commands are experimental and the kernel code is at the beginning, just a single process can be checkpointed / restarted. Before being able to checkpoint multiple processes that will take awhile, especially to have it in the kernel mainline.
good
Quoted 7 lines
Quoted 3 lines
I guess the quota do not need
to be checkpointed as it is part of the file system, so it is always saved.Not just concerning with file system, the quota I meant also includes how many CPU cycle, memory, bandwidth are allowed to use for a running container. Ideally, it should also be changed dynamically.
Ah ok, I see, quota + accounting. I think there is some work around that in this mailing list :)
Quoted 9 lines
Quoted 4 lines
Right now, this is only a directory entry. I plan to change that to something more powerful, for example use the union mount, iso image and more.
I agree.
Best Regards,
Thanks
-- Daniel
A message that is not in the archive (sent off-list or not yet received): 49062482.7070006-NmTC/[email protected]
Thank Daniel, Serge, and all the friends in the maillist. Now, I have given a bird view about current status of container. The finial question is that how to do contribution:
* what matters/fashion should I obey to develop codes?
* how to upload my patch or my extension codes?
* where should I upload the codes? sourceforge, or gforge?
Are there nay policies I should pay attention?
Thanks again,
Ian
Quoted 4 lines
Thanks
-- Daniel
A message that is not in the archive (sent off-list or not yet received): 8f34198c0810280110s6a7fd98dm355b4747a26ff895-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 8 lines · Ian jonhson
Thank Daniel, Serge, and all the friends in the maillist. Now, I have given a bird view about current status of container. The finial question is that how to do contribution:
* what matters/fashion should I obey to develop codes?
* how to upload my patch or my extension codes?
* where should I upload the codes? sourceforge, or gforge?Are there nay policies I should pay attention?
Hi Ian,
I added the CONTRIBUTING file in the source tree directory, that gives the guidelines to submit contribution to 'lxc'.
Regards.
-- Daniel.
Hi again,
Quoted 6 lines
The container will be more or less isolated depending of what you specify in the configuration file.
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a
I have played with lxc-0.3.0 for several days, trying to transplant existing applications
on the base of container.
The default configuration can present a view of isolated pid, ipc (e.g. run the ps or top), however the memory seems not to be isolated. The process in a container still can see the whole physical memory. Also, my test routine can still apply more memory than the quota defined in "memory.limit_in_bytes".
Are there anything missing?
Best Regards,
Ian
A message that is not in the archive (sent off-list or not yet received): 8f34198c0812112333k6c1474f3hb67e963b39a0d0d6-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 13 lines · Ian jonhson
Hi again,
Quoted 6 lines
The container will be more or less isolated depending of what you specify in the configuration file.
Without any configuration file, you will have pid, ipc and mount points isolated. If you specify the utsname, it will be isolated and if you specify the network you will have a new network stack allowing to run for example a
I have played with lxc-0.3.0 for several days, trying to transplant existing applications
on the base of container.
Quoted 7 lines
The default configuration can present a view of isolated pid, ipc (e.g. run the ps or top), however the memory seems not to be isolated. The process in a container still can see the whole physical memory. Also, my test routine can still apply more memory than the quota defined in "memory.limit_in_bytes".
Are there anything missing?
Hi, Ian,
With what version of the kernel due you see this problem. The memory controller is undergoing a churn and I think we found something that potentially breaks the memory controller in the current -mm. To verify
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
2. Check tasks file to see that the container tasks are indeed in the container.
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): 20081212081238.GA4365-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Quoted 4 lines
With what version of the kernel due you see this problem. The memory controller is undergoing a churn and I think we found something that potentially breaks the memory controller in the current -mm. To verify
I am not sure whether the -mm is opened. I just opened the options as said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
2.6.27-rc8.
Quoted 2 lines
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
How can I know the number correspond to the control group?
Quoted 3 lines
2. Check tasks file to see that the container tasks are indeed in the container.
My program forks a child and its child (grandson of previous process) call the the /bin/bash to open a shell. I just can find the parent process is in container
tasks file, but its child and grandson process are not.
Any help?
Quoted 1 line
A message that is not in the archive (sent off-list or not yet received): 8f34198c0812120106k390a1097ub31905cd2bc44676-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 8 lines · Ian jonhson
Quoted 4 lines
With what version of the kernel due you see this problem. The memory controller is undergoing a churn and I think we found something that potentially breaks the memory controller in the current -mm. To verify
I am not sure whether the -mm is opened. I just opened the options as said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
2.6.27-rc8.
OK, so this is not an lxc kernel, but a mainline kernel.
Quoted 5 lines
Quoted 3 lines
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
How can I know the number correspond to the control group?
From outside the container, figure out the process id of init and then do a cat /proc/<pid>/cgroup. Then go to that directory and cat memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify the setup
Quoted 10 lines
Quoted 4 lines
2. Check tasks file to see that the container tasks are indeed in the container.
My program forks a child and its child (grandson of previous process) call the the
/bin/bashto open a shell. I just can find the parent process is in container
tasks file, but its child and grandson process are not.
Hmmm.. but the child and grandson are alive right? Do a cat /proc/<pid>/cgroup on them to see where they went or where they are.
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): 20081212091829.GB4365-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Quoted 3 lines
OK, so this is not an lxc kernel, but a mainline kernel.
So, what is the lxc kernel you meat? Is it different from the 2.6.27-rc8. I use that version recommended by Daniel.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
How can I know the number correspond to the control group?
From outside the container, figure out the process id of init and then do a cat /proc/<pid>/cgroup. Then go to that directory and cat memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify the setup
OK, I have done the checking, but I don't understand what is meaning of "corresponding". The content is shown as follows:
$ cat /proc/4506/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/ $ cat memory.usage_in_bytes 223932416
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
2. Check tasks file to see that the container tasks are indeed in the container.
Quoted 7 lines
Quoted 5 lines
My program forks a child and its child (grandson of previous process) call the the
/bin/bashto open a shell. I just can find the parent process is in container
tasks file, but its child and grandson process are not.Hmmm.. but the child and grandson are alive right?
yes... I can see them by a container-outside bash running with ps
Quoted 3 lines
Do a cat
/proc/<pid>/cgroup on them to see where they went or where they are.
They are all there, similar with pid=4506.
$ cat /proc/4506/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/ $ cat /proc/4507/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507 $ cat /proc/4508/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506
Any idea?
A message that is not in the archive (sent off-list or not yet received): 8f34198c0812120202t39582a27k47c98b88246f18ce-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 19 lines · Ian jonhson
Quoted 3 lines
OK, so this is not an lxc kernel, but a mainline kernel.
So, what is the lxc kernel you meat? Is it different from the 2.6.27-rc8. I use that version recommended by Daniel.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
How can I know the number correspond to the control group?
From outside the container, figure out the process id of init and then do a cat /proc/<pid>/cgroup. Then go to that directory and cat memory.usage_in_bytes, memory.stat and memory.limit_in_bytes to verify the setup
Quoted 7 lines
OK, I have done the checking, but I don't understand what is meaning of "corresponding". The content is shown as follows:
$ cat /proc/4506/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/ $ cat memory.usage_in_bytes 223932416
OK, so memory.usage_in_bytes is 223932416 for the root cgroup. You process 4506, which is (I assume) a container process is running in "/" or the root cgroup. By default root cgroups are unlimited. You can verify by reading memory.limit_in_bytes
Quoted 20 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
2. Check tasks file to see that the container tasks are indeed in the container.
Quoted 7 lines
Quoted 5 lines
My program forks a child and its child (grandson of previous process) call the the
/bin/bashto open a shell. I just can find the parent process is in container
tasks file, but its child and grandson process are not.Hmmm.. but the child and grandson are alive right?
yes... I can see them by a container-outside bash running with ps
Quoted 3 lines
Do a cat
/proc/<pid>/cgroup on them to see where they went or where they are.
Quoted 9 lines
They are all there, similar with pid=4506.
$ cat /proc/4506/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/ $ cat /proc/4507/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507 $ cat /proc/4508/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506
Are there three containers on the system
Can you send me the output of <cgroup mount point>/4507/memory.* and 4506/memory.*. cgroup mount point can be obtained from /proc/mounts.
Signature · 1 line
Balbir
A message that is not in the archive (sent off-list or not yet received): 20081212142529.GD4365-SINUvgVNF2CyUtPGxGje5AC/[email protected]
Quoted 10 lines
Quoted 9 lines
They are all there, similar with pid=4506.
$ cat /proc/4506/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/ $ cat /proc/4507/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4507 $ cat /proc/4508/cgroup 255:freezer,devices,memory,cpuacct,cpu,ns,debug,cpuset:/4506
Quoted 2 lines
Are there three containers on the system
No. one container on system. They are parent-child relationship.
Quoted 4 lines
Can you send me the output of <cgroup mount point>/4507/memory.* and 4506/memory.*. cgroup mount point can be obtained from
/proc/mounts.
There is only the directory of 4507 in <cgroup mount point>, not exist the 4506. I print the stuff as follows:
$ ls memory.* memory.failcnt memory.limit_in_bytes memory.stat memory.force_empty memory.max_usage_in_bytes memory.usage_in_bytes
$ cat memory.failcnt 0
$ cat memory.force_empty cat: memory.force_empty: Invalid argument
$ cat memory.limit_in_bytes 9223372036854775807
$ cat memory.max_usage_in_bytes 1236992
$ cat memory.stat cache 0 rss 851968 pgpgin 879 pgpgout 671 active 851968 inactive 0
I don't know why memory.force_empty can not be printed the content.
Any idea?
A message that is not in the archive (sent off-list or not yet received): 8f34198c0812130018j27b5ee34r15eed114013dfba4-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 20 lines
There is only the directory of 4507 in <cgroup mount point>, not exist the 4506. I print the stuff as follows:
$ ls memory.* memory.failcnt memory.limit_in_bytes memory.stat memory.force_empty memory.max_usage_in_bytes memory.usage_in_bytes$ cat memory.failcnt 0$ cat memory.force_empty cat: memory.force_empty: Invalid argument$ cat memory.limit_in_bytes 9223372036854775807$ cat memory.max_usage_in_bytes 1236992
Quoted 9 lines
$ cat memory.stat cache 0 rss 851968 pgpgin 879 pgpgout 671 active 851968 inactive 0
Sorry for missing the last one:
$ cat memory.usage_in_bytes 856064
Quoted 7 lines · Ian jonhson
Quoted 4 lines
With what version of the kernel due you see this problem. The memory controller is undergoing a churn and I think we found something that potentially breaks the memory controller in the current -mm. To verify
I am not sure whether the -mm is opened. I just opened the options as said in REAEME of lxc-0.3.0. And, the kernel version in my VM is:
2.6.27-rc8.
2.6.27.8 is out, perhaps you should try this version which contains the latest bug fixes.
Quoted 4 lines
Quoted 2 lines
1. Can you see the memory.usage_in_bytes corresponding to the control group where you run the container
How can I know the number correspond to the control group?
Assuming you created a container named "foo", you should be able retrieve these values with the following commands:
lxc-cgroup -n foo tasks
lxc-cgroup -n foo memory.usage_in_bytes
The lxc-0.5.1 has been released and it fixes some bugs related with the cgroup management.
Quoted 12 lines
Quoted 3 lines
2. Check tasks file to see that the container tasks are indeed in the container.
My program forks a child and its child (grandson of previous process) call the the
/bin/bashto open a shell. I just can find the parent process is in container
tasks file, but its child and grandson process are not.Any help?