trying to build simple checkpoint/restart recipes
What I've done so far:
created a KVM vm and installed up-to-date maverick
add-apt-repository ppa:appcr/ppa
apt-get update && apt-get dist-upgrade
apt-get install libvirt-bin lxc linux-image-2.6.34-1cr4
sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Ubuntu, with Linux 2.6.34-1cr4-generic"/' /etc/default/grub update-grub
replaced 122 with 123 in /etc/libvirt/qemu/networks/default.xml and /var/lib/libvirt/network/default.xml reboot
# The following should go into an upstart script shipped with the appcr packages # as they must be done on each boot
chmod 666 /dev/pts/ptmx
rm /dev/ptmx
ln -s /dev/pts/ptmx /dev/ptmx
mkdir -p /cgroup
mount -t cgroup cggroup /cgroup/
echo /bin/remove_dead_cgroup.sh > /cgroup/release_agent
echo 1 > /cgroup/notify_on_release
#
cat > /etc/lxc-basic.conf << EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOF
lxc-create -f /etc/lxc-basic.conf -n cr1 -t ubuntu cd /var/lib/lxc/cr1/rootfs/sbin
mv init upstart
cat > init << EOF
#!/bin/sh
rm -f /shutdown
hostname cr1
exec 0<&-
exec 0</dev/null
exec 1>&-
exec 1>nohup.out
exec 2>&-
exec 2>nohup.out
mkdir -p /tmp2
mount --bind /tmp2 /tmp
mount -a
mount -t proc proc /proc
mount -t tmpfs varrun /var/run
mkdir /var/run/network
mkdir /var/run/sshd
ifconfig eth0 192.168.123.21 up
screen -A -d -m -S console
/usr/sbin/sshd
while [ ! -f /shutdown ]; do
sleep 4s
done
EOF
lxc-start -n cr1
(in another console)
ssh 192.168.123.21
screen -r
ps
ctrl-a d
exit
lxc-freeze -n cr1
lxc-checkout -n cr1 -S /root/cr1.s1
So far, so good. Note that I couldn't use upstart for my init bc upstart uses inotify, which we don't yet checkpoint. The kernel is compiled without ipv6 bc that was also causing a problem (though I thought ipv6 was supported for checkpoint?) and therefore I needed a custom libvirt package which didn't break when ipv6 is not there.
The problem now is when attempting to restart:
lxc-stop -n cr1 lxc-restart -n cr1 -S /root/cr1.s1
There are two issues:
1. how to re-create the mounts. Kernel doesn't do it yet. There isn't (that I know of) a clean way to hook lxc-restart to do it. Comments?
2. likewise there *may* end up being a question of where to best hook the backup/snapshot and restore of filesystems. Though for these examples (screen and next vncserver) it shouldn't be necessary.
I'm trying to do this using lxc-checkpoint and lxc-restart so as to keep the instructions as simple as possible. The hope is in the next few weeks to have a few recipes that people can try out. But if it's not (cleanly) possible using lxc-restart, then I guess I can switch to using user-cr and some container tarballs, which seems a lot easier in the short term but less useful in the long term.
thanks,
-serge
Quoted 71 lines · Serge E. Hallyn
What I've done so far:
created a KVM vm and installed up-to-date maverick
add-apt-repository ppa:appcr/ppa
apt-get update && apt-get dist-upgrade
apt-get install libvirt-bin lxc linux-image-2.6.34-1cr4
sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT="Ubuntu, with Linux 2.6.34-1cr4-generic"/'/etc/default/grubupdate-grubreplaced 122 with 123 in
/etc/libvirt/qemu/networks/default.xmland/var/lib/libvirt/network/default.xmlreboot# The following should go into an upstart script shipped with the appcr packages # as they must be done on each boot
chmod 666/dev/pts/ptmx
rm/dev/ptmx
ln -s/dev/pts/ptmx/dev/ptmx
mkdir -p /cgroup
mount -t cgroup cggroup /cgroup/
echo/bin/remove_dead_cgroup.sh> /cgroup/release_agent
echo 1 > /cgroup/notify_on_release
#cat >
/etc/lxc-basic.conf<< EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOFlxc-create -f
/etc/lxc-basic.conf-n cr1 -t ubuntu cd/var/lib/lxc/cr1/rootfs/sbin
mv init upstartcat > init << EOF
#!/bin/sh
rm -f /shutdown
hostname cr1exec 0<&-
exec 0</dev/null
exec 1>&-
exec 1>nohup.out
exec 2>&-
exec 2>nohup.outmkdir -p /tmp2
mount --bind /tmp2 /tmpmount -a
mount -t proc proc /proc
mount -t tmpfs varrun/var/run
mkdir/var/run/network
mkdir/var/run/sshd
ifconfig eth0 192.168.123.21 up
screen -A -d -m -S console
/usr/sbin/sshd
while [ ! -f /shutdown ]; do
sleep 4s
done
EOFlxc-start -n cr1
(in another console)
ssh 192.168.123.21
screen -r
ps
ctrl-a d
exit
Quoted 5 lines
lxc-freeze -n cr1
lxc-checkout -n cr1 -S/root/cr1.s1So far, so good. Note that I couldn't use upstart for my init bc upstart uses inotify, which we don't yet checkpoint. The kernel is compiled without
Interesting, I didn't know that. What does upstart use inotify for?
Quoted 6 lines
ipv6 bc that was also causing a problem (though I thought ipv6 was supported for checkpoint?) and therefore I needed a custom libvirt package which didn't break when ipv6 is not there.
The problem now is when attempting to restart:
Quoted 8 lines
lxc-stop -n cr1 lxc-restart -n cr1 -S /root/cr1.s1There are two issues:
1. how to re-create the mounts. Kernel doesn't do it yet. There isn't (that I know of) a clean way to hook lxc-restart to do it. Comments?
It's incomplete but I think you can save the most important portions of a mount namespace with a simple 1-line command:
lxc-attach -n cr1 cat /proc/self/mountinfo > cr1.mountinfo
It's incomplete because:
1. It does not adequately address cross-mount-ns bind mounts (IIRC).
2. It won't work for nested containers (though I don't know if lxc supports this already it's not *too* far fetched to expect folks will ask for it in the future). We can extend the hack to deal with this by making a small change in sys_checkpoint but I can't see how to fix #1 without doing it all in-kernel anyway.
The restoration of the mounts is not scriptable however. It involves parsing the mountinfo file and coordinating the mounts with those done by lxc itself during lxc-restart. I honestly haven't looked at that closely enough yet to say how pretty/ugly that'd be but it entails modifications to lxc-restart itself. And since #1 above would still be an issue I'm not sure it's worth doing it that way.
Cheers,
-Matt Helsley
Quoted 4 lines · Matt Helsley
Quoted 2 lines
So far, so good. Note that I couldn't use upstart for my init bc upstart uses inotify, which we don't yet checkpoint. The kernel is compiled without
Interesting, I didn't know that. What does upstart use inotify for?
Dunno :) I was quite put out though.
Quoted 13 lines
Quoted 5 lines
There are two issues:
1. how to re-create the mounts. Kernel doesn't do it yet. There isn't (that I know of) a clean way to hook lxc-restart to do it. Comments?
It's incomplete but I think you can save the most important portions of a mount namespace with a simple 1-line command:
lxc-attach -n cr1 cat
/proc/self/mountinfo> cr1.mountinfoIt's incomplete because:
Quoted 8 lines
1. It does not adequately address cross-mount-ns bind mounts (IIRC).
2. It won't work for nested containers (though I don't know if lxc supports this already it's not *too* far fetched to expect folks will ask for it in the future). We can extend the hack to deal with this by making a small change in
sys_checkpointbut I can't see how to fix #1 without doing it all in-kernel anyway.
Heck, for these examples I don't mind just having a sort of dummy fstab file which both the dummy init and restart use.
Quoted 3 lines
The restoration of the mounts is not scriptable however. It involves parsing the mountinfo file and coordinating the mounts with those done by lxc itself during lxc-restart. I honestly haven't looked at that closely
I'd be fine with requiring some bit of hand-parsing. But right, even once we get a list of the mounts to be restored, I don't know of any good way to get those mounts re-created at the right time.
I suppose I could hack lxc-restart to do it. But I'm sort of hoping we can get something less hacked and more true to the 'real' upstream code.
Quoted 3 lines
enough yet to say how pretty/ugly that'd be but it entails modifications to lxc-restart itself. And since #1 above would still be an issue I'm not sure it's worth doing it that way.
So do you know of anyone who's been working on re-creation of mounts in the kernel? If not, what have you been doing, hand-scripting all container creation, checkpoint, and restart?
thanks,
-serge
Quoted 7 lines
Quoted 3 lines
The restoration of the mounts is not scriptable however. It involves parsing the mountinfo file and coordinating the mounts with those done by lxc itself during lxc-restart. I honestly haven't looked at that closely
I'd be fine with requiring some bit of hand-parsing. But right, even once we get a list of the mounts to be restored, I don't know of any good way to get those mounts re-created at the right time.
Mount code is one of my old stomping grounds from back when I wrote the busybox mount and switch_root commands and had to learn more implementation details about it than I ever wanted to know. :)
I never could find a proper mount spec, and kept meaning to write one, but I blathered about some of the less obvious details here:
http://www.mail-archive.com/[email protected]/msg07013.html
There are four top level categories of filesystem: Block backed, ram backed, pipe backed (network and fuse and so on), and synthetic (sysfs, procfs, devtmpfs...). And that's not counting bind mounts (which are internal to the VFS and not really a filesystem), and loopback devices (which are sort of the _opposite_ of a filesystem)...
Quoted 3 lines
I suppose I could hack lxc-restart to do it. But I'm sort of hoping we can get something less hacked and more true to the 'real' upstream code.
Which upstream code?
Quoted 3 lines
So do you know of anyone who's been working on re-creation of mounts in the kernel? If not, what have you been doing, hand-scripting all container creation, checkpoint, and restart?
I express interest in this topic.
Rob
Quoted 8 lines · Rob Landley
Quoted 7 lines
Quoted 3 lines
The restoration of the mounts is not scriptable however. It involves parsing the mountinfo file and coordinating the mounts with those done by lxc itself during lxc-restart. I honestly haven't looked at that closely
I'd be fine with requiring some bit of hand-parsing. But right, even once we get a list of the mounts to be restored, I don't know of any good way to get those mounts re-created at the right time.
Quoted 8 lines
Mount code is one of my old stomping grounds from back when I wrote the busybox mount and
switch_rootcommands and had to learn more implementation details about it than I ever wanted to know. :)I never could find a proper mount spec, and kept meaning to write one, but I blathered about some of the less obvious details here:
Bookmarked :)
Quoted 5 lines
There are four top level categories of filesystem: Block backed, ram backed, pipe backed (network and fuse and so on), and synthetic (sysfs, procfs, devtmpfs...). And that's not counting bind mounts (which are internal to the VFS and not really a filesystem), and loopback devices (which are sort of the _opposite_ of a filesystem)...
Right, for starters handling only bind mounts would be useful. It's feasible for userspace to rsync the contents of tmpfs filesystems during checkpoint and before restart - but it's harder to find the right place for the bind mounts to get re-attached if done in userspace, because we don't want to do it too early and risk having mount leaks (so we can't checkpoint later), and it's hard to coordinate doing it later since someone inside the container has to do it (unlesss, again, we have leaks - well, or maybe having MNT_SHARED / for the container would suffice).
Quoted 5 lines
Quoted 3 lines
I suppose I could hack lxc-restart to do it. But I'm sort of hoping we can get something less hacked and more true to the 'real' upstream code.
Which upstream code?
Heh, I should have said upstream-destined code. Referring to lxc.sf.net and the kernel at www.linux-cr.org.
Quoted 5 lines
Quoted 3 lines
So do you know of anyone who's been working on re-creation of mounts in the kernel? If not, what have you been doing, hand-scripting all container creation, checkpoint, and restart?
I express interest in this topic.
Awesome. Note that we've had lots of prior discussions about the topic, it's just that we never came to a conclusion, so some fresh experienced blood would be very helpful.
The last time I went into detail on the topic was at
http://www.mail-archive.com/[email protected]/msg21418.html
while some older notes on the simpler topics are at
https://ckpt.wiki.kernel.org/index.php/Mounts
thanks,
-serge