Userspace checkpoint/restart hack: cryo

A guy named Marc Vertes wrote this as a little demonstration of checkpoint/restart. I've been using it to experiment with checkpoint/restart. I thought it might be of some use as we move subsystems to being helped by the kernel to checkpoint and restart.

It's ptrace-based, and stuck on i386 for now. It can probably be ported
elsewhere without too much trouble. It doesn't support *anything
fancy like multuiple tasks :). It has the advantage of being very feature-bare, and I think it is pretty easy to hack on. Whatever c/r support we add to the kernel could easily be added on and tested.

http://userweb.kernel.org/~daveh/cryo/cryo-001.tar.gz

Usage:
cr -p pidof task > checkpoint.cryo
cr -r < checkpoint.cryo

If anyone else has something simpler or easier to hack on, I'm all ears.

Quoted 11 lines · Dave Hansen

A guy named Marc Vertes wrote this as a little demonstration of checkpoint/restart. I've been using it to experiment with checkpoint/restart. I thought it might be of some use as we move subsystems to being helped by the kernel to checkpoint and restart.

It's ptrace-based, and stuck on i386 for now. It can probably be ported
elsewhere without too much trouble. It doesn't support *anything
fancy like multuiple tasks :). It has the advantage of being very feature-bare, and I think it is pretty easy to hack on. Whatever c/r support we add to the kernel could easily be added on and tested.

Quoted 7 lines

http://userweb.kernel.org/~daveh/cryo/cryo-001.tar.gz

Usage:
cr -p pidof task > checkpoint.cryo
cr -r < checkpoint.cryo

If anyone else has something simpler or easier to hack on, I'm all ears.

Indeed. It looks simple enough.

do you have some kernel requirement ? I run Fedora 8

Here's my first try on a program calculating decimal of PI :

	$ ./cr -p `pidof pi1` > pi1.cryo
	attaching to pid: 11082
	[11087 cr.c:243 getfdinfo()] n : 0
	WARNING (sci.c:242) unexpected signal for 11082: 11
	[11087 sci.c:228 ptrace_waitsyscall()] WTERMSIG(status) : 11
	ERROR (sci.c:383) ptrace_getregs(11082, 0xbfe4a3d0) errno=3: No such process
	./cr[0x8051f10]
	./cr[0x8049ce9]
	./cr[0x804b7d2]
	./cr[0x804f75b]
	***STOP***

other terminal :

	$ pi1 20000
	pi1 - 20000 digits, 78.1 kbytes
	Segmentation fault (core dumped)

Thanks,

C.

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

Quoted 4 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

Quoted 11 lines · Cedric Le Goater

Quoted 4 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

2.6.25-mm1 (sorry for the late answer - plenty of days off in France in May).

Regards,
Nadia

Quoted 15 lines · Nadia Derbey

Quoted 12 lines · Cedric Le Goater

Quoted 5 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

Quoted 5 lines

2.6.25-mm1 (sorry for the late answer - plenty of days off in France in May).

Regards,
Nadia

I'm playing with features in cryo, and keeping a git tree at:

git://git.sr71.net/~hallyn/cryodev.git

It's meant to exploit the extras which are in the -lxc kernel at lxc.sf.net. Current latest kernel patch is at
http://lxc.sourceforge.net/patches/2.6.26/2.6.26-rc2-mm1-lxc4/ This -lxc tree contains, for instance, Nadia's next_id patches, exploitation for setting ids for sysvipc and for tasks at fork, and updated ipc_setall patches (also using next_id). The version of cryo in my git tree exploits these. If you're root when you restart a task, it will clone a new set of namespaces and recreate your sysvipc objects, and it will reset your pids (even if you're not root if the pids are available).

Cryo is still rather touchy - I'm trying to track this down. Older distros seem to actually have the most success (I'm told FC6 is somewhat useful, while my FC8 kvm image is practically useless with cryo).

So if you want to play with it, please clone and do so, and send me any patches you think should go in.

thanks,
-serge
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Quoted 28 lines · Serge E. Hallyn

Quoted 15 lines · Nadia Derbey

Quoted 12 lines · Cedric Le Goater

Quoted 5 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

Quoted 5 lines

2.6.25-mm1 (sorry for the late answer - plenty of days off in France in May).

Regards,
Nadia

I'm playing with features in cryo, and keeping a git tree at:

git://git.sr71.net/~hallyn/cryodev.git

Quoted 9 lines

It's meant to exploit the extras which are in the -lxc kernel at lxc.sf.net. Current latest kernel patch is at
http://lxc.sourceforge.net/patches/2.6.26/2.6.26-rc2-mm1-lxc4/ This -lxc tree contains, for instance, Nadia's next_id patches, exploitation for setting ids for sysvipc and for tasks at fork, and updated ipc_setall patches (also using next_id). The version of cryo in my git tree exploits these. If you're root when you restart a task, it will clone a new set of namespaces and recreate your sysvipc objects, and it will reset your pids (even if you're not root if the pids are available).

Serge,

I noticed that the sys_hijack() has disappeared from the lxc dev tree:
would you mind putting it back. I think it might be useful if we want to start a task in a newly defined cgroup, checkpoint it and then try to restart it. We will need to 'join' the restarted container to check if everything has correctly be restarted. Or may be is there another way to do that?

Quoted 55 lines · Serge E. Hallyn

Quoted 39 lines · Nadia Derbey

Quoted 27 lines · Serge E. Hallyn

Quoted 14 lines · Nadia Derbey

Quoted 11 lines · Cedric Le Goater

Quoted 5 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

Quoted 6 lines

2.6.25-mm1 (sorry for the late answer - plenty of days off in France in May).

Regards,
Nadia

I'm playing with features in cryo, and keeping a git tree at:

git://git.sr71.net/~hallyn/cryodev.git

Quoted 10 lines

It's meant to exploit the extras which are in the -lxc kernel at lxc.sf.net. Current latest kernel patch is at
http://lxc.sourceforge.net/patches/2.6.26/2.6.26-rc2-mm1-lxc4/ This -lxc tree contains, for instance, Nadia's next_id patches, exploitation for setting ids for sysvipc and for tasks at fork, and updated ipc_setall patches (also using next_id). The version of cryo in my git tree exploits these. If you're root when you restart a task, it will clone a new set of namespaces and recreate your sysvipc objects, and it will reset your pids (even if you're not root if the pids are available).

Quoted 9 lines

Serge,

I noticed that the sys_hijack() has disappeared from the lxc dev
tree: would you mind putting it back. I think it might be useful if we want to start a task in a newly defined cgroup, checkpoint it and then try to restart it. We will need to 'join' the restarted container to check if everything has correctly be restarted. Or may be is there another way to do that?

Well it *could* be done similar to how cryo itself works, by ptracing the destination task and making it fork a task which execve()s a process to do the querying. But yuck.

Quoted 6 lines

Kathy, I'm sorry, I know I asked you to take sys_hijack() out. Could you please put it back in? Preferably at the end of the queue, as I don't want other patches having to be ported on top of it since its future is very suspect... Let me know if you have trouble porting it, but it should be pretty simple.

I added sys_hijack() (namespaces-introduce-sys_hijack.patch) back, with a little bit of massaging to get it to port properly (as Cedrics clone64-change-clone_flag-type-to-u64.patch changes unsigned long to u64) but was unable to compile. I received the following errors:

/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c: In function 'do_fork_task':
/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c:1342:
warning: format '%1lx' expects type 'long unsigned int', but argument 3 has type 'u64' /scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c: At top level:
/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c:1416:
error: conflicting types for 'do_fork'

/scratch/kathys/containers/kernel_trees/upstream/include/linux/sched.h:1868:
error: previous declaration of 'do_fork' was here

I changed the patch as follows so it would apply properly, but I can't work out why it breaks now, but not previously:

Original patch from 2.6.26-rc2-mm1-lxc3:

(quoted diff) +2 -7

@@ -1307,13 +1313,8 @@ static int fork_traceflag(unsigned clone
 return 0;
 }
 
-/*
- * Ok, this is the main fork-routine.
- *
- * It copies the process, and if successful kick-starts
- * it and waits for it to finish using the VM if required.
- */
-long do_fork(unsigned long clone_flags,
+long do_fork_task(struct cgroup *cgroup,
+ unsigned long clone_flags,
 unsigned long stack_start,
 struct pt_regs *regs,
 unsigned long stack_size,

Changed to:

(quoted diff) +2 -7

@@ -1308,13 +1314,8 @@ static int fork_traceflag(u64 clone_flag
 return 0;
 }
 
-/*
- * Ok, this is the main fork-routine.
- *
- * It copies the process, and if successful kick-starts
- * it and waits for it to finish using the VM if required.
- */
-long do_fork(u64 clone_flags,
+long do_fork_task(struct cgroup *cgroup,
+ u64 clone_flags,
 unsigned long stack_start,
 struct pt_regs *regs,
 unsigned long stack_size,

Quoted 11 lines

thanks,
-serge
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers

Rather than just documenting this in the readme, actually spit out a warning on it.

You can pull from: git://git.sr71.net/~dave/cryo-forhallyn.git/

-- Dave

Dave Hansen [[email protected]] wrote:
| Rather than just documenting this in the readme, actually spit | out a warning on it.

Why not just bail out ? Its mostly unreliable at that point anyway. Besides, the warning can get buried in lot of other output.
---

Quoted 1 line

From 84d005031a8a17bdca62dc541c296a3bea74658c Mon Sep 17 00:00:00 2001

From: Sukadev Bhattiprolu <[email protected]>
Date: Wed, 11 Jun 2008 11:22:17 -0700
Subject: [PATCH] cryo currently requires randomize_va_space to be 0. Fail if it is not.

---
 cr.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

cr.c +0 -0

index db3ada0..217c5e7 100644

Quoted 87 lines · kathys

Quoted 63 lines · Serge E. Hallyn

Quoted 46 lines · Nadia Derbey

Quoted 32 lines · Serge E. Hallyn

Quoted 17 lines · Nadia Derbey

Quoted 13 lines · Cedric Le Goater

Quoted 6 lines

Looks like it has worked for me (msg1 creates 1000 msg queues, sleeps for a while and then removes the msg queues).

cool. which kernel are you using ?

C.

Quoted 7 lines

2.6.25-mm1 (sorry for the late answer - plenty of days off in France in May).

Regards,
Nadia

I'm playing with features in cryo, and keeping a git tree at:

git://git.sr71.net/~hallyn/cryodev.git

Quoted 11 lines

It's meant to exploit the extras which are in the -lxc kernel at lxc.sf.net. Current latest kernel patch is at
http://lxc.sourceforge.net/patches/2.6.26/2.6.26-rc2-mm1-lxc4/ This -lxc tree contains, for instance, Nadia's next_id patches, exploitation for setting ids for sysvipc and for tasks at fork, and updated ipc_setall patches (also using next_id). The version of cryo in my git tree exploits these. If you're root when you restart a task, it will clone a new set of namespaces and recreate your sysvipc objects, and it will reset your pids (even if you're not root if the pids are available).

Quoted 10 lines

Serge,

I noticed that the sys_hijack() has disappeared from the lxc dev
tree: would you mind putting it back. I think it might be useful if we want to start a task in a newly defined cgroup, checkpoint it and then try to restart it. We will need to 'join' the restarted container to check if everything has correctly be restarted. Or may be is there another way to do that?

Well it *could* be done similar to how cryo itself works, by ptracing the destination task and making it fork a task which execve()s a process to do the querying. But yuck.

Quoted 7 lines

Kathy, I'm sorry, I know I asked you to take sys_hijack() out. Could you please put it back in? Preferably at the end of the queue, as I don't want other patches having to be ported on top of it since its future is very suspect... Let me know if you have trouble porting it, but it should be pretty simple.

I added sys_hijack() (namespaces-introduce-sys_hijack.patch) back, with a little bit of massaging to get it to port properly (as Cedrics clone64-change-clone_flag-type-to-u64.patch changes unsigned long to u64) but was unable to compile. I received the following errors:

/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c: In function 'do_fork_task':
/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c:1342:
warning: format '%1lx' expects type 'long unsigned int', but argument 3 has type 'u64' /scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c: At top level:
/scratch/kathys/containers/kernel_trees/upstream/kernel/fork.c:1416:
error: conflicting types for 'do_fork'

Quoted 3 lines

/scratch/kathys/containers/kernel_trees/upstream/include/linux/sched.h:1868:
error: previous declaration of 'do_fork' was here

Did you change the do_fork definition in sched.h?

-serge

Quoted 53 lines

I changed the patch as follows so it would apply properly, but I can't work out why it breaks now, but not previously:

Original patch from 2.6.26-rc2-mm1-lxc3:

(quoted diff) +2 -7

@@ -1307,13 +1313,8 @@ static int fork_traceflag(unsigned clone
 return 0;
 }
 
-/*
- * Ok, this is the main fork-routine.
- *
- * It copies the process, and if successful kick-starts
- * it and waits for it to finish using the VM if required.
- */
-long do_fork(unsigned long clone_flags,
+long do_fork_task(struct cgroup *cgroup,
+ unsigned long clone_flags,
 unsigned long stack_start,
 struct pt_regs *regs,
 unsigned long stack_size,

Changed to:

(quoted diff) +2 -7

@@ -1308,13 +1314,8 @@ static int fork_traceflag(u64 clone_flag
 return 0;
 }
 
-/*
- * Ok, this is the main fork-routine.
- *
- * It copies the process, and if successful kick-starts
- * it and waits for it to finish using the VM if required.
- */
-long do_fork(u64 clone_flags,
+long do_fork_task(struct cgroup *cgroup,
+ u64 clone_flags,
 unsigned long stack_start,
 struct pt_regs *regs,
 unsigned long stack_size,

Quoted 13 lines

thanks,
-serge
_______________________________________________
Containers mailing list Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linux-foundation.org/mailman/listinfo/containers