: First try at integrating LXC and USER-CR
cover : First try at integrating LXC and USER-CR
Following two sets of patches is an early attempt to integrate LXC and USER-CR.
Overview:
Have USER-CR export the core checkpoint and restart functionality into a library (/lib/libcheckpoint.a and <usercr.h>) and have LXC link with this library.
TODO:
1. For now, libcheckpoint.a implements only the restart functionality and so only lxc_restart command is implemented. Implementing the checkpoint functionality and lxc_checkpoint command can be done similarly and is hopefully easier than the restart functionality.
2. The restart() functionality in user-cr makes extensive use of global variables and debug code. The API must be extended to properly include these variables/debug code in the API.
Similarly, the 'struct restart_args' may need to be sanitized for use in a formal API.
3. lxc_restart command restarts entire containers only (specifically it simulates the --pidns --pids --mount-pty arguments to /bin/restart).
4. Link lxc_restart and lxc_checkpoint with the shared library liblxc.so (currently links statically)
5. ...
STATUS:
I was able to checkpoint/restart a simple '/bin/sleep 1000' LXC container, except for a cgroup naming issue after restart (see below).
STEPS:
1. [USER-CR] Build/install /lib/libcheckpoint.a, /usr/include/usercr.h
1.1 Apply the attached [user-cr] patches to the user-cr git tree
(I tested with following commit as base)
commit 67cfee9329670ab28eb1a52e94745252b614718f
Author: Oren Laadan <[email protected]>
Date: Mon Feb 22 18:00:06 2010 -0500
1.2 Build/install user-cr binaries/libraries/includes
$ make all
$ make install
This should install /lib/libcheckpoint.a and /usr/include/usercr.h
2. [LXC] Build lxc_restart using USER-CR API (usercr.h, libcheckpoint.a)
2.1 Apply attached [lxc] patches to Daniel Lezcano's lxc.git tree (0.6.5)
2.2 Build lxc_restart (this uses static linking for now)
$ make -f Makefile2 lxc_restart
3. Create and checkpoint a simple LXC container
$ lxc-execute --name foo --rcfile lxc-macvlan.conf -- /bin/sleep 1000
$ lxc-freeze --name foo
TODO: lxc_checkpoint --name foo should checkpoint the container, For now, use "lxc-ps --name foo" to find pid of lxc-init and checkpoint using: $ /bin/checkpoint --output=/tmp/sleep.ckpt <pid-of-lxc-init> $ lxc-unfreeze --name foo $ lxc-stop --name foo
4. Restart a checkpointed LXC container
$ ./lxc_restart --statefile /tmp/sleep.ckpt --name bar
# Test some common lxc commands after restart
$ lxc-ps --name "bar/1" CONTAINER PID TTY TIME CMD bar/1 8511 ? 00:00:00 lxc-init bar/1 8512 ? 00:00:00 sleep
$ lxc-freeze --name "bar/1"
$ grep State /proc/8511/status State: D (disk sleep)
$ grep State /proc/8512/status State: D (disk sleep)
NOTE: For some reason, the container name after restart is "bar/1"
instead of "bar". Due to this, when the lxc_restart is exiting, I get a "-EBUSY - failed to remove "/cgroup/bar" error. I need to fix this still.
A message that is not in the archive (sent off-list or not yet received): 20100224083452.GB18758-r/[email protected]
PATCH 1/4 Move common definitions to restart.h
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 15:25:31 -0800
Subject: [PATCH 1/4][user-cr] Move common definitions to restart.h
To make restart() a library interface, the main() and related code in restart.c should be moved to a separate file. To prepare for the move, move some common definitions to a new file, 'restart.h'
--- restart.c | 44 ++------------------------------------------ restart.h | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 42 deletions(-) create mode 100644 restart.h
restart.c +2 -42
@@ -39,6 +39,7 @@#include "eclone.h" #include "genstack.h" #include "compat.h" +#include "restart.h" static char usage_str[] ="usage: restart [opts]\n"
@@ -128,12 +129,7 @@ static char usage_str[] =printf (__VA_ARGS__); \ }while (0 ) -#define SIGNAL_ENTRY (signal) { SIG ## signal, #signal } - -struct { -int signum; -char *sigstr; -} signal_array[] = { +struct signal_array signal_array[] = { {0 ,"NONE" },SIGNAL_ENTRY (ALRM ),SIGNAL_ENTRY (HUP ),
@@ -196,7 +192,6 @@ inline static int restart(pid_t pid, int fd, unsigned long flags, int logfd)return syscall (__NR_restart, pid, fd, flags, logfd); } -#define BUFSIZE (4 * 4096)struct hashent {long key;
@@ -205,29 +200,6 @@ struct hashent { };struct task ; -struct ckpt_ctx ; - -struct task { -int flags;/* state and (later) actions */ - -struct task *children;/* pointers to first child, next and prev */ -struct task *next_sib;/* sibling, and the creator of a process */ -struct task *prev_sib; -struct task *creator; - -struct task *phantom;/* pointer to place-holdler task (if any) */ - -pid_t pid;/* process IDs, our bread-&-butter */ -pid_t ppid; -pid_t tgid; -pid_t sid; - -pid_t rpid;/* [restart without vpids] actual (real) pid */ - -struct ckpt_ctx *ctx;/* points back to the c/r context */ - -pid_t real_parent;/* pid of task's real parent */ -};/* zero_task represents creator of root_task (all pids 0) */ struct task zero_task;
@@ -360,18 +332,6 @@ struct args {int keep_lsm; }; -#define CKPT_COND_PIDZERO 0x1 -#define CKPT_COND_MNTPROC 0x2 -#define CKPT_COND_MNTPTY 0x4 - -#define CKPT_COND_NONE 0 -#define CKPT_COND_ANY ULONG_MAX - -/* default for skip/warn/fail */ -#define CKPT_COND_WARN (CKPT_COND_MNTPROC | \ - CKPT_COND_MNTPTY) -#define CKPT_COND_FAIL (CKPT_COND_NONE) -static void usage (char *str) {fprintf (stderr,"%s" , str);
restart.h new file +45 -0
45 lines not on this page · raw message
Signature · 1 line
1.6.6.1
A message that is not in the archive (sent off-list or not yet received): 20100224083534.GC18758-r/[email protected]
PATCH 1/1 c/r: fix regression (in "fix scheduling in atomic while restoring ipc shm")
So... here's a simple fix to a silly regression - this should bring an end to my little fiasco.
--- ipc/checkpoint_msg.c | 2 +- ipc/checkpoint_sem.c | 2 +- ipc/checkpoint_shm.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
ipc/checkpoint_msg.c +1 -1
@@ -359,7 +359,7 @@ int restore_ipc_msg(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&msg_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&msg_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); msq = container_of(ipc, struct msg_queue, q_perm);
ipc/checkpoint_sem.c +1 -1
@@ -216,7 +216,7 @@ int restore_ipc_sem(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&sem_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&sem_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); sem = container_of(ipc, struct sem_array, sem_perm);
ipc/checkpoint_shm.c +1 -1
@@ -270,7 +270,7 @@ int restore_ipc_shm(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&shm_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&shm_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); shp = container_of(ipc, struct shmid_kernel, shm_perm);
Signature · 1 line
1.6.3.3
A message that is not in the archive (sent off-list or not yet received): 1267574649-14269-1-git-send-email-orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org
Serge E. Hallyn
Tested-byRe: [PATCH] c/r: fix regression (in "fix scheduling in atomic while restoring ipc shm")
Quoted 4 lines · Oren Laadan
So... here's a simple fix to a silly regression - this should bring an end to my little fiasco.
Signed-off-by Oren Laadan <[email protected]>
Quoted 47 lines
--- ipc/checkpoint_msg.c | 2 +- ipc/checkpoint_sem.c | 2 +- ipc/checkpoint_shm.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)ipc/checkpoint_msg.c +1
-1@@ -359,7 +359,7 @@ int restore_ipc_msg(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&msg_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&msg_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); msq = container_of(ipc, struct msg_queue, q_perm);ipc/checkpoint_sem.c +1
-1@@ -216,7 +216,7 @@ int restore_ipc_sem(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&sem_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&sem_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); sem = container_of(ipc, struct sem_array, sem_perm);ipc/checkpoint_shm.c +1
-1@@ -270,7 +270,7 @@ int restore_ipc_shm(struct ckpt_ctx *ctx, struct ipc_namespace *ns)* ipc-ns, we will need to re-examine this. */ - ipc =idr_find (&shm_ids->ipcs_idr, h->perms.id); + ipc =idr_find (&shm_ids->ipcs_idr,ipcid_to_idx (h->perms.id));BUG_ON (!ipc); shp = container_of(ipc, struct shmid_kernel, shm_perm);Signature · 1 line
1.6.3.3
PATCH 2/4 Rename struct args to struct restart_args
In reply to a message not in the archiveFrom 8a6fad2170fbbeb7a5e44c32239f9703c0b7d2f9 Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 15:33:04 -0800
Subject: [PATCH 2/4][user-cr] Rename struct args to struct restart_args
Rename struct args to struct restart_args and move to a new header file, usercr.h. usercr.h will become part of the external API for application checkpoint restart code.
Also move the bulk of the restart code into a separate function, app_restart() and have main() call the function.
TODO:
Cleanup 'struct restart_args' for a more consistent external API.
--- restart.c | 77 ++++++++++++++++++++++++------------------------------------ usercr.h | 24 +++++++++++++++++++ 2 files changed, 55 insertions(+), 46 deletions(-) create mode 100644 usercr.h
restart.c +31 -46
@@ -40,6 +40,7 @@#include "genstack.h" #include "compat.h" #include "restart.h" +#include "usercr.h" static char usage_str[] ="usage: restart [opts]\n"
@@ -237,7 +238,7 @@ struct ckpt_ctx {char container[BUFSIZE ];char tree[BUFSIZE ];char buf[BUFSIZE ]; -struct args *args; +struct restart_args *args;char *freezer; };
@@ -310,28 +311,6 @@ struct pid_swap {pid_t new; }; -struct args { -int self; -int pids; -int pidns; -int no_pidns; -int inspect; -char *root; -int wait; -int mntns; -int mnt_pty; -int show_status; -int copy_status; -char *freezer; -char *input; -int infd; -char *logfile; -int logfd; -long warn; -long fail; -int keep_lsm; -}; -static void usage (char *str) {fprintf (stderr,"%s" , str);
@@ -382,7 +361,7 @@ static inline int ckpt_cond_fail(struct ckpt_ctx *ctx, long mask)return (ctx->args->fail & mask); } -static void parse_args (struct args *args,int argc,char *argv[]) +static void parse_args (struct restart_args *args,int argc,char *argv[]) {static struct option opts[] = { {"help" , no_argument,NULL ,'h' },
@@ -712,71 +691,77 @@ static int freezer_register(struct ckpt_ctx *ctx, pid_t pid)int main (int argc,char *argv[]) { +struct restart_args args; + +parse_args (&args, argc, argv); + +return app_restart (&args); +} + +int app_restart (struct restart_args *args) +{struct ckpt_ctx ctx; -struct args args;int ret;memset (&ctx,0 ,sizeof (ctx)); - -parse_args (&args, argc, argv); - ctx.args = &args; + ctx.args = args;/* input file ? */ -if (args.input) { - args.infd =open (args.input,O_RDONLY ,0 ); -if (args.infd <0 ) { +if (args->input) { + args->infd =open (args->input,O_RDONLY ,0 ); +if (args->infd <0 ) {perror ("open input file" );exit (1 ); } }/* input file descriptor (default: stdin) */ -if (args.infd >=0 ) { -if (dup2 (args.infd,STDIN_FILENO ) <0 ) { +if (args->infd >=0 ) { +if (dup2 (args->infd,STDIN_FILENO ) <0 ) {perror ("dup2 input file" );exit (1 ); } -if (args.infd !=STDIN_FILENO ) -close (args.infd); +if (args->infd !=STDIN_FILENO ) +close (args->infd); }/* (optional) log file */ -if (args.logfile) { - args.logfd =open (args.logfile, +if (args->logfile) { + args->logfd =open (args->logfile,O_RDWR |O_CREAT |O_EXCL ,0644 ); -if (args.logfd <0 ) { +if (args->logfd <0 ) {perror ("open log file" );exit (1 ); } }/* output file descriptor (default: none) */ -if (args.logfd <0 ) - args.logfd =CHECKPOINT_FD_NONE ; +if (args->logfd <0 ) + args->logfd =CHECKPOINT_FD_NONE ;/* freezer preparation */ -if (args.freezer &&freezer_prepare (&ctx) <0 ) +if (args->freezer &&freezer_prepare (&ctx) <0 )exit (1 );/* private mounts namespace ? */ -if (args.mntns &&unshare (CLONE_NEWNS |CLONE_FS ) <0 ) { +if (args->mntns &&unshare (CLONE_NEWNS |CLONE_FS ) <0 ) {perror ("unshare" );exit (1 ); }/* chroot ? */ -if (args.root &&chroot (args.root) <0 ) { +if (args->root &&chroot (args->root) <0 ) {perror ("chroot" );exit (1 ); }/* remount /dev/pts ? */ -if (args.mnt_pty &&ckpt_remount_devpts (&ctx) <0 ) +if (args->mnt_pty &&ckpt_remount_devpts (&ctx) <0 )exit (1 );/* self-restart ends here: */ -if (args.self) { -restart (getpid (),STDIN_FILENO ,RESTART_TASKSELF , args.logfd); +if (args->self) { +restart (getpid (),STDIN_FILENO ,RESTART_TASKSELF , args->logfd);/* reach here if restart(2) failed ! */ perror ("restart" );exit (1 );
usercr.h new file +24 -0
@@ -0,0 +1,24 @@ + +struct restart_args { +int self; +int pids; +int pidns; +int no_pidns; +int inspect; +char *root; +int wait; +int mntns; +int mnt_pty; +int show_status; +int copy_status; +char *freezer; +char *input; +int infd; +char *logfile; +int logfd; +long warn; +long fail; +int keep_lsm; +}; + +extern int app_restart (struct restart_args *args);
Signature · 1 line
1.6.6.1
PATCH 3/4 Move main() in restart.c to restart-main.c
In reply to a message not in the archiveFrom 6553b7da33f56e9cb6927e7469af22df56b7f55f Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 15:55:34 -0800
Subject: [PATCH 3/4][user-cr] Move main() in restart.c to restart-main.c
This would enable us to put restart.o into a library.
TODO:
restart.c still has a whole bunch of external variables that would need to be addressed (maybe encapsulate in structure)
--- Makefile | 2 + restart-main.c | 312 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ restart.c | 293 ---------------------------------------------------- 3 files changed, 314 insertions(+), 293 deletions(-) create mode 100644 restart-main.c
Makefile +2 -0
@@ -21,6 +21,8 @@ CFLAGS += -g $(WARNS) $(CKPT_INCLUDE) $(DEBUG) BIN_INSTALL_DIR = /bin LIB_INSTALL_DIR = /lib +restart: restart.o restart-main.o + ECLONE_PROGS = restart nsexec PROGS = checkpoint ckptinfo $(ECLONE_PROGS) LIB_ECLONE = libeclone.a
restart-main.c new file +312 -0
312 lines not on this page · raw message
restart.c +0 -293
323 lines not on this page · raw message
Signature · 1 line
1.6.6.1
PATCH 4/4 Rename libeclone.a to libcheckpoint.a
In reply to a message not in the archiveFrom c4063a8976fd8eca9b8d62a12b95c3125c8471c7 Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 16:25:36 -0800
Subject: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
Export app_restart() and usercr.h to user in a new library libcheckpoint.a. Until eclone() makes it into glibc, include eclone() also in this new library and remove the libeclone.a.
TODO:
- cr-tests should be modified to look for libcheckpoint.a instead of libeclone.a
- libcheckpoint.a should eventually include the app_checkpoint() interface. I am guessing extracting the app_checkpoint() from checkpoint.c would be similar to (or somewhat easier than) the way we extracted app_restart() from restart.c
--- Makefile | 33 ++++++++++++++++++++------------- restart.c | 6 ++++++ 2 files changed, 26 insertions(+), 13 deletions(-)
Makefile +20 -13
@@ -18,14 +18,14 @@ WARNS := -Wall -Wstrict-prototypes -Wno-trigraphs CFLAGS += -g $(WARNS) $(CKPT_INCLUDE) $(DEBUG) # install dir +INC_INSTALL_DIR = /usr/include BIN_INSTALL_DIR = /bin LIB_INSTALL_DIR = /lib -restart: restart.o restart-main.o - ECLONE_PROGS = restart nsexec PROGS = checkpoint ckptinfo $(ECLONE_PROGS) -LIB_ECLONE = libeclone.a +LIB_CHECKPOINT = libcheckpoint.a +INC_CHECKPOINT = usercr.h # other cleanup OTHER = ckptinfo_types.c
@@ -34,32 +34,37 @@ LDLIBS = -lm .PHONY: all distclean clean headers install -all: $(PROGS) +all: $(PROGS) $(LIB_CHECKPOINT) @make -C test -$(LIB_ECLONE): - ar ruv $(LIB_ECLONE) $^ +restart: restart-main.o + +$(PROGS): $(LIB_CHECKPOINT) + +$(LIB_CHECKPOINT): + ar ruv $(LIB_CHECKPOINT) $^ # restart needs to be thread-safe restart: CFLAGS += -D__REENTRANT -pthread +$(LIB_CHECKPOINT): restart.o + # eclone() is architecture specific ifneq ($(SUBARCH),) -$(ECLONE_PROGS): $(LIB_ECLONE) $(ECLONE_PROGS): CFLAGS += -DARCH_HAS_ECLONE -$(LIB_ECLONE): clone_$(SUBARCH).o genstack.o +$(LIB_CHECKPOINT): clone_$(SUBARCH).o genstack.o endif # on powerpc, need also assembly file ifeq ($(SUBARCH),ppc) CFLAGS += -m32 ASFLAGS += -m32 -$(LIB_ECLONE): clone_$(SUBARCH)_.o +$(LIB_CHECKPOINT): clone_$(SUBARCH)_.o endif ifeq ($(SUBARCH),ppc64) CFLAGS += -m64 ASFLAGS += -m64 -$(LIB_ECLONE): clone_$(SUBARCH)_.o +$(LIB_CHECKPOINT): clone_$(SUBARCH)_.o endif # ckptinfo dependencies
@@ -71,8 +76,10 @@ ckptinfo_types.c: $(CKPT_HEADERS) ckptinfo.py install: @echo /usr/bin/install -m 755 checkpoint restart nsexec ckptinfo $(BIN_INSTALL_DIR) @/usr/bin/install -m 755 checkpoint restart ckptinfo nsexec $(BIN_INSTALL_DIR) - @echo /usr/bin/install -m 755 $(LIB_ECLONE) $(LIB_INSTALL_DIR) - @/usr/bin/install -m 755 $(LIB_ECLONE) $(LIB_INSTALL_DIR) + @echo /usr/bin/install -m 755 $(INC_CHECKPOINT) $(INC_INSTALL_DIR) + @/usr/bin/install -m 755 $(INC_CHECKPOINT) $(INC_INSTALL_DIR) + @echo /usr/bin/install -m 755 $(LIB_CHECKPOINT) $(LIB_INSTALL_DIR) + @/usr/bin/install -m 755 $(LIB_CHECKPOINT) $(LIB_INSTALL_DIR) $(CKPT_HEADERS): %: ./scripts/extract-headers.sh -s $(KERNELSRC) -o ./include
@@ -83,5 +90,5 @@ distclean: clean @rm -f $(CKPT_HEADERS) clean: - @rm -f $(PROGS) $(LIB_ECLONE) $(OTHER) *~ *.o headers.h + @rm -f $(PROGS) $(LIB_CHECKPOINT) $(OTHER) *~ *.o headers.h @make -C test clean
restart.c +6 -0
@@ -530,6 +530,12 @@ int app_restart(struct restart_args *args) ret =ckpt_coordinator (&ctx); } +/* +* Hack for LXC - which currently only uses pidns == 1, pid0 == 1; +*/ +if (ret >=0 ) + ret = global_child_pid; +return ret; }
Signature · 1 line
1.6.6.1
A message that is not in the archive (sent off-list or not yet received): 20100224083726.GF18758-r/[email protected]
Re: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
Quoted 117 lines · Sukadev Bhattiprolu
Quoted 2 lines
From
c4063a8976fd8eca9b8d62a12b95c3125c8471c7Mon Sep 17 00:00:00 2001From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 16:25:36 -0800
Subject: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.aExport
app_restart()andusercr.hto user in a new library libcheckpoint.a. Untileclone()makes it into glibc, includeeclone()also in this new library and remove the libeclone.a.TODO:
- cr-tests should be modified to look for libcheckpoint.a instead of libeclone.a- libcheckpoint.a should eventually include the
app_checkpoint()interface. I am guessing extracting theapp_checkpoint()fromcheckpoint.cwould be similar to (or somewhat easier than) the way we extractedapp_restart()fromrestart.cSigned-off-by Sukadev Bhattiprolu <[email protected]>--- Makefile | 33 ++++++++++++++++++++------------- restart.c | 6 ++++++ 2 files changed, 26 insertions(+), 13 deletions(-)Makefile +20
-13@@ -18,14 +18,14 @@ WARNS := -Wall -Wstrict-prototypes -Wno-trigraphs CFLAGS += -g $(WARNS) $(CKPT_INCLUDE) $(DEBUG) # install dir +INC_INSTALL_DIR = /usr/include BIN_INSTALL_DIR = /bin LIB_INSTALL_DIR = /lib -restart: restart.o restart-main.o - ECLONE_PROGS = restart nsexec PROGS = checkpoint ckptinfo $(ECLONE_PROGS) -LIB_ECLONE = libeclone.a +LIB_CHECKPOINT = libcheckpoint.a +INC_CHECKPOINT = usercr.h # other cleanup OTHER = ckptinfo_types.c@@ -34,32 +34,37 @@ LDLIBS = -lm .PHONY: all distclean clean headers install -all: $(PROGS) +all: $(PROGS) $(LIB_CHECKPOINT) @make -C test -$(LIB_ECLONE): - ar ruv $(LIB_ECLONE) $^ +restart: restart-main.o + +$(PROGS): $(LIB_CHECKPOINT) + +$(LIB_CHECKPOINT): + ar ruv $(LIB_CHECKPOINT) $^ # restart needs to be thread-safe restart: CFLAGS += -D__REENTRANT -pthread +$(LIB_CHECKPOINT): restart.o + # eclone() is architecture specific ifneq ($(SUBARCH),) -$(ECLONE_PROGS): $(LIB_ECLONE) $(ECLONE_PROGS): CFLAGS += -DARCH_HAS_ECLONE -$(LIB_ECLONE): clone_$(SUBARCH).o genstack.o +$(LIB_CHECKPOINT): clone_$(SUBARCH).o genstack.o endif # on powerpc, need also assembly file ifeq ($(SUBARCH),ppc) CFLAGS += -m32 ASFLAGS += -m32 -$(LIB_ECLONE): clone_$(SUBARCH)_.o +$(LIB_CHECKPOINT): clone_$(SUBARCH)_.o endif ifeq ($(SUBARCH),ppc64) CFLAGS += -m64 ASFLAGS += -m64 -$(LIB_ECLONE): clone_$(SUBARCH)_.o +$(LIB_CHECKPOINT): clone_$(SUBARCH)_.o endif # ckptinfo dependencies@@ -71,8 +76,10 @@ ckptinfo_types.c: $(CKPT_HEADERS) ckptinfo.py install: @echo /usr/bin/install -m 755 checkpoint restart nsexec ckptinfo $(BIN_INSTALL_DIR) @/usr/bin/install -m 755 checkpoint restart ckptinfo nsexec $(BIN_INSTALL_DIR) - @echo /usr/bin/install -m 755 $(LIB_ECLONE) $(LIB_INSTALL_DIR) - @/usr/bin/install -m 755 $(LIB_ECLONE) $(LIB_INSTALL_DIR) + @echo /usr/bin/install -m 755 $(INC_CHECKPOINT) $(INC_INSTALL_DIR) + @/usr/bin/install -m 755 $(INC_CHECKPOINT) $(INC_INSTALL_DIR) + @echo /usr/bin/install -m 755 $(LIB_CHECKPOINT) $(LIB_INSTALL_DIR) + @/usr/bin/install -m 755 $(LIB_CHECKPOINT) $(LIB_INSTALL_DIR) $(CKPT_HEADERS): %: ./scripts/extract-headers.sh -s $(KERNELSRC) -o ./include@@ -83,5 +90,5 @@ distclean: clean @rm -f $(CKPT_HEADERS) clean: - @rm -f $(PROGS) $(LIB_ECLONE) $(OTHER) *~ *.o headers.h + @rm -f $(PROGS) $(LIB_CHECKPOINT) $(OTHER) *~ *.o headers.h @make -C test cleanrestart.c +0
-0
restart.c int app_restart(struct restart_args *args) · line 538
Quoted 9 lines
restart.c +6
-0@@ -530,6 +530,12 @@ int app_restart(struct restart_args *args) ret =ckpt_coordinator (&ctx); } +/* +* Hack for LXC - which currently only uses pidns == 1, pid0 == 1; +*/ +if (ret >=0 ) + ret = global_child_pid; +
Hmm? Maybe I'm looking at the wrong context, but isn't this specifically the existing pidns case? Should the second block ("new pidns with init") be the one to execute if your comment is correct?
Quoted 5 lines
restart.c +0
-0return ret; }Signature · 1 line
1.6.6.1
A message that is not in the archive (sent off-list or not yet received): 20100224151919.GB6425-r/[email protected]
Re: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
Serge E. Hallyn [serue-r/[email protected]] wrote:
| > diff --git a/restart.c b/restart.c | > index b1518f2..a163862 100644 | > --- a/restart.c | > +++ b/restart.c | > @@ -530,6 +530,12 @@ int app_restart(struct restart_args *args) | > ret = ckpt_coordinator(&ctx); | > } | > | > + /* | > + * Hack for LXC - which currently only uses pidns == 1, pid0 == 1; | > + */ | > + if (ret >= 0) | > + ret = global_child_pid; | > + | | Hmm? Maybe I'm looking at the wrong context, but isn't this specifically | the existing pidns case?
Yes it is.
| Should the second block ("new pidns with init") | be the one to execute if your comment is correct?
Oh, you mean I could just return the global_child_pid from within the "new pid ns with init" block ?. I could do that for now.
BTW, from a long-term API pov, should app_restart():
- support the other two types of restart ("new pidns without init" and "subtree in current pidns"). LXC may not use those, but any reason for the API to not support those cases ?
- If the API supports them, should app_restart() always return the pid of the coordinator (i.e global_child_pid). That way the caller knows the root of the restarted process-tree ?
- Also, like the kernel's API, how about app_checkpoint() and app_restart() take a logfp/logfd and we send all the debugging output to that file ?
Thanks,
Sukadev
| | > return ret; | > } | > | > -- | > 1.6.6.1
Re: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
In reply to a message not in the archiveQuoted 8 lines · Sukadev Bhattiprolu
From
c4063a8976fd8eca9b8d62a12b95c3125c8471c7Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 16:25:36 -0800
Subject: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.aExport
app_restart()andusercr.hto user in a new library libcheckpoint.a. Untileclone()makes it into glibc, includeeclone()also in this new library and remove the libeclone.a.
I thought libeclone.a is also used by the test suite ?
Oren.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
Quoted 8 lines · Oren Laadan [[email protected]] wrote:
Quoted 5 lines · Sukadev Bhattiprolu
From
c4063a8976fd8eca9b8d62a12b95c3125c8471c7Mon Sep 17 00:00:00 2001
From: Sukadev Bhattiprolu <[email protected]>
Date: Tue, 23 Feb 2010 16:25:36 -0800
Subject: [PATCH 4/4][user-cr] Rename libeclone.a to libcheckpoint.a
Quoted 5 lines
Quoted 3 lines
Export
app_restart()andusercr.hto user in a new library libcheckpoint.a. Untileclone()makes it into glibc, includeeclone()also in this new library and remove the libeclone.a.I thought libeclone.a is also used by the test suite ?
Yes and I have a todo in the patchset to fix cr-tests if/when we do end up changing the name.
libeclone.a looks like a short-term name (if/when eclone() is merged in libc, the library is no longer required).
libcheckpoint.a (or equivalent) has more long term need. If we are having a libcheckpoint, no point keeping libeclone around too ?
Sukadev
Quoted 31 lines · Sukadev Bhattiprolu
Following two sets of patches is an early attempt to integrate LXC and USER-CR.
Overview:
Have USER-CR export the core checkpoint and restart functionality into a library (
/lib/libcheckpoint.aand <usercr.h>) and have LXC link with this library.TODO:
1. For now, libcheckpoint.a implements only the restart functionality and so only
lxc_restartcommand is implemented. Implementing the checkpoint functionality andlxc_checkpointcommand can be done similarly and is hopefully easier than the restart functionality.2. The
restart()functionality in user-cr makes extensive use of global variables and debug code. The API must be extended to properly include these variables/debug code in the API.Similarly, the 'struct
restart_args' may need to be sanitized for use in a formal API.3.
lxc_restartcommand restarts entire containers only (specifically it simulates the --pidns --pids --mount-pty arguments to/bin/restart).4. Link
lxc_restartandlxc_checkpointwith the shared library liblxc.so (currently links statically)
Quoted 6 lines
5. ...
STATUS:
I was able to checkpoint/restart a simple '/bin/sleep1000' LXC container, except for a cgroup naming issue after restart (see below).
Very nice, Suka.
-serge
Quoted 9 lines · Sukadev Bhattiprolu
Following two sets of patches is an early attempt to integrate LXC and USER-CR.
Overview:
Have USER-CR export the core checkpoint and restart functionality into a library (
/lib/libcheckpoint.aand<usercr.h>) and have LXC link with this library.
suka,
if these patches were accepted in the user-cr git tree, it would be easier for us to follow with your experiments and move forward on some more tests.
cheers,
C.
Suka,
This is good stuff: convert restart to a library and intergate with LXC is excellent target.
We need to give a thought to the API design and the sort of functionality that the library will offer. I'd prefer to get a better view of the whole plan before committing to any incremental step.
For example:
* what prefix will the library calls have ? e.g. cr_checkpoint(), cr_restart() etc. Could also be checkpoint_... or ckpt_... In any case, must be consistent and unique, and also agree with the library name: libcheckpoint.a and usercr.h doesn't make sense to me.
* what arguments are really necessary in struct restart_args ? for example, a ->logfd is necessary, but ->logfile (pathname) is not (the caller should already open the file)
* the library should provide an api to initialize the default args (e.g. the logfd should be -1 by default), e.g. cr_init_restart_args()
* similarly, only constants and macros relevant to a caller should be exposed, not internal data structures or macros.
* verbosity, debugging, warn and fail: currently that can be configurable to some extent; but we should never impose on the caller a perror() - instead the caller should pass FILE * for stdout and stderr (if NULL - the library will keep silent).
* probably makes sense to add interface to freeze a process tree identified by its root pid, and thaw a process tree (or a cgroup).
* does the mnt-ns manipulation belong to cr_restart() ? perhaps make it a separate api in the library ?
* we need to change the way restart detects errors and cleans up from relying on signals to, e.g. on pipes. I've been wanting to do this for the longest time, and it will eliminate most (if not all) of the global variables.
Oren.
Quoted 105 lines · Sukadev Bhattiprolu
Following two sets of patches is an early attempt to integrate LXC and USER-CR.
Overview:
Have USER-CR export the core checkpoint and restart functionality into a library (
/lib/libcheckpoint.aand <usercr.h>) and have LXC link with this library.TODO:
1. For now, libcheckpoint.a implements only the restart functionality and so only
lxc_restartcommand is implemented. Implementing the checkpoint functionality andlxc_checkpointcommand can be done similarly and is hopefully easier than the restart functionality.2. The
restart()functionality in user-cr makes extensive use of global variables and debug code. The API must be extended to properly include these variables/debug code in the API.Similarly, the 'struct
restart_args' may need to be sanitized for use in a formal API.3.
lxc_restartcommand restarts entire containers only (specifically it simulates the --pidns --pids --mount-pty arguments to/bin/restart).4. Link
lxc_restartandlxc_checkpointwith the shared library liblxc.so (currently links statically)5. ...
STATUS:
I was able to checkpoint/restart a simple '/bin/sleep1000' LXC container, except for a cgroup naming issue after restart (see below).STEPS:
1. [USER-CR] Build/install
/lib/libcheckpoint.a,/usr/include/usercr.h1.1 Apply the attached [user-cr] patches to the user-cr git tree
(I tested with following commit as base)commit
67cfee9329670ab28eb1a52e94745252b614718f
Author: Oren Laadan <[email protected]>
Date: Mon Feb 22 18:00:06 2010 -05001.2 Build/install user-cr binaries/libraries/includes
$ make all$ make installThis should install
/lib/libcheckpoint.aand/usr/include/usercr.h2. [LXC] Build
lxc_restartusing USER-CR API (usercr.h, libcheckpoint.a)2.1 Apply attached [lxc] patches to Daniel Lezcano's lxc.git tree (0.6.5)
2.2 Build
lxc_restart(this uses static linking for now)$ make -f Makefile2 lxc_restart3. Create and checkpoint a simple LXC container
$ lxc-execute --name foo --rcfile lxc-macvlan.conf -- /bin/sleep 1000$ lxc-freeze --name fooTODO: lxc_checkpoint --name foo should checkpoint the container, For now, use "lxc-ps --name foo" to find pid of lxc-init and checkpoint using: $ /bin/checkpoint --output=/tmp/sleep.ckpt <pid-of-lxc-init> $ lxc-unfreeze --name foo $ lxc-stop --name foo4. Restart a checkpointed LXC container
$ ./lxc_restart --statefile /tmp/sleep.ckpt --name bar# Test some common lxc commands after restart
$ lxc-ps --name "bar/1" CONTAINER PID TTY TIME CMD bar/1 8511 ? 00:00:00 lxc-init bar/1 8512 ? 00:00:00 sleep$ lxc-freeze --name "bar/1"$ grep State /proc/8511/status State: D (disk sleep)$ grep State /proc/8512/status State: D (disk sleep)NOTE: For some reason, the container name after restart is "bar/1"
instead of "bar". Due to this, when thelxc_restartis exiting, I get a "-EBUSY- failed to remove "/cgroup/bar" error. I need to fix this still.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Oren Laadan [[email protected]] wrote:
Suka,
This is good stuff: convert restart to a library and intergate with LXC is excellent target.
We need to give a thought to the API design and the sort of functionality that the library will offer. I'd prefer to get a better view of the whole plan before committing to any incremental step.
Quoted 7 lines
For example:
* what prefix will the library calls have ? e.g.
cr_checkpoint(),cr_restart()etc. Could also becheckpoint_... orckpt_... In any case, must be consistent and unique, and also agree with the library name: libcheckpoint.a andusercr.hdoesn't make sense to me.
Agree. I have been worried about publishing an interface and getting stuck with a bad interface. But I have also been trying to prototype interface with LIBLXC available so did not yet spend lot of time on the interface.
I have been meaning to add a disclaimer to usercr.h that the structures will change without regard to backward compatibility
libcheckpoint.a looked like a reasonable name but for the header, we already have a checkpoint.h from the kernel :-) Open to new/better names/prefixes for the library calls.
Anyway, these names are just for the prototype and to start a discussion, My guess is that we are not ready to publish a formal interface at this point, but we can start moving towards that.
My current pass has been to eliminate all global references in app_restart() and app_checkpoint() (like perror(), stderr, stdout etc). while preserving the current behavior. Once we have an interface that LIBLXC can use, maybe we could focus more on stabilizing the interface ? It seemed to hard to do that with all the globals in the code.
Quoted 4 lines
* what arguments are really necessary in struct
restart_args? for example, a ->logfd is necessary, but ->logfile (pathname) is not (the caller should already open the file)
I have several small patches to clean this up. Was hoping to post later today. They remove ->logfile, ->input, ->output fromt he structures and just leave the relevant fds.
Quoted 11 lines
* the library should provide an api to initialize the default args (e.g. the logfd should be -1 by default), e.g.
cr_init_restart_args()* similarly, only constants and macros relevant to a caller should be exposed, not internal data structures or macros.
* verbosity, debugging, warn and fail: currently that can be configurable to some extent; but we should never impose on the caller a
perror()- instead the caller should pass FILE * for stdout and stderr (if NULL - the library will keep silent).
Yes, my current patchset uses a restart_args->ulogfd for the user-space error messages. The restart-main.c sets this field to fileno(stderr).
Quoted 3 lines
* probably makes sense to add interface to freeze a process tree identified by its root pid, and thaw a process tree (or a cgroup).
True. For the near-term, LIBLXC has a way to freeze/thaw the containers
Quoted 3 lines
* does the mnt-ns manipulation belong to
cr_restart()? perhaps make it a separate api in the library ?
Good point. We need to think about that.
Quoted 5 lines
* we need to change the way restart detects errors and cleans up from relying on signals to, e.g. on pipes. I've been wanting to do this for the longest time, and it will eliminate most (if not all) of the global variables.
Ok.
A message that is not in the archive (sent off-list or not yet received): 20100227001002.GA22965-r/[email protected]
Quoted 24 lines · Sukadev Bhattiprolu
Quoted 10 lines · Oren Laadan [[email protected]] wrote:
Suka,
This is good stuff: convert restart to a library and intergate with LXC is excellent target.
We need to give a thought to the API design and the sort of functionality that the library will offer. I'd prefer to get a better view of the whole plan before committing to any incremental step.
Quoted 7 lines
For example:
* what prefix will the library calls have ? e.g.
cr_checkpoint(),cr_restart()etc. Could also becheckpoint_... orckpt_... In any case, must be consistent and unique, and also agree with the library name: libcheckpoint.a andusercr.hdoesn't make sense to me.Agree. I have been worried about publishing an interface and getting stuck with a bad interface. But I have also been trying to prototype interface with LIBLXC available so did not yet spend lot of time on the interface.
Quoted 6 lines
I have been meaning to add a disclaimer to
usercr.hthat the structures will change without regard to backward compatibilitylibcheckpoint.a looked like a reasonable name but for the header, we already have a
checkpoint.hfrom the kernel :-) Open to new/better names/prefixes for the library calls.
we have sys/checkpoint.h, so it makes sense to have checkpoint.h :)
Quoted 12 lines
Anyway, these names are just for the prototype and to start a discussion, My guess is that we are not ready to publish a formal interface at this point, but we can start moving towards that.
My current pass has been to eliminate all global references in
app_restart()andapp_checkpoint()(likeperror(), stderr, stdout etc). while preserving the current behavior. Once we have an interface that LIBLXC can use, maybe we could focus more on stabilizing the interface ? It seemed to hard to do that with all the globals in the code.
Quoted 7 lines
Quoted 3 lines
* what arguments are really necessary in struct
restart_args? for example, a ->logfd is necessary, but ->logfile (pathname) is not (the caller should already open the file)I have several small patches to clean this up. Was hoping to post later today. They remove ->logfile, ->input, ->output fromt he structures and just leave the relevant fds.
What I suggest is to avoid unnecessary work and fixes, it's worth our while to discuss what could be the api and functionality even before getting into the implementation details.etc.
Also, two important points that I had meant to mention are:
* versioning: we should provision for api extensions, e.g. in the struct restart_args, or at the very least a way to figure out the version.
* hooks / plugins: allow the user to request hooks or plugins, for example after checkpoint completes but before applications are thawed, or in restart after the tree is created but before tasks call sys_restart(). another example - handling the mount- namespace (and remounting the pty) could be a plugin.
Oren.
Quoted 33 lines
Quoted 11 lines
* the library should provide an api to initialize the default args (e.g. the logfd should be -1 by default), e.g.
cr_init_restart_args()* similarly, only constants and macros relevant to a caller should be exposed, not internal data structures or macros.
* verbosity, debugging, warn and fail: currently that can be configurable to some extent; but we should never impose on the caller a
perror()- instead the caller should pass FILE * for stdout and stderr (if NULL - the library will keep silent).Yes, my current patchset uses a
restart_args->ulogfdfor the user-space error messages. Therestart-main.csets this field to fileno(stderr).Quoted 2 lines
* probably makes sense to add interface to freeze a process tree identified by its root pid, and thaw a process tree (or a cgroup).
True. For the near-term, LIBLXC has a way to freeze/thaw the containers
Quoted 2 lines
* does the mnt-ns manipulation belong to
cr_restart()? perhaps make it a separate api in the library ?Good point. We need to think about that.
Quoted 4 lines
* we need to change the way restart detects errors and cleans up from relying on signals to, e.g. on pipes. I've been wanting to do this for the longest time, and it will eliminate most (if not all) of the global variables.
Ok.
Suka,
To make these patches available for those who want to try out lxc + c/r, I pulled them to a separate branch: ckpt-v19-suka.
(The last two did not apply cleanly because of recent changes to the Makefile so I merged them manually - let me know if it breaks).
Oren
Quoted 105 lines · Sukadev Bhattiprolu
Following two sets of patches is an early attempt to integrate LXC and USER-CR.
Overview:
Have USER-CR export the core checkpoint and restart functionality into a library (
/lib/libcheckpoint.aand <usercr.h>) and have LXC link with this library.TODO:
1. For now, libcheckpoint.a implements only the restart functionality and so only
lxc_restartcommand is implemented. Implementing the checkpoint functionality andlxc_checkpointcommand can be done similarly and is hopefully easier than the restart functionality.2. The
restart()functionality in user-cr makes extensive use of global variables and debug code. The API must be extended to properly include these variables/debug code in the API.Similarly, the 'struct
restart_args' may need to be sanitized for use in a formal API.3.
lxc_restartcommand restarts entire containers only (specifically it simulates the --pidns --pids --mount-pty arguments to/bin/restart).4. Link
lxc_restartandlxc_checkpointwith the shared library liblxc.so (currently links statically)5. ...
STATUS:
I was able to checkpoint/restart a simple '/bin/sleep1000' LXC container, except for a cgroup naming issue after restart (see below).STEPS:
1. [USER-CR] Build/install
/lib/libcheckpoint.a,/usr/include/usercr.h1.1 Apply the attached [user-cr] patches to the user-cr git tree
(I tested with following commit as base)commit
67cfee9329670ab28eb1a52e94745252b614718f
Author: Oren Laadan <[email protected]>
Date: Mon Feb 22 18:00:06 2010 -05001.2 Build/install user-cr binaries/libraries/includes
$ make all$ make installThis should install
/lib/libcheckpoint.aand/usr/include/usercr.h2. [LXC] Build
lxc_restartusing USER-CR API (usercr.h, libcheckpoint.a)2.1 Apply attached [lxc] patches to Daniel Lezcano's lxc.git tree (0.6.5)
2.2 Build
lxc_restart(this uses static linking for now)$ make -f Makefile2 lxc_restart3. Create and checkpoint a simple LXC container
$ lxc-execute --name foo --rcfile lxc-macvlan.conf -- /bin/sleep 1000$ lxc-freeze --name fooTODO: lxc_checkpoint --name foo should checkpoint the container, For now, use "lxc-ps --name foo" to find pid of lxc-init and checkpoint using: $ /bin/checkpoint --output=/tmp/sleep.ckpt <pid-of-lxc-init> $ lxc-unfreeze --name foo $ lxc-stop --name foo4. Restart a checkpointed LXC container
$ ./lxc_restart --statefile /tmp/sleep.ckpt --name bar# Test some common lxc commands after restart
$ lxc-ps --name "bar/1" CONTAINER PID TTY TIME CMD bar/1 8511 ? 00:00:00 lxc-init bar/1 8512 ? 00:00:00 sleep$ lxc-freeze --name "bar/1"$ grep State /proc/8511/status State: D (disk sleep)$ grep State /proc/8512/status State: D (disk sleep)NOTE: For some reason, the container name after restart is "bar/1"
instead of "bar". Due to this, when thelxc_restartis exiting, I get a "-EBUSY- failed to remove "/cgroup/bar" error. I need to fix this still.