Resending, c/r series v2

Hi, this series hopefully in a good shape

- sys_kcmp now depends on CONFIG_CHECKPOINT_RESTORE

- the extension of /proc/pid/stat now done against linux-next/master

Please letme know if I've missed something.

Thanks,
Cyrill

When we do checkpoint of a task we need to know the list of children the task, has but there is no easy and fast way to generate reverse parent->children chain from arbitrary <pid> (while a parent pid is provided in "PPid" field of /proc/<pid>/status).

So instead of walking over all pids in the system (creating one big process tree in memory, just to figure out which children a task has) -- we add explicit /proc/<pid>/task/<tid>/children entry, because the kernel already has this kind of information but it is not yet exported.

This is a first level children, not the whole process tree.

---
 Documentation/filesystems/proc.txt |   18 +++++
 fs/proc/array.c                    |  123 +++++++++++++++++++++++++++++++++++++
 fs/proc/base.c                     |    3 
 fs/proc/internal.h                 |    1 
 4 files changed, 145 insertions(+)

Index: linux-2.6.git/Documentation/filesystems/proc.txt
===================================================================
@@ -40,6 +40,7 @@ Table of Contents
   3.4	/proc/<pid>/coredump_filter - Core dump filtering settings
   3.5	/proc/<pid>/mountinfo - Information about mounts
   3.6	/proc/<pid>/comm  & /proc/<pid>/task/<tid>/comm
+  3.7   /proc/<pid>/task/<tid>/children - Information about task children
 
   4	Configuring procfs
   4.1	Mount options

@@ -1549,6 +1550,23 @@ then the kernel's TASK_COMM_LEN (current
 comm value.
 
 
+3.7	/proc/<pid>/task/<tid>/children - Information about task children
+-------------------------------------------------------------------------
+This file provides a fast way to retrieve first level children pids
+of a task pointed by <pid>/<tid> pair. The format is a space separated
+stream of pids.
+
+Note the "first level" here -- if a child has own children they will
+not be listed here, one needs to read /proc/<children-pid>/task/<tid>/children
+to obtain the descendants.
+
+Since this interface is intended to be fast and cheap it doesn't
+guarantee to provide precise results and some children might be
+skipped, especially if they've exited right after we printed their
+pids, so one need to either stop or freeze processes being inspected
+if precise results are needed.
+
+
 ------------------------------------------------------------------------------
 Configuring procfs
 ------------------------------------------------------------------------------

Index: linux-2.6.git/fs/proc/base.c
===================================================================
@@ -3350,6 +3350,9 @@ static const struct pid_entry tid_base_s
 	ONE("stat",      S_IRUGO, proc_tid_stat),
 	ONE("statm",     S_IRUGO, proc_pid_statm),
 	REG("maps",      S_IRUGO, proc_maps_operations),
+#ifdef CONFIG_CHECKPOINT_RESTORE
+	REG("children",  S_IRUGO, proc_tid_children_operations),
+#endif
 #ifdef CONFIG_NUMA
 	REG("numa_maps", S_IRUGO, proc_numa_maps_operations),
 #endif

Index: linux-2.6.git/fs/proc/internal.h
===================================================================
@@ -53,6 +53,7 @@ extern int proc_pid_statm(struct seq_fil
 				struct pid *pid, struct task_struct *task);
 extern loff_t mem_lseek(struct file *file, loff_t offset, int orig);
 
+extern const struct file_operations proc_tid_children_operations;
 extern const struct file_operations proc_maps_operations;
 extern const struct file_operations proc_numa_maps_operations;
 extern const struct file_operations proc_smaps_operations;

While doing the checkpoint-restore in the user space one need to determine whether various kernel objects (like mm_struct-s of file_struct-s) are shared between tasks and restore this state.

The 2nd step can be solved by using appropriate CLONE_ flags and the unshare syscall, while there's currently no ways for solving the 1st one.

One of the ways for checking whether two tasks share e.g. mm_struct is to provide some mm_struct ID of a task to its proc file, but showing such info considered to be not that good for security reasons.

Thus after some debates we end up in conclusion that using that named 'comparison' syscall might be the best candidate. So here is it -- __NR_kcmp.

It takes up to 5 arguments - the pids of the two tasks (which characteristics should be compared), the comparison type and (in case of comparison of files) two file descriptors.

Lookups for pids are done in the caller's PID namespace only.

At moment only x86 is supported and tested.

---
 arch/x86/syscalls/syscall_32.tbl         |    1 
 arch/x86/syscalls/syscall_64.tbl         |    1 
 include/linux/kcmp.h                     |   17 +++
 include/linux/syscalls.h                 |    2 
 kernel/Makefile                          |    3 
 kernel/kcmp.c                            |  155 +++++++++++++++++++++++++++++++
 kernel/sys_ni.c                          |    3 
 tools/testing/selftests/kcmp/Makefile    |   36 +++++++
 tools/testing/selftests/kcmp/kcmp_test.c |   84 ++++++++++++++++
 tools/testing/selftests/run_tests        |    2 
 10 files changed, 303 insertions(+), 1 deletion(-)

Index: linux-2.6.git/arch/x86/syscalls/syscall_32.tbl
===================================================================
@@ -355,3 +355,4 @@
 346	i386	setns			sys_setns
 347	i386	process_vm_readv	sys_process_vm_readv		compat_sys_process_vm_readv
 348	i386	process_vm_writev	sys_process_vm_writev		compat_sys_process_vm_writev
+349	i386	kcmp			sys_kcmp

Index: linux-2.6.git/arch/x86/syscalls/syscall_64.tbl
===================================================================
@@ -318,3 +318,4 @@
 309	64	getcpu			sys_getcpu
 310	64	process_vm_readv	sys_process_vm_readv
 311	64	process_vm_writev	sys_process_vm_writev
+312	64	kcmp			sys_kcmp

Index: linux-2.6.git/include/linux/kcmp.h
===================================================================
@@ -0,0 +1,17 @@
+#ifndef _LINUX_KCMP_H
+#define _LINUX_KCMP_H
+
+/* Comparison type */
+enum kcmp_type {
+	KCMP_FILE,
+	KCMP_VM,
+	KCMP_FILES,
+	KCMP_FS,
+	KCMP_SIGHAND,
+	KCMP_IO,
+	KCMP_SYSVSEM,
+
+	KCMP_TYPES,
+};
+
+#endif /* _LINUX_KCMP_H */

Index: linux-2.6.git/include/linux/syscalls.h
===================================================================
@@ -857,4 +857,6 @@ asmlinkage long sys_process_vm_writev(pi
 				      unsigned long riovcnt,
 				      unsigned long flags);
 
+asmlinkage long sys_kcmp(pid_t pid1, pid_t pid2, int type,
+			 unsigned long idx1, unsigned long idx2);
 #endif

Index: linux-2.6.git/kernel/Makefile
===================================================================
@@ -25,6 +25,9 @@ endif
 obj-y += sched/
 obj-y += power/
 
+ifeq ($(CONFIG_CHECKPOINT_RESTORE),y)
+obj-$(CONFIG_X86) += kcmp.o
+endif
 obj-$(CONFIG_FREEZER) += freezer.o
 obj-$(CONFIG_PROFILING) += profile.o
 obj-$(CONFIG_SYSCTL_SYSCALL_CHECK) += sysctl_check.o

Index: linux-2.6.git/kernel/sys_ni.c
===================================================================
@@ -203,3 +203,6 @@ cond_syscall(sys_fanotify_mark);
 cond_syscall(sys_name_to_handle_at);
 cond_syscall(sys_open_by_handle_at);
 cond_syscall(compat_sys_open_by_handle_at);
+
+/* compare kernel pointers */
+cond_syscall(sys_kcmp);

Index: linux-2.6.git/tools/testing/selftests/kcmp/Makefile
===================================================================
@@ -0,0 +1,36 @@
+ifeq ($(strip $(V)),)
+	E = @echo
+	Q = @
+else
+	E = @\#
+	Q =
+endif
+export E Q
+
+uname_M := $(shell uname -m 2>/dev/null || echo not)
+ARCH ?= $(shell echo $(uname_M) | sed -e s/i.86/i386/)
+ifeq ($(ARCH),i386)
+        ARCH := X86
+	CFLAGS := -DCONFIG_X86_32 -D__i386__
+endif
+ifeq ($(ARCH),x86_64)
+	ARCH := X86
+	CFLAGS := -DCONFIG_X86_64 -D__x86_64__
+endif
+
+CFLAGS += -I../../../../arch/x86/include/generated/
+CFLAGS += -I../../../../include/
+CFLAGS += -I../../../../usr/include/
+
+all:
+ifeq ($(ARCH),X86)
+	$(E) "  CC run_test"
+	$(Q) gcc $(CFLAGS) kcmp_test.c -o run_test
+else
+	$(E) "Not an x86 target, can't build kcmp selftest"
+endif
+
+clean:
+	$(E) "  CLEAN"
+	$(Q) rm -fr ./run_test
+	$(Q) rm -fr ./test-file

Index: linux-2.6.git/tools/testing/selftests/run_tests
===================================================================
@@ -1,6 +1,6 @@
 #!/bin/bash
 
-TARGETS=breakpoints
+TARGETS="breakpoints kcmp"
 
 for TARGET in $TARGETS
 do

On Tue, Feb 14, 2012 at 03:13:12PM -0800, Andrew Morton wrote:
...

Hmm, thanks for pointing.

make V=1 will show the command line, I tried to make the output to look the same as in case of building kernel.

...

Initially I made it to look into arch/x86 directory and find __NR_kcmp definition there but then I realised (and someone pointed me as well) that headers should be taken from those which are generated by kernel, ie the result of 'make headers_install'. So if you run make headers_install then the test program should work. If this is wrong, I can fix it and make a patch on top.

Ie instead of

+#include <linux/unistd.h>

there will be

+#ifdef CONFIG_X86_64
+#include <asm/unistd_64.h>
+#else
+#include <asm/unistd_32.h>
+#endif

Cyrill

What? <linux/unistd.h> should be used, and it should handle the.inclusions.

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Peter, I maybe missing something which directories I should add to -I option, letme recheck (but without headers_intall target the syscall number is unresolved for me).

Cyrill

Ah, I missed to add

+CFLAGS += -I../../../../arch/x86/include/

into Makefile. Andrew, should I send update as a separate patch?

Cyrill

That worked.

What's with all those warnings, btw?

From: Andrew Morton <[email protected]>
Subject: syscalls-x86-add-__nr_kcmp-syscall-v8-fix-fix-fix

kcmp_test.c: In function 'main':
kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 4 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 5 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 6 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 7 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 8 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 9 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 10 has type 'long int' kcmp_test.c:69: warning: format '%2d' expects type 'int', but argument 11 has type 'long int'

---

 tools/testing/selftests/kcmp/kcmp_test.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

@@ -54,8 +54,9 @@ int main(int argc, char **argv)
 		}
 
 		/* An example of output and arguments */
-		printf("pid1: %6d pid2: %6d FD: %2d FILES: %2d VM: %2d FS: %2d "
-		       "SIGHAND: %2d IO: %2d SYSVSEM: %2d INV: %2d\n",
+		printf("pid1: %6d pid2: %6d FD: %2ld FILES: %2ld VM: %2ld "
+		       "FS: %2ld SIGHAND: %2ld IO: %2ld SYSVSEM: %2ld "
+		       "INV: %2ld\n",
 		       pid1, pid2,
 		       sys_kcmp(pid1, pid2, KCMP_FILE,		fd1, fd2),
 		       sys_kcmp(pid1, pid2, KCMP_FILES,		0, 0),

_

Indeed it's nit, wonder why I've not beed warned by compiler before (nothing serious since results do fit integer anyway but _must_
be fixed). I'll update shortly. Sorry for that.

Cyrill

Heh, I just realized that you've fixed it already. Thanks a lot!

Cyrill

We would like to have an ability to restore command line arguments and program environment pointers but first we need to obtain them somehow. Thus we put these values into /proc/$pid/stat. The exit_code is needed to restore zombie tasks.

---
 Documentation/filesystems/proc.txt |    5 +++++
 fs/proc/array.c                    |   20 +++++++++++++++++---
 2 files changed, 22 insertions(+), 3 deletions(-)

Index: linux-2.6.git/Documentation/filesystems/proc.txt
===================================================================
@@ -311,6 +311,11 @@ Table 1-4: Contents of the stat files (a
   start_data    address above which program data+bss is placed
   end_data      address below which program data+bss is placed
   start_brk     address above which program heap can be expanded with brk()
+  arg_start     address above which program command line is placed
+  arg_end       address below which program command line is placed
+  env_start     address above which program environment is placed
+  env_end       address below which program environment is placed
+  exit_code     the thread's exit_code in the form reported by the waitpid system call
 ..............................................................................
 
 The /proc/PID/maps file containing the currently mapped memory regions and

Index: linux-2.6.git/fs/proc/array.c
===================================================================
@@ -508,9 +508,23 @@ static int do_task_stat(struct seq_file
 	seq_put_decimal_ull(m, ' ', delayacct_blkio_ticks(task));
 	seq_put_decimal_ull(m, ' ', cputime_to_clock_t(gtime));
 	seq_put_decimal_ll(m, ' ', cputime_to_clock_t(cgtime));
-	seq_put_decimal_ull(m, ' ', (mm && permitted) ? mm->start_data : 0);
-	seq_put_decimal_ull(m, ' ', (mm && permitted) ? mm->end_data : 0);
-	seq_put_decimal_ull(m, ' ', (mm && permitted) ? mm->start_brk : 0);
+
+	if (mm && permitted) {
+		seq_put_decimal_ull(m, ' ', mm->start_data);
+		seq_put_decimal_ull(m, ' ', mm->end_data);
+		seq_put_decimal_ull(m, ' ', mm->start_brk);
+		seq_put_decimal_ull(m, ' ', mm->arg_start);
+		seq_put_decimal_ull(m, ' ', mm->arg_end);
+		seq_put_decimal_ull(m, ' ', mm->env_start);
+		seq_put_decimal_ull(m, ' ', mm->env_end);
+	} else
+		seq_printf(m, " 0 0 0 0 0 0 0");
+
+	if (permitted)
+		seq_put_decimal_ll(m, ' ', task->exit_code);
+	else
+		seq_put_decimal_ll(m, ' ', 0);
+
 	seq_putc(m, '\n');
 	if (mm)
 		mmput(mm);

During checkpoint we dump whole process memory to a file and the dump includes process stack memory. But among stack data itself, the stack carries additional parameters such as command line arguments, environment data and auxiliary vector.

So when we do restore procedure and once we've restored stack data itself we need to setup mm_struct::arg_start/end, env_start/end, so restored process would be able to find command line arguments and environment data it had at checkpoint time. The same applies to auxiliary vector.

For this reason additional PR_SET_MM_(ARG_START | ARG_END | ENV_START | ENV_END | AUXV) codes are introduced.

v2: Add BUILD_BUG_ON guard

---
 include/linux/prctl.h |    5 ++
 kernel/sys.c          |  113 +++++++++++++++++++++++++++++++++++---------------
 2 files changed, 85 insertions(+), 33 deletions(-)

Index: linux-2.6.git/include/linux/prctl.h
===================================================================
@@ -113,5 +113,10 @@
 # define PR_SET_MM_START_STACK		5
 # define PR_SET_MM_START_BRK		6
 # define PR_SET_MM_BRK			7
+# define PR_SET_MM_ARG_START		8
+# define PR_SET_MM_ARG_END		9
+# define PR_SET_MM_ENV_START		10
+# define PR_SET_MM_ENV_END		11
+# define PR_SET_MM_AUXV			12
 
 #endif /* _LINUX_PRCTL_H */

Thus far our (my) approach has been to trickle the c/r support code into mainline as it is developed. Under the assumption that the end result will be acceptable and useful kernel code.

I'm afraid that I'm losing confidence in that approach. We have this patchset, we have Stanislav's "IPC: checkpoint/restore in userspace enhancements" (which apparently needs to get more complex to support LSM context c/r). I simply *don't know* what additional patchsets are expected. And from what you told me it sounds like networking support is at a very early stage and I fear for what the end result of that will look like.

So I don't feel that I can continue feeding these things into mainline until someone can convince me that we won't have a nasty mess (and/or an unsufficiently useful feature) at the end of the project.

The traditional approach is to develop the feature out-of-tree until it is "finished". That's a lot more hackwork for you guys and it leads to a poorer feature - this approach inevitably has a lower level of review and inhibits code rework.

An alternative is for me to buffer the patches in my tree until it is all sufficiently finished. That also is more work for your team, but it will produce better code, because of additional review and code rework resulting from that review.

I don't know how many patches that would end up being (this is part of the problem!) nor how long they would be carried for.

So. Please talk to me. How long is this all going to take, and what will the final result look like?

I understand. But there was a confidence that nobody wanted the c/r stuff to be the "one big kernel subsystem", but it should rather be "a bunch of small API-s for what is required". The amount of code for the initial C/R attempt was ~100 patches. The amount of code to support our user-space C/R implementation *only* is ~10 and the feature-set of both is already comparable.

As far as the networking is concerned -- we will not require any additional patches to implement the basic netns configuration migration (ip can show and re-configure all we need about routing, interfaces, devices, etc. and the iptables-save/iptables-restore will handle 99.9% of the netfilter part). For what we currently need is the ability to explore sockets queues, but currently this doesn't turn out to be a lot of code -- I have 60-lines patch for unix sockets and Tejun showed the way how to do the same with TCP using 130 lines of code. UDP won't require anything, its queues can be silently dropped. The recent 50 patches with *_diag stuff doesn't count, because it works not for C/R only, the ss tool can benefit from 100% of the added functionality (this, btw, shows that not every piece of code we add for C/R is for C/R *only*).

Isn't the CONFIG_CHECKPOINT_RESTORE option turned off by default enough?

That's why we started sending patches early.

Neither do I :(

The Big Intermediate Result we're trying to achieve is -- take a basic OpenVZ or LXC container based on e.g. rhel6 template and make sure we can checkpoint and restore it without breaking one.

The More-or-less Finished state of the project would be when it's able to do all the stuff that the OpenVZ's implementation can. The list of major features which are yet absent in the CRIU and for which we will require the kernel support includes

* shared kernel objects (this thread)
* tcp connection
* pty stuff
* sysvipc
* iterative working set migration

The latter one is an ability to find out which pages processes use and catch when they change data on them. I planned to discuss this on LSF, but we can start earlier if you want.

Other currently missing stuff is quite minor or doesn't require any new things form the kernel like signalfd-s or netfilter.

The Ultimate Goal is hard to describe because we have the variety of ideas about what the CRIU can do including such things as checkpointing desktop apps' with their xserver state or live-migrating parts of a multi-process app from one box to another.

Thanks,
Pavel

Andrew, I hope Pavel has addressed all your concerns? What I personally trying to achieve mostly -- the patches should be as minimum as possible, still usable. I believe the patches which are already in tree are useful for other projects as well (for example -- /proc/pid/task/tid/"children" to find all children and build process topology fast). prctl extension look a bit redundant for kernel in general, but they are easily turnable off via Kconfig option. /proc/pid/map_files/ might be redundant too but it could be eliminated via Kconfig as well. So I think the both series actually do not bring much noise into kernel itself.

Cyrill