Skip to contentSkip to conversation

capabilities: Introduce CAP_CHECKPOINT_RESTORE

PATCH Superseded Adrian Reber 2 participants 5 messages Jun 18, 2020

Subsystems (8)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover capabilities: Introduce CAP_CHECKPOINT_RESTORE

This is v3 of the 'Introduce CAP_CHECKPOINT_RESTORE' patchset. There is only one change from v2:

* made if condition easier to read as requested by Cyrill

Besides that there were no further comments on the changes proposed in this patchset.

There was the discussion from Andrei that PTRACE_O_SUSPEND_SECCOMP is also needed for checkpointing. CRIU already has the possibility to detect if a process is using seccomp and could so tell the user that it cannot checkpoint a process if the process is using seccomp. As seccomp has not come up in the requests from users to use CRIU as non-root so far and as there was some push back from Christian to allow PTRACE_O_SUSPEND_SECCOMP if CAP_CHECKPOINT_RESTORE is set I would like to leave this open for the future.

Another discussion was around relaxing the existing map_files check from capable() to ns_capable() or even completely removing it. Even if this happens we still need CAP_CHECKPOINT_RESTORE and the removal or change to ns_capable() is not blocked by this patchset.

Besides that there was nothing speaking against CAP_CHECKPOINT_RESTORE during the v2 discussions.

Adrian Reber (2):
capabilities: Introduce CAP_CHECKPOINT_RESTORE
selftests: add clone3() CAP_CHECKPOINT_RESTORE test

Nicolas Viennot (1):
prctl: Allow ptrace capable processes to change exe_fd

 fs/proc/base.c                                |   8 +-
 include/linux/capability.h                    |   6 +
 include/uapi/linux/capability.h               |   9 +-
 kernel/pid.c                                  |   2 +-
 kernel/pid_namespace.c                        |   2 +-
 kernel/sys.c                                  |  21 +-
 security/selinux/include/classmap.h           |   5 +-
 tools/testing/selftests/clone3/Makefile       |   4 +-
 .../clone3/clone3_cap_checkpoint_restore.c    | 203 ++++++++++++++++++
 9 files changed, 245 insertions(+), 15 deletions(-)
 create mode 100644 tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c

base-commit: 5fcb9628fd1227a5f11d87171cb1b8b5c414d9d9

Signature · 1 line
2.26.2
#1 lore raw

PATCH v3 1/3 capabilities: Introduce CAP_CHECKPOINT_RESTORE

This patch introduces CAP_CHECKPOINT_RESTORE, a new capability facilitating checkpoint/restore for non-root users.

Over the last years, The CRIU (Checkpoint/Restore In Userspace) team has been asked numerous times if it is possible to checkpoint/restore a process as non-root. The answer usually was: 'almost'.

The main blocker to restore a process as non-root was to control the PID of the restored process. This feature available via the clone3 system call, or via /proc/sys/kernel/ns_last_pid is unfortunately guarded by CAP_SYS_ADMIN.

In the past two years, requests for non-root checkpoint/restore have increased due to the following use cases:
* Checkpoint/Restore in an HPC environment in combination with a resource manager distributing jobs where users are always running as non-root. There is a desire to provide a way to checkpoint and restore long running jobs.
* Container migration as non-root
* We have been in contact with JVM developers who are integrating CRIU into a Java VM to decrease the startup time. These checkpoint/restore applications are not meant to be running with CAP_SYS_ADMIN.

We have seen the following workarounds:
* Use a setuid wrapper around CRIU:
See https://github.com/FredHutch/slurm-examples/blob/master/checkpointer/lib/checkpointer/checkpointer-suid.c
* Use a setuid helper that writes to ns_last_pid.
Unfortunately, this helper delegation technique is impossible to use with
clone3, and is thus prone to races.
See https://github.com/twosigma/set_ns_last_pid
* Cycle through PIDs with fork() until the desired PID is reached:
This has been demonstrated to work with cycling rates of 100,000 PIDs/s
See https://github.com/twosigma/set_ns_last_pid
* Patch out the CAP_SYS_ADMIN check from the kernel
* Run the desired application in a new user and PID namespace to provide
a local CAP_SYS_ADMIN for controlling PIDs. This technique has limited use in
typical container environments (e.g., Kubernetes) as /proc is
typically protected with read-only layers (e.g., /proc/sys) for hardening
purposes. Read-only layers prevent additional /proc mounts (due to proc's SB_I_USERNS_VISIBLE property), making the use of new PID namespaces limited as
certain applications need access to /proc matching their PID namespace.

The introduced capability allows to:
* Control PIDs when the current user is CAP_CHECKPOINT_RESTORE capable for the corresponding PID namespace via ns_last_pid/clone3.
* Open files in /proc/pid/map_files when the current user is CAP_CHECKPOINT_RESTORE capable in the root namespace, useful for recovering files that are unreachable via the file system such as deleted files, or memfd files.

See corresponding selftest for an example with clone3().

---
v2:
- Renamed CAP_RESTORE to CAP_CHECKPOINT_RESTORE
- Added a test
- Added details about CRIU's use of map_files
- Allow changing /proc/self/exe link with CAP_CHECKPOINT_RESTORE
v3:
- made if condition easier to read

---
 fs/proc/base.c                      | 8 ++++----
 include/linux/capability.h          | 6 ++++++
 include/uapi/linux/capability.h     | 9 ++++++++-
 kernel/pid.c                        | 2 +-
 kernel/pid_namespace.c              | 2 +-
 security/selinux/include/classmap.h | 5 +++--
 6 files changed, 23 insertions(+), 9 deletions(-)
fs/proc/base.c +4 -4
index d86c0afc8a85..ad806069c778 100644
@@ -2189,16 +2189,16 @@ struct map_files_info {
 };
 
 /*
- * Only allow CAP_SYS_ADMIN to follow the links, due to concerns about how the
- * symlinks may be used to bypass permissions on ancestor directories in the
- * path to the file in question.
+ * Only allow CAP_SYS_ADMIN and CAP_CHECKPOINT_RESTORE to follow the links, due
+ * to concerns about how the symlinks may be used to bypass permissions on
+ * ancestor directories in the path to the file in question.
  */
 static const char *
 proc_map_files_get_link(struct dentry *dentry,
 			struct inode *inode,
 		        struct delayed_call *done)
 {
-	if (!capable(CAP_SYS_ADMIN))
+	if (!capable(CAP_SYS_ADMIN) && !capable(CAP_CHECKPOINT_RESTORE))
 		return ERR_PTR(-EPERM);
 
 	return proc_pid_get_link(dentry, inode, done);
include/linux/capability.h +6 -0
index b4345b38a6be..1e7fe311cabe 100644
@@ -261,6 +261,12 @@ static inline bool bpf_capable(void)
 	return capable(CAP_BPF) || capable(CAP_SYS_ADMIN);
 }
 
+static inline bool checkpoint_restore_ns_capable(struct user_namespace *ns)
+{
+	return ns_capable(ns, CAP_CHECKPOINT_RESTORE) ||
+		ns_capable(ns, CAP_SYS_ADMIN);
+}
+
 /* audit system wants to get cap info from files as well */
 extern int get_vfs_caps_from_disk(const struct dentry *dentry, struct cpu_vfs_cap_data *cpu_caps);
 
include/uapi/linux/capability.h +8 -1
index 48ff0757ae5e..395dd0df8d08 100644
@@ -408,7 +408,14 @@ struct vfs_ns_cap_data {
  */
 #define CAP_BPF			39
 
-#define CAP_LAST_CAP         CAP_BPF
+
+/* Allow checkpoint/restore related operations */
+/* Allow PID selection during clone3() */
+/* Allow writing to ns_last_pid */
+
+#define CAP_CHECKPOINT_RESTORE	40
+
+#define CAP_LAST_CAP         CAP_CHECKPOINT_RESTORE
 
 #define cap_valid(x) ((x) >= 0 && (x) <= CAP_LAST_CAP)
 
kernel/pid.c +1 -1
index 3122043fe364..ada55c7b2b19 100644
@@ -198,7 +198,7 @@ struct pid *alloc_pid(struct pid_namespace *ns, pid_t *set_tid,
 			if (tid != 1 && !tmp->child_reaper)
 				goto out_free;
 			retval = -EPERM;
-			if (!ns_capable(tmp->user_ns, CAP_SYS_ADMIN))
+			if (!checkpoint_restore_ns_capable(tmp->user_ns))
 				goto out_free;
 			set_tid_size--;
 		}
kernel/pid_namespace.c +1 -1
index 0e5ac162c3a8..ac135bd600eb 100644
@@ -269,7 +269,7 @@ static int pid_ns_ctl_handler(struct ctl_table *table, int write,
 	struct ctl_table tmp = *table;
 	int ret, next;
 
-	if (write && !ns_capable(pid_ns->user_ns, CAP_SYS_ADMIN))
+	if (write && !checkpoint_restore_ns_capable(pid_ns->user_ns))
 		return -EPERM;
 
 	/*
security/selinux/include/classmap.h +3 -2
index 98e1513b608a..40cebde62856 100644
@@ -27,9 +27,10 @@
 	    "audit_control", "setfcap"
 
 #define COMMON_CAP2_PERMS  "mac_override", "mac_admin", "syslog", \
-		"wake_alarm", "block_suspend", "audit_read", "perfmon", "bpf"
+		"wake_alarm", "block_suspend", "audit_read", "perfmon", "bpf", \
+		"checkpoint_restore"
 
-#if CAP_LAST_CAP > CAP_BPF
+#if CAP_LAST_CAP > CAP_CHECKPOINT_RESTORE
 #error New capability defined, please update COMMON_CAP2_PERMS.
 #endif
 
Signature · 1 line
2.26.2
#2 lore raw

PATCH v3 2/3 selftests: add clone3() CAP_CHECKPOINT_RESTORE test

In reply to Adrian Reber

This adds a test that changes its UID, uses capabilities to get CAP_CHECKPOINT_RESTORE and uses clone3() with set_tid to create a process with a given PID as non-root.

---
 tools/testing/selftests/clone3/Makefile       |   4 +-
 .../clone3/clone3_cap_checkpoint_restore.c    | 203 ++++++++++++++++++
 2 files changed, 206 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c
tools/testing/selftests/clone3/Makefile +3 -1
index cf976c732906..ef7564cb7abe 100644
@@ -1,6 +1,8 @@
 # SPDX-License-Identifier: GPL-2.0
 CFLAGS += -g -I../../../../usr/include/
+LDLIBS += -lcap
 
-TEST_GEN_PROGS := clone3 clone3_clear_sighand clone3_set_tid
+TEST_GEN_PROGS := clone3 clone3_clear_sighand clone3_set_tid \
+	clone3_cap_checkpoint_restore
 
 include ../lib.mk
tools/testing/selftests/clone3/clone3_cap_checkpoint_restore.c new file +203 -0

203 lines not on this page · raw message

Signature · 1 line
2.26.2
#3 lore raw

PATCH v3 3/3 prctl: Allow ptrace capable processes to change exe_fd

In reply to Adrian Reber

From: Nicolas Viennot <[email protected]>

The current process is authorized to change its /proc/self/exe link via two policies:
1) The current user can do checkpoint/restore In other words is CAP_SYS_ADMIN or CAP_CHECKPOINT_RESTORE capable.
2) The current user can use ptrace.

With access to ptrace facilities, a process can do the following: fork a child, execve() the target executable, and have the child use ptrace() to replace the memory content of the current process. This technique makes it possible to masquerade an arbitrary program as any executable, even setuid ones.

This commit also changes the permission error code from -EINVAL to -EPERM for consistency with the rest of the prctl() syscall when checking capabilities.

---
 kernel/sys.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)
kernel/sys.c +16 -5
index 00a96746e28a..ce77012a42d7 100644
@@ -2007,12 +2007,23 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long data
 
 	if (prctl_map.exe_fd != (u32)-1) {
 		/*
-		 * Make sure the caller has the rights to
-		 * change /proc/pid/exe link: only local sys admin should
-		 * be allowed to.
+		 * The current process is authorized to change its
+		 * /proc/self/exe link via two policies:
+		 * 1) The current user can do checkpoint/restore
+		 *    In other words is CAP_SYS_ADMIN or
+		 *    CAP_CHECKPOINT_RESTORE capable.
+		 * 2) The current user can use ptrace.
+		 *
+		 * With access to ptrace facilities, a process can do the
+		 * following: fork a child, execve() the target executable,
+		 * and have the child use ptrace() to replace the memory
+		 * content of the current process. This technique makes it
+		 * possible to masquerade an arbitrary program as the target
+		 * executable, even if it is setuid.
 		 */
-		if (!ns_capable(current_user_ns(), CAP_SYS_ADMIN))
-			return -EINVAL;
+		if (!(checkpoint_restore_ns_capable(current_user_ns()) ||
+		      security_ptrace_access_check(current, PTRACE_MODE_ATTACH_REALCREDS)))
+			return -EPERM;
 
 		error = prctl_set_mm_exe_file(mm, prctl_map.exe_fd);
 		if (error)
Signature · 1 line
2.26.2
#4 lore raw

Re: [PATCH v3 3/3] prctl: Allow ptrace capable processes to change exe_fd

Quoted 15 lines · Adrian Reber

The current process is authorized to change its /proc/self/exe link via two policies:
1) The current user can do checkpoint/restore In other words is CAP_SYS_ADMIN or CAP_CHECKPOINT_RESTORE capable.
2) The current user can use ptrace.

With access to ptrace facilities, a process can do the following: fork a child, execve() the target executable, and have the child use ptrace() to replace the memory content of the current process. This technique makes it possible to masquerade an arbitrary program as any executable, even setuid ones.

This commit also changes the permission error code from -EINVAL to -EPERM for consistency with the rest of the prctl() syscall when checking capabilities.

[...]

Quoted 1 line
kernel/sys.c +0 -0

[...]

Quoted 10 lines
kernel/sys.c +3 -3
@@ -2007,12 +2007,23 @@ static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long data
 
        if (prctl_map.exe_fd != (u32)-1) {
                /*
-                * Make sure the caller has the rights to
-                * change /proc/pid/exe link: only local sys admin should
-                * be allowed to.
+                * The current process is authorized to change its
+                * /proc/self/exe link via two policies:
+                * 1) The current user can do checkpoint/restore

kernel/sys.c static int prctl_set_mm_map(int opt, const void __user *addr, unsigned long data · line 2022

Quoted 10 lines
kernel/sys.c +10 -0
+                *    In other words is CAP_SYS_ADMIN or
+                *    CAP_CHECKPOINT_RESTORE capable.
+                * 2) The current user can use ptrace.
+                *
+                * With access to ptrace facilities, a process can do the
+                * following: fork a child, execve() the target executable,
+                * and have the child use ptrace() to replace the memory
+                * content of the current process. This technique makes it
+                * possible to masquerade an arbitrary program as the target
+                * executable, even if it is setuid.

(That is not necessarily true in the presence of LSMs like SELinux:
You'd have to be able to FILE__EXECUTE_NO_TRANS the target executable according to the system's security policy.)