seccomp: continue syscall from notifier
Subsystems (6)
v2 cover seccomp: continue syscall from notifier
Hey everyone,
/* v2 */
This is the patchset coming out of the KSummit session Kees and I gave in Lisbon last week (cf. [3] which also contains slides with more details on related things such as deep argument inspection). The simple idea is to extend the seccomp notifier to allow for the continuation of a syscall. The rationale for this can be found in the commit message to [1]. For the curious there is more detail in [2]. This patchset would unblock supervising an extended set of syscalls such as mount() where a privileged process is supervising the syscalls of a lesser privileged process and emulates the syscall for the latter in userspace.
For more comments on security see [1] and the comments in include/uapi/linux/seccomp.h added by this patchset.
Kees, if you prefer a pr the series can be pulled from:
[email protected]:pub/scm/linux/kernel/git/brauner/linux tags/seccomp-notify-syscall-continue-v5.5
For anyone who wants to play with this it's sitting in:
https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=seccomp_syscall_continue
/* v1 */
- Kees Cook <[email protected]>:
- dropped patch because it is already present in linux-next [PATCH 2/4] seccomp: add two missing ptrace ifdefines
/* v0 */
Thanks!
Christian
*** BLURB HERE ***
Christian Brauner (3):
seccomp: add SECCOMP_USER_NOTIF_FLAG_CONTINUE
seccomp: avoid overflow in implicit constant conversion
seccomp: test SECCOMP_USER_NOTIF_FLAG_CONTINUE
include/uapi/linux/seccomp.h | 28 +++++ kernel/seccomp.c | 28 ++++- tools/testing/selftests/seccomp/seccomp_bpf.c | 110 +++++++++++++++++- 3 files changed, 159 insertions(+), 7 deletions(-)
Signature · 1 line
2.23.0
PATCH v2 1/3 seccomp: add SECCOMP_USER_NOTIF_FLAG_CONTINUE
This allows the seccomp notifier to continue a syscall. A positive discussion about this feature was triggered by a post to the ksummit-discuss mailing list (cf. [3]) and took place during KSummit (cf. [1]) and again at the containers/checkpoint-restore micro-conference at Linux Plumbers.
Recently we landed seccomp support for SECCOMP_RET_USER_NOTIF (cf. [4]) which enables a process (watchee) to retrieve an fd for its seccomp filter. This fd can then be handed to another (usually more privileged) process (watcher). The watcher will then be able to receive seccomp messages about the syscalls having been performed by the watchee.
This feature is heavily used in some userspace workloads. For example, it is currently used to intercept mknod() syscalls in user namespaces aka in containers.
The mknod() syscall can be easily filtered based on dev_t. This allows us to only intercept a very specific subset of mknod() syscalls. Furthermore, mknod() is not possible in user namespaces toto coelo and so intercepting and denying syscalls that are not in the whitelist on accident is not a big deal. The watchee won't notice a difference.
In contrast to mknod(), a lot of other syscall we intercept (e.g. setxattr()) cannot be easily filtered like mknod() because they have pointer arguments. Additionally, some of them might actually succeed in user namespaces (e.g. setxattr() for all "user.*" xattrs). Since we currently cannot tell seccomp to continue from a user notifier we are stuck with performing all of the syscalls in lieu of the container. This is a huge security liability since it is extremely difficult to correctly assume all of the necessary privileges of the calling task such that the syscall can be successfully emulated without escaping other additional security restrictions (think missing CAP_MKNOD for mknod(), or MS_NODEV on a filesystem etc.). This can be solved by telling seccomp to resume the syscall.
One thing that came up in the discussion was the problem that another thread could change the memory after userspace has decided to let the syscall continue which is a well known TOCTOU with seccomp which is present in other ways already.
The discussion showed that this feature is already very useful for any syscall without pointer arguments. For any accidentally intercepted non-pointer syscall it is safe to continue.
For syscalls with pointer arguments there is a race but for any cautious userspace and the main usec cases the race doesn't matter. The notifier is intended to be used in a scenario where a more privileged watcher supervises the syscalls of lesser privileged watchee to allow it to get around kernel-enforced limitations by performing the syscall for it whenever deemed save by the watcher. Hence, if a user tricks the watcher into allowing a syscall they will either get a deny based on kernel-enforced restrictions later or they will have changed the arguments in such a way that they manage to perform a syscall with arguments that they would've been allowed to do anyway.
In general, it is good to point out again, that the notifier fd was not intended to allow userspace to implement a security policy but rather to work around kernel security mechanisms in cases where the watcher knows that a given action is safe to perform.
/* References */
[1]: https://linuxplumbersconf.org/event/4/contributions/560
[2]: https://linuxplumbersconf.org/event/4/contributions/477
[3]: https://lore.kernel.org/r/[email protected]
[4]: commit 6a21cc50f0c7 ("seccomp: add a return code to trap to userspace")
---
/* v2 */
- Jann Horn <[email protected]>:
- mention that SECCOMP_USER_NOTIF_FLAG_CONTINUE can be used to override lower SECCOMP_RET_USER_NOTIF and SECCOMP_RET_TRACE filters
/* v1 */
- Kees Cook <[email protected]>, Tycho Andersen <[email protected]>:
- s/SECCOMP_RET_USER_NOTIF_ALLOW/SECCOMP_USER_NOTIF_FLAG_CONTINUE/g
- Kees Cook <[email protected]>:
- put giant warning about the dangers, and correct usage of the SECCOMP_USER_NOTIF_FLAG_CONTINUE flag
- Kees Cook <[email protected]>:
- change return type for seccomp_do_user_notification() to int to align with similar functions
/* v0 */
--- include/uapi/linux/seccomp.h | 28 ++++++++++++++++++++++++++++ kernel/seccomp.c | 28 ++++++++++++++++++++++------ 2 files changed, 50 insertions(+), 6 deletions(-)
include/uapi/linux/seccomp.h +28 -0
@@ -76,6 +76,34 @@ struct seccomp_notif {struct seccomp_data data; }; +/* +* Valid flags for struct seccomp_notif_resp +* +* Note, the SECCOMP_USER_NOTIF_FLAG_CONTINUE flag must be used with caution! +* If set by the process supervising the syscalls of another process the +* syscall will continue. This is problematic because of an inherent TOCTOU. +* An attacker can exploit the time while the supervised process is waiting on +* a response from the supervising process to rewrite syscall arguments which +* are passed as pointers of the intercepted syscall. +* It should be absolutely clear that this means that the seccomp notifier +* _cannot_ be used to implement a security policy! It should only ever be used +* in scenarios where a more privileged process supervises the syscalls of a +* lesser privileged process to get around kernel-enforced security +* restrictions when the privileged process deems this safe. In other words, +* in order to continue a syscall the supervising process should be sure that +* another security mechanism or the kernel itself will sufficiently block +* syscalls if arguments are rewritten to something unsafe. +* +* Similar precautions should be applied when stacking SECCOMP_RET_USER_NOTIF. +* For SECCOMP_RET_USER_NOTIF filters acting on the same syscall the uppermost +* filter takes precedence. This means that the uppermost +* SECCOMP_RET_USER_NOTIF filter can override any SECCOMP_IOCTL_NOTIF_SEND from +* lower filters essentially allowing all syscalls to pass by using +* SECCOMP_USER_NOTIF_FLAG_CONTINUE. Note that SECCOMP_RET_USER_NOTIF can +* equally be overriden by SECCOMP_USER_NOTIF_FLAG_CONTINUE. +*/ +#define SECCOMP_USER_NOTIF_FLAG_CONTINUE BIT(0) +struct seccomp_notif_resp {__u64 id;__s64 val;
kernel/seccomp.c +22 -6
@@ -75,6 +75,7 @@ struct seccomp_knotif {/* The return values, only valid when in SECCOMP_NOTIFY_REPLIED */ int error;long val; +u32 flags;/* Signals when this has entered SECCOMP_NOTIFY_REPLIED */ struct completion ready;
@@ -732,11 +733,12 @@ static u64 seccomp_next_notify_id(struct seccomp_filter *filter)return filter->notif->next_id++; } -static void seccomp_do_user_notification (int this_syscall, -struct seccomp_filter *match, -const struct seccomp_data *sd) +static int seccomp_do_user_notification (int this_syscall, +struct seccomp_filter *match, +const struct seccomp_data *sd) {int err; +u32 flags =0 ;long ret =0 ;struct seccomp_knotif n = {};
@@ -764,6 +766,7 @@ static void seccomp_do_user_notification(int this_syscall,if (err ==0 ) { ret = n.val; err = n.error; + flags = n.flags; }/*
@@ -780,8 +783,14 @@ static void seccomp_do_user_notification(int this_syscall,list_del (&n.list); out:mutex_unlock (&match->notify_lock); + +/* Userspace requests to continue the syscall. */ +if (flags &SECCOMP_USER_NOTIF_FLAG_CONTINUE ) +return 0 ; +syscall_set_return_value (current,task_pt_regs (current), err, ret); +return -1 ; } static int __seccomp_filter(int this_syscall, const struct seccomp_data *sd,
@@ -867,8 +876,10 @@ static int __seccomp_filter(int this_syscall, const struct seccomp_data *sd,return 0 ;case SECCOMP_RET_USER_NOTIF : -seccomp_do_user_notification (this_syscall, match, sd); - goto skip; +if (seccomp_do_user_notification (this_syscall, match, sd)) + goto skip; + +return 0 ;case SECCOMP_RET_LOG :seccomp_log (this_syscall,0 , action, true);
@@ -1087,7 +1098,11 @@ static long seccomp_notify_send(struct seccomp_filter *filter,if (copy_from_user (&resp, buf,sizeof (resp)))return -EFAULT ; -if (resp.flags) +if (resp.flags & ~SECCOMP_USER_NOTIF_FLAG_CONTINUE ) +return -EINVAL ; + +if ((resp.flags &SECCOMP_USER_NOTIF_FLAG_CONTINUE ) && + (resp.error || resp.val))return -EINVAL ; ret =mutex_lock_interruptible (&filter->notify_lock);
@@ -1116,6 +1131,7 @@ static long seccomp_notify_send(struct seccomp_filter *filter, knotif->state =SECCOMP_NOTIFY_REPLIED ; knotif->error = resp.error; knotif->val = resp.val; + knotif->flags = resp.flags;complete (&knotif->ready); out:mutex_unlock (&filter->notify_lock);
Signature · 1 line
2.23.0
Kees Cook
AppliedRe: [PATCH v2 1/3] seccomp: add SECCOMP_USER_NOTIF_FLAG_CONTINUE
Quoted 6 lines · Christian Brauner
+ * Similar precautions should be applied when stacking
SECCOMP_RET_USER_NOTIF.
+ * ForSECCOMP_RET_USER_NOTIFfilters acting on the same syscall the uppermost
+ * filter takes precedence. This means that the uppermost
+ *SECCOMP_RET_USER_NOTIFfilter can override anySECCOMP_IOCTL_NOTIF_SENDfrom
+ * lower filters essentially allowing all syscalls to pass by using
+ *SECCOMP_USER_NOTIF_FLAG_CONTINUE. Note thatSECCOMP_RET_USER_NOTIFcan
^^^^^^^^^^^^^^
This is meant to read RET_TRACE, yes?
Quoted 1 line
+ * equally be overriden by
SECCOMP_USER_NOTIF_FLAG_CONTINUE.
I rewrote this paragraph with that corrected and swapping some "upper/lower" to "most recently added" etc:
+ * Similar precautions should be applied when stacking SECCOMP_RET_USER_NOTIF
+ * or SECCOMP_RET_TRACE. For SECCOMP_RET_USER_NOTIF filters acting on the
+ * same syscall, the most recently added filter takes precedence. This means
+ * that the new SECCOMP_RET_USER_NOTIF filter can override any
+ * SECCOMP_IOCTL_NOTIF_SEND from earlier filters, essentially allowing all
+ * such filtered syscalls to be executed by sending the response
+ * SECCOMP_USER_NOTIF_FLAG_CONTINUE. Note that SECCOMP_RET_TRACE can equally
+ * be overriden by SECCOMP_USER_NOTIF_FLAG_CONTINUE.
Ultimately, I think this caveat is fine. RET_USER_NOTIF and RET_TRACE are both used from the "process manager" use-case. The benefits of "continue" semantics here outweighs the RET_USER_NOTIF and RET_TRACE "bypass". If we end up in a situation where we need to deal with some kind of nesting where this is a problem in practice, we can revisit this.
Applied to my for-next/seccomp. Thanks!
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/3] seccomp: add SECCOMP_USER_NOTIF_FLAG_CONTINUE
Quoted 9 lines · Kees Cook
Quoted 6 lines · Christian Brauner
+ * Similar precautions should be applied when stacking
SECCOMP_RET_USER_NOTIF.
+ * ForSECCOMP_RET_USER_NOTIFfilters acting on the same syscall the uppermost
+ * filter takes precedence. This means that the uppermost
+ *SECCOMP_RET_USER_NOTIFfilter can override anySECCOMP_IOCTL_NOTIF_SENDfrom
+ * lower filters essentially allowing all syscalls to pass by using
+ *SECCOMP_USER_NOTIF_FLAG_CONTINUE. Note thatSECCOMP_RET_USER_NOTIFcan^^^^^^^^^^^^^^This is meant to read
RET_TRACE, yes?
Yes. :)
Quoted 16 lines
Quoted 2 lines
+ * equally be overriden by
SECCOMP_USER_NOTIF_FLAG_CONTINUE.I rewrote this paragraph with that corrected and swapping some "upper/lower" to "most recently added" etc:
+ * Similar precautions should be applied when stacking
SECCOMP_RET_USER_NOTIF
+ * orSECCOMP_RET_TRACE. ForSECCOMP_RET_USER_NOTIFfilters acting on the
+ * same syscall, the most recently added filter takes precedence. This means
+ * that the newSECCOMP_RET_USER_NOTIFfilter can override any
+ *SECCOMP_IOCTL_NOTIF_SENDfrom earlier filters, essentially allowing all
+ * such filtered syscalls to be executed by sending the response
+ *SECCOMP_USER_NOTIF_FLAG_CONTINUE. Note thatSECCOMP_RET_TRACEcan equally
+ * be overriden bySECCOMP_USER_NOTIF_FLAG_CONTINUE.
Quoted 7 lines
Ultimately, I think this caveat is fine.
RET_USER_NOTIFandRET_TRACEare both used from the "process manager" use-case. The benefits of "continue" semantics here outweighs theRET_USER_NOTIFandRET_TRACE"bypass". If we end up in a situation where we need to deal with some kind of nesting where this is a problem in practice, we can revisit this.Applied to my for-next/seccomp. Thanks!
Thanks!
Christian
PATCH v2 2/3 seccomp: avoid overflow in implicit constant conversion
In reply to Christian BraunerUSER_NOTIF_MAGIC is assigned to int variables in this test so set it to INT_MAX to avoid warnings:
seccomp_bpf.c: In function ‘user_notification_continue’:
seccomp_bpf.c:3088:26: warning: overflow in implicit constant conversion [-Woverflow]
#define USER_NOTIF_MAGIC 116983961184613L
^
seccomp_bpf.c:3572:15: note: in expansion of macro ‘USER_NOTIF_MAGIC’
resp.error = USER_NOTIF_MAGIC;
^~~~~~~~~~~~~~~~---
/* v2 */
unchanged
/* v1 */
unchanged
/* v0 */
--- tools/testing/selftests/seccomp/seccomp_bpf.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/seccomp/seccomp_bpf.c +2 -1
@@ -35,6 +35,7 @@#include <stdbool.h> #include <string.h> #include <time.h> +#include <limits.h> #include <linux/elf.h> #include <sys/uio.h> #include <sys/utsname.h>
@@ -3072,7 +3073,7 @@ static int user_trap_syscall(int nr, unsigned int flags)return seccomp (SECCOMP_SET_MODE_FILTER , flags, &prog); } -#define USER_NOTIF_MAGIC 116983961184613L +#define USER_NOTIF_MAGIC INT_MAXTEST (user_notification_basic) {pid_t pid;
Signature · 1 line
2.23.0
Re: [PATCH v2 2/3] seccomp: avoid overflow in implicit constant conversion
Quoted 11 lines · Christian Brauner
USER_NOTIF_MAGICis assigned to int variables in this test so set it toINT_MAXto avoid warnings:
seccomp_bpf.c: In function ‘user_notification_continue’:
seccomp_bpf.c:3088:26: warning: overflow in implicit constant conversion [-Woverflow]#define USER_NOTIF_MAGIC 116983961184613L ^ seccomp_bpf.c:3572:15: note: in expansion of macro ‘USER_NOTIF_MAGIC’ resp.error = USER_NOTIF_MAGIC; ^~~~~~~~~~~~~~~~
Quoted 13 lines
Fixes 6a21cc50f0c7 ("seccomp: add a return code to trap to userspace")Signed-off-by Christian Brauner <[email protected]>Reviewed-by Tyler Hicks <[email protected]>Cc Kees Cook <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Will Drewry <[email protected]>Cc Shuah Khan <[email protected]>Cc Alexei Starovoitov <[email protected]>Cc Daniel Borkmann <[email protected]>Cc Martin KaFai Lau <[email protected]>Cc Song Liu <[email protected]>Cc Yonghong Song <[email protected]>Cc Tycho Andersen <[email protected]>
You can also add,
for this one.
Tycho
PATCH v2 3/3 seccomp: test SECCOMP_USER_NOTIF_FLAG_CONTINUE
In reply to Christian BraunerTest whether a syscall can be performed after having been intercepted by the seccomp notifier. The test uses dup() and kcmp() since it allows us to nicely test whether the dup() syscall actually succeeded by comparing whether the fds refer to the same underlying struct file.
---
/* v2 */
- Shuah Khan <[email protected]>:
- skip test on missing precondition
/* v1 */
- Christian Brauner <[email protected]>:
- adapt to new flag name SECCOMP_USER_NOTIF_FLAG_CONTINUE
/* v0 */
--- tools/testing/selftests/seccomp/seccomp_bpf.c | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+)
tools/testing/selftests/seccomp/seccomp_bpf.c +107 -0
@@ -44,6 +44,7 @@#include <sys/times.h> #include <sys/socket.h> #include <sys/ioctl.h> +#include <linux/kcmp.h> #include <unistd.h> #include <sys/syscall.h>
@@ -167,6 +168,10 @@ struct seccomp_metadata {#define SECCOMP_RET_USER_NOTIF 0x7fc00000U +#ifndef SECCOMP_USER_NOTIF_FLAG_CONTINUE +#define SECCOMP_USER_NOTIF_FLAG_CONTINUE 0x00000001 +#endif +#define SECCOMP_IOC_MAGIC '!'#define SECCOMP_IO (nr) _IO(SECCOMP_IOC_MAGIC, nr)#define SECCOMP_IOR (nr, type) _IOR(SECCOMP_IOC_MAGIC, nr, type)
@@ -3481,6 +3486,108 @@ TEST(seccomp_get_notif_sizes)EXPECT_EQ (sizes.seccomp_notif_resp,sizeof (struct seccomp_notif_resp )); } +static int filecmp (pid_t pid1,pid_t pid2,int fd1,int fd2) +{ +#ifdef __NR_kcmp +return syscall (__NR_kcmp, pid1, pid2,KCMP_FILE , fd1, fd2); +#else + errno =ENOSYS ; +return -1 ; +#endif +} + +TEST (user_notification_continue) +{ +pid_t pid; +long ret; +int status, listener; +struct seccomp_notif req = {}; +struct seccomp_notif_resp resp = {}; +struct pollfd pollfd; + + ret =prctl (PR_SET_NO_NEW_PRIVS ,1 ,0 ,0 ,0 ); + ASSERT_EQ(0, ret) { +TH_LOG ("Kernel does not support PR_SET_NO_NEW_PRIVS!" ); + } + + listener =user_trap_syscall (__NR_dup,SECCOMP_FILTER_FLAG_NEW_LISTENER ); +ASSERT_GE (listener,0 ); + + pid =fork (); +ASSERT_GE (pid,0 ); + +if (pid ==0 ) { +int dup_fd, pipe_fds[2 ]; +pid_t self; + + ret =pipe (pipe_fds); +if (ret <0 ) +exit (1 ); + + dup_fd =dup (pipe_fds[0 ]); +if (dup_fd <0 ) +exit (1 ); + + self =getpid (); + + ret =filecmp (self, self, pipe_fds[0 ], dup_fd); +if (ret) +exit (2 ); + +exit (0 ); + } + + pollfd.fd = listener; + pollfd.events =POLLIN |POLLOUT ; + +EXPECT_GT (poll (&pollfd,1 ,-1 ),0 ); +EXPECT_EQ (pollfd.revents,POLLIN ); + +EXPECT_EQ (ioctl (listener,SECCOMP_IOCTL_NOTIF_RECV , &req),0 ); + + pollfd.fd = listener; + pollfd.events =POLLIN |POLLOUT ; + +EXPECT_GT (poll (&pollfd,1 ,-1 ),0 ); +EXPECT_EQ (pollfd.revents,POLLOUT ); + +EXPECT_EQ (req.data.nr, __NR_dup); + + resp.id = req.id; + resp.flags =SECCOMP_USER_NOTIF_FLAG_CONTINUE ; + +/* +* Verify that setting SECCOMP_USER_NOTIF_FLAG_CONTINUE enforces other +* args be set to 0. +*/ + resp.error =0 ; + resp.val =USER_NOTIF_MAGIC ; +EXPECT_EQ (ioctl (listener,SECCOMP_IOCTL_NOTIF_SEND , &resp),-1 ); +EXPECT_EQ (errno,EINVAL ); + + resp.error =USER_NOTIF_MAGIC ; + resp.val =0 ; +EXPECT_EQ (ioctl (listener,SECCOMP_IOCTL_NOTIF_SEND , &resp),-1 ); +EXPECT_EQ (errno,EINVAL ); + + resp.error =0 ; + resp.val =0 ; + EXPECT_EQ(ioctl(listener, SECCOMP_IOCTL_NOTIF_SEND, &resp), 0) { +if (errno ==EINVAL ) + XFAIL(goto skip, "Kernel does not support SECCOMP_USER_NOTIF_FLAG_CONTINUE"); + } + +skip: +EXPECT_EQ (waitpid (pid, &status,0 ), pid); +EXPECT_EQ (true,WIFEXITED (status)); + EXPECT_EQ(0, WEXITSTATUS(status)) { +if (WEXITSTATUS (status) ==2 ) { +XFAIL (return,"Kernel does not support kcmp() syscall" ); +return ; + } + } +} +/* * TODO: * - add microbenchmarks
Signature · 1 line
2.23.0