um: skas: harden the seccomp userspace stub
cover um: skas: harden the seccomp userspace stub
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the stub.c "Known security issues" list could not be addressed by the filter alone:
- a hijacked stub could mmap() arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and
- a hijacked stub could block SIGALRM via a crafted rt_sigreturn to evade preemption and wedge the monitor indefinitely.
This series closes both:
1-2: route the stub's mmap() through a SECCOMP_RET_USER_NOTIF listener
owned by the monitor (no behavioural change yet).
3-4: validate each mmap() against the mm's page table -- allowed iff the
PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validate munmap() the same way (range-confined below
STUB_START).
6: add a watchdog thread that detects a stub which stops reporting
back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.
7: drop the now-resolved "Known security issues" note and refresh the
seccomp= help text.
After the series a hijacked stub is confined to the frames its own page tables reference and can no longer reach arbitrary guest/host memory; one that evades preemption is detected out of band and killed rather than wedging the monitor.
Verified on UML (UP and 2-CPU SMP): boots and survives fork/exec storms and heavy mmap/munmap churn with zero false denials or false kills; an artificially SIGALRM-blocked busy loop is killed in ~5s and the monitor recovers, while syscall-making processes are untouched. Each patch builds and the series is bisectable.
---
Cong Wang (7):
um: skas: create a seccomp USER_NOTIF listener and hand it to the
monitor
um: skas: gate stub mmap() through the USER_NOTIF monitor
um: skas: validate stub mmap() against the guest page table
um: skas: handle out-of-batch stub mmap notifications
um: skas: validate stub munmap() against the guest address range
um: skas: kill stubs that block SIGALRM via a watchdog thread
um: skas: refresh stub security notes after closing the known issues
arch/um/include/shared/skas/mm_id.h | 1 + arch/um/include/shared/skas/skas.h | 5 + arch/um/kernel/skas/stub.c | 22 -- arch/um/kernel/skas/stub_exe.c | 19 +- arch/um/kernel/skas/uaccess.c | 48 +++++ arch/um/os-Linux/skas/process.c | 315 ++++++++++++++++++++++++---- arch/um/os-Linux/start_up.c | 6 - 7 files changed, 344 insertions(+), 72 deletions(-)
base-commit: 1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
Signature · 1 line
2.43.0
PATCH 1/7 um: skas: create a seccomp USER_NOTIF listener and hand it to the monitor
From: Cong Wang <[email protected]>
First step toward validating stub mmap() calls (the intra-guest / host-escape disclosure issue in stub.c "Known security issues"): give the monitor a SECCOMP_RET_USER_NOTIF listener for each stub's filter, so a later change can route mmap to the monitor for per-call validation.
The stub installs its seccomp filter with NEW_LISTENER (plus TSYNC_ESRCH, which the kernel requires to combine NEW_LISTENER with TSYNC); the seccomp() return value is the listener fd. mmap still returns RET_ALLOW, so there is no behavioural change yet.
The stub cannot hand the fd over itself: once the filter is installed, every syscall it makes from outside the stub page traps with SIGSYS instead of executing, so it can neither sendmsg() nor close() the fd. Instead the monitor pulls it with pidfd_getfd(): the listener is, by construction, fd 1 in the stub (close_range() left only fd 0 open before seccomp() allocated it). Leaving the stub's copy open is harmless:
ioctl (NOTIF_RECV/SEND) is not on the syscall allowlist, so a hijacked stub cannot self-approve. The monitor stores the fd in mm_id.
--- arch/um/include/shared/skas/mm_id.h | 1 + arch/um/kernel/skas/stub_exe.c | 12 ++++++--- arch/um/os-Linux/skas/process.c | 38 +++++++++++++++++++++++++++-- 3 files changed, 45 insertions(+), 6 deletions(-)
arch/um/include/shared/skas/mm_id.h +1 -0
@@ -17,6 +17,7 @@ struct mm_id {/* Only used with SECCOMP mode */ int sock; +int seccomp_notify_fd;int syscall_fd_num;int syscall_fd_map[STUB_MAX_FDS ]; };
arch/um/kernel/skas/stub_exe.c +8 -4
@@ -196,10 +196,14 @@ noinline static void real_init(void) .len = sizeof(filter) / sizeof(filter[0]), .filter = filter, }; - -if (stub_syscall3 (__NR_seccomp,SECCOMP_SET_MODE_FILTER , -SECCOMP_FILTER_FLAG_TSYNC , - (unsigned long )&prog) !=0 ) +long listener; + + listener =stub_syscall3 (__NR_seccomp,SECCOMP_SET_MODE_FILTER , +SECCOMP_FILTER_FLAG_TSYNC | +SECCOMP_FILTER_FLAG_NEW_LISTENER | +SECCOMP_FILTER_FLAG_TSYNC_ESRCH , + (unsigned long )&prog); +if (listener <0 )stub_syscall1 (__NR_exit,21 );/* Fall through, the exit syscall will cause SIGSYS */
arch/um/os-Linux/skas/process.c +36 -2
@@ -428,6 +428,34 @@ __initcall(init_stub_exe_fd);int using_seccomp; +/* +* Obtain the SECCOMP_RET_USER_NOTIF listener fd the stub created. The stub +* cannot hand it over itself: after installing the filter, every syscall it +* makes traps with SIGSYS rather than executing (so it can neither sendmsg() +* it nor close() it). Instead the listener is, by construction, the first free +* fd in the stub -- fd 1, since close_range() left only fd 0 open before +* seccomp() allocated it -- so the monitor duplicates it directly with +* pidfd_getfd(). ptrace_may_access() holds because the monitor is the stub's +* parent. +*/ +#define STUB_LISTENER_FD 1 +static int get_stub_listener (struct mm_id *mm_id) +{ +int pidfd, lfd; + + pidfd =syscall (__NR_pidfd_open, mm_id->pid,0 ); +if (pidfd <0 ) +return -errno; + + lfd =syscall (__NR_pidfd_getfd, pidfd,STUB_LISTENER_FD ,0 ); +close (pidfd); +if (lfd <0 ) +return -errno; + + mm_id->seccomp_notify_fd = lfd; +return 0 ; +} +/** * start_userspace() - prepare a new userspace process * @mm_id: The corresponding struct mm_id
@@ -449,6 +477,8 @@ int start_userspace(struct mm_id *mm_id)unsigned long sp;int status, n, err; + mm_id->seccomp_notify_fd =-1 ; +/* setup a temporary stack page */ stack = mmap(NULL, UM_KERN_PAGE_SIZE, PROT_READ | PROT_WRITE | PROT_EXEC,
@@ -522,10 +552,14 @@ int start_userspace(struct mm_id *mm_id) }close (tramp_data.sockpair[0 ]); -if (using_seccomp) +if (using_seccomp) { mm_id->sock = tramp_data.sockpair[1 ]; -else + err =get_stub_listener (mm_id); +if (err) + goto out_kill; + }else {close (tramp_data.sockpair[1 ]); + }return 0 ;
Signature · 1 line
2.43.0
PATCH 2/7 um: skas: gate stub mmap() through the USER_NOTIF monitor
In reply to Cong WangFrom: Cong Wang <[email protected]>
Route the stub's mmap to SECCOMP_RET_USER_NOTIF instead of RET_ALLOW, and have the monitor service the resulting notifications inline. This is the mechanism that a following change uses to validate each mmap's arguments; for now every stub mmap is allowed (responded with SECCOMP_USER_NOTIF_FLAG_CONTINUE), so behaviour is unchanged.
CONTINUE is safe for mmap: its arguments are all scalars captured in seccomp_data, so there is no TOCTOU re-read of user memory.
The stub runs queued mmap batches in two places: syscall_stub_flush() and the userspace() resume path, both of which wake the stub via wait_stub_done_seccomp(). Servicing therefore lives there: after waking, the monitor issues one NOTIF_RECV/CONTINUE per STUB_SYSCALL_MMAP in the batch the stub is about to run. Signals are masked while the stub executes the batch inside its SIGSYS handler, so notifications arrive in queued order with nothing interleaved, and a simple counted loop is sufficient. The pre-existing wake logic is factored into wake_seccomp_stub() so both the wake and the wait-only paths share it.
Verified on UML: guest boots and survives a fork/exec storm plus heavy demand paging with every stub mmap round-tripping through the monitor.
--- arch/um/kernel/skas/stub_exe.c | 5 +- arch/um/os-Linux/skas/process.c | 121 ++++++++++++++++++++++---------- 2 files changed, 89 insertions(+), 37 deletions(-)
arch/um/kernel/skas/stub_exe.c +4 -1
@@ -173,7 +173,7 @@ noinline static void real_init(void)BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K ,__NR_close,5 ,0 ),BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K ,STUB_MMAP_NR , -4 ,0 ), +5 ,0 ),BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_munmap,3 ,0 ), #ifdef __i386__
@@ -191,6 +191,9 @@ noinline static void real_init(void)/* [18] Permitted call for the stub */ BPF_STMT (BPF_RET |BPF_K ,SECCOMP_RET_ALLOW ), + +/* [19] mmap: route to the monitor for validation */ + BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_USER_NOTIF), };struct sock_fprog prog = { .len = sizeof(filter) / sizeof(filter[0]),
arch/um/os-Linux/skas/process.c +85 -36
147 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH 3/7 um: skas: validate stub mmap() against the guest page table
In reply to Cong WangFrom: Cong Wang <[email protected]>
Replace the allow-all USER_NOTIF response with the actual security check. When a stub mmap traps to the monitor, validate its arguments against the mm's page table: the call is allowed iff this mm's PTE for the target address maps exactly the physical frame named by the mmap offset, and the requested protection does not exceed what the PTE grants. Otherwise the mmap is rejected with -EPERM.
This is what closes the disclosure issue documented in stub.c. A hijacked stub (jumping to the in-stub mmap with crafted registers) can no longer choose an arbitrary physmem offset: it is confined to the frames its own mm's PTEs already reference. On this base (single physmem fd, no kernel/user split) the same check also blocks host escape: a mapping of a UML-kernel frame has no authorizing user PTE, so it is denied.
The check is a pure function of state the monitor already owns (it *is* the UML kernel and holds the guest pgd, freshly synced before the stub mmap is issued), so it needs no per-batch bookkeeping or fd-identity tracking. stub_mmap_allowed() lives in skas/uaccess.c next to virt_to_pte(); the os-Linux notify handler calls it and responds CONTINUE or -EPERM.
Verified on UML: guest boots and survives a fork/exec storm plus heavy demand paging with zero false denials.
--- arch/um/include/shared/skas/skas.h | 3 +++ arch/um/kernel/skas/uaccess.c | 36 ++++++++++++++++++++++++++++++ arch/um/os-Linux/skas/process.c | 21 ++++++++--------- 3 files changed, 50 insertions(+), 10 deletions(-)
arch/um/include/shared/skas/skas.h +3 -0
@@ -18,4 +18,7 @@ extern void current_mm_sync(void);void initial_jmpbuf_lock (void );void initial_jmpbuf_unlock (void ); +int stub_mmap_allowed (struct mm_id *id,unsigned long addr, +unsigned long prot,unsigned long offset); +#endif
arch/um/kernel/skas/uaccess.c +36 -0
@@ -13,6 +13,18 @@#include <kern_util.h> #include <asm/futex.h> #include <os.h> +#include <skas.h> + +/* +* Same mapping as MMAP_OFFSET() in <sysdep/stub.h>, but usable from kernel +* code (that header pulls in the host <sys/mman.h>). 64-bit stubs use mmap() +* with a byte offset; 32-bit stubs use mmap2() with a page offset. +*/ +#ifdef CONFIG_64BIT +#define stub_mmap_offset (phys) (phys) +#else +#define stub_mmap_offset (phys) ((phys) >> PAGE_SHIFT) +#endif pte_t *virt_to_pte (struct mm_struct *mm,unsigned long addr) {
@@ -43,6 +55,30 @@ pte_t *virt_to_pte(struct mm_struct *mm, unsigned long addr)return pte_offset_kernel (pmd, addr); } +int stub_mmap_allowed (struct mm_id *id,unsigned long addr, +unsigned long prot,unsigned long offset) +{ + struct mm_context *ctx = container_of(id, struct mm_context, id); + struct mm_struct *mm = container_of(ctx, struct mm_struct, context); +pte_t *pte; + + pte =virt_to_pte (mm, addr); +if (pte ==NULL || !pte_present (*pte)) +return 0 ; + +/* Must map exactly the frame this PTE references. */ +if (stub_mmap_offset (pte_val (*pte) &PAGE_MASK ) != offset) +return 0 ; + +/* Must not grant more access than the PTE allows. */ +if ((prot &UM_PROT_WRITE ) && !pte_write (*pte)) +return 0 ; +if ((prot &UM_PROT_EXEC ) && !pte_exec (*pte)) +return 0 ; + +return 1 ; +} +static pte_t *maybe_map (unsigned long virt,int is_write) {pte_t *pte =virt_to_pte (current->mm, virt);
arch/um/os-Linux/skas/process.c +11 -10
@@ -187,10 +187,11 @@ static void wake_seccomp_stub(struct mm_id *mm_idp) FUTEX_WAKE, 1, NULL, NULL, 0)); } -static int seccomp_notify_serve (int notify_fd) +static int seccomp_notify_serve (struct mm_id *mm_idp) {struct seccomp_notif req = {};struct seccomp_notif_resp resp = {}; +int notify_fd = mm_idp->seccomp_notify_fd;int ret;CATCH_EINTR (ret =ioctl (notify_fd,SECCOMP_IOCTL_NOTIF_RECV , &req));
@@ -198,7 +199,14 @@ static int seccomp_notify_serve(int notify_fd)return -errno; resp.id = req.id; - resp.flags =SECCOMP_USER_NOTIF_FLAG_CONTINUE ; + +if (req.data.nr ==STUB_MMAP_NR && +stub_mmap_allowed (mm_idp, req.data.args[0 ], req.data.args[2 ], + req.data.args[5 ])) { + resp.flags =SECCOMP_USER_NOTIF_FLAG_CONTINUE ; + }else { + resp.error = -EPERM ; + }CATCH_EINTR (ret =ioctl (notify_fd,SECCOMP_IOCTL_NOTIF_SEND , &resp));if (ret <0 )
@@ -220,7 +228,7 @@ static void seccomp_serve_mmaps(struct mm_id *mm_idp) n_mmaps++;for (i =0 ; i < n_mmaps; i++) -seccomp_notify_serve (mm_idp->seccomp_notify_fd); +seccomp_notify_serve (mm_idp); }void wait_stub_done_seccomp (struct mm_id *mm_idp,int running,int wait_sigsys)
@@ -288,13 +296,6 @@ void wait_stub_done_seccomp(struct mm_id *mm_idp, int running, int wait_sigsys)fatal_sigsegv (); } -/* -* Service one SECCOMP_RET_USER_NOTIF notification from a stub mmap: read the -* suspended call, then respond CONTINUE so the stub's real mmap runs. CONTINUE -* is safe here because mmap takes only scalar arguments (no TOCTOU on user -* memory). Validation of (addr, len, prot, fd, offset) is added later; for now -* every stub mmap is allowed. -*/ extern unsigned long current_stub_stack (void );static void get_skas_faultinfo (int pid,struct faultinfo *fi)
Signature · 1 line
2.43.0
PATCH 4/7 um: skas: handle out-of-batch stub mmap notifications
In reply to Cong WangFrom: Cong Wang <[email protected]>
The mmap validation so far only runs for mmaps the monitor itself queued. A hijacked stub can instead jump to the in-stub mmap on its own, outside any batch. Such an mmap traps to the USER_NOTIF listener and suspends in-kernel (so it discloses nothing) but it never signals the monitor via the futex, so the monitor, waiting only on the futex, would block forever (a guest-triggered DoS of the whole UML).
Drain the listener whenever the wait wakes: after each FUTEX_WAIT in wait_stub_done_seccomp() (which already wakes periodically on SIGALRM and tolerates EINTR), poll the listener non-blocking and service anything pending. An out-of-batch mmap therefore stays suspended only until the next wakeup and is then validated like any other: denied with -EPERM unless it maps exactly what its PTE allows. No disclosure occurs while it is pending, and the monitor no longer hangs.
--- arch/um/os-Linux/skas/process.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
arch/um/os-Linux/skas/process.c +19 -0
@@ -32,6 +32,7 @@#include <linux/seccomp.h> #include <linux/threads.h> #include <sys/ioctl.h> +#include <poll.h> #include <timetravel.h> #include <asm-generic/rwonce.h> #include "../internal.h"
@@ -215,6 +216,21 @@ static int seccomp_notify_serve(struct mm_id *mm_idp)return 0 ; } +static void seccomp_notify_drain (struct mm_id *mm_idp) +{ +struct pollfd pfd = { + .fd = mm_idp->seccomp_notify_fd, + .events =POLLIN , + }; + +if (mm_idp->seccomp_notify_fd <0 ) +return ; + +while (poll (&pfd,1 ,0 ) >0 && (pfd.revents &POLLIN )) +if (seccomp_notify_serve (mm_idp) <0 ) +break ; +} +static void seccomp_serve_mmaps (struct mm_id *mm_idp) {struct stub_data *data = (void *)mm_idp->stack;
@@ -265,6 +281,9 @@ void wait_stub_done_seccomp(struct mm_id *mm_idp, int running, int wait_sigsys) __func__, errno); goto out_kill; } + +/* Handle any stub mmap that trapped out of band. */ +seccomp_notify_drain (mm_idp); }while (data->futex ==FUTEX_IN_CHILD );if (__READ_ONCE (mm_idp->pid) <0 )
Signature · 1 line
2.43.0
PATCH 5/7 um: skas: validate stub munmap() against the guest address range
In reply to Cong WangFrom: Cong Wang <[email protected]>
Route stub munmap() through the USER_NOTIF monitor too, and validate it before letting it run. munmap() was previously SECCOMP_RET_ALLOW, so a hijacked stub (jumping to the in-stub munmap with crafted registers) could unmap arbitrary ranges, including the stub's own code/data pages, which would sever the monitor's control over it, or guest mappings outside what it is allowed to manage. After mmap, munmap was the remaining memory primitive a hijacked stub could invoke with arbitrary arguments.
Unlike mmap(), there is no PTE left to check: by the time the stub unmaps a guest page the kernel has already cleared the corresponding entry. So stub_munmap_allowed() is range-based instead: the request must be non-empty, must not wrap, and must lie entirely below STUB_START. That confines the stub to the guest address space and keeps its own reserved region off-limits. Both arguments are scalars captured in seccomp_data, so CONTINUE carries no TOCTOU risk, same as mmap().
stub_munmap_allowed() lives in skas/uaccess.c next to stub_mmap_allowed(); the os-Linux notify handler dispatches on the syscall number and responds CONTINUE or -EPERM, and the batch server counts STUB_SYSCALL_MUNMAP as well as STUB_SYSCALL_MMAP.
Verified on UML: guest boots and survives heavy mmap/munmap churn with zero false denials; the legitimate boot-time clear of the whole user address space [0, STUB_START) is allowed (end == STUB_START), while a range overlapping the stub region is denied.
--- arch/um/include/shared/skas/skas.h | 2 ++ arch/um/kernel/skas/stub_exe.c | 4 ++-- arch/um/kernel/skas/uaccess.c | 12 ++++++++++++ arch/um/os-Linux/skas/process.c | 31 ++++++++++++++++++------------ 4 files changed, 35 insertions(+), 14 deletions(-)
arch/um/include/shared/skas/skas.h +2 -0
@@ -20,5 +20,7 @@ void initial_jmpbuf_unlock(void);int stub_mmap_allowed (struct mm_id *id,unsigned long addr,unsigned long prot,unsigned long offset); +int stub_munmap_allowed (struct mm_id *id,unsigned long addr, +unsigned long len);#endif
arch/um/kernel/skas/stub_exe.c +2 -2
@@ -175,7 +175,7 @@ noinline static void real_init(void)BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K ,STUB_MMAP_NR ,5 ,0 ),BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_munmap, -3 ,0 ), +4 ,0 ), #ifdef __i386__BPF_JUMP (BPF_JMP |BPF_JEQ |BPF_K , __NR_set_thread_area, 2, 0),
@@ -192,7 +192,7 @@ noinline static void real_init(void)/* [18] Permitted call for the stub */ BPF_STMT (BPF_RET |BPF_K ,SECCOMP_RET_ALLOW ), -/* [19] mmap: route to the monitor for validation */ +/* [19] mmap and munmap: route to the monitor for validation */ BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_USER_NOTIF), }; struct sock_fprog prog = {
arch/um/kernel/skas/uaccess.c +12 -0
@@ -14,6 +14,7 @@#include <asm/futex.h> #include <os.h> #include <skas.h> +#include <as-layout.h> /* * Same mapping as MMAP_OFFSET() in <sysdep/stub.h>, but usable from kernel
@@ -79,6 +80,17 @@ int stub_mmap_allowed(struct mm_id *id, unsigned long addr,return 1 ; } +int stub_munmap_allowed (struct mm_id *id,unsigned long addr,unsigned long len) +{ +if (len ==0 || addr + len < addr) +return 0 ; + +if (addr + len >STUB_START ) +return 0 ; + +return 1 ; +} +static pte_t *maybe_map (unsigned long virt,int is_write) {pte_t *pte =virt_to_pte (current->mm, virt);
arch/um/os-Linux/skas/process.c +19 -12
@@ -193,7 +193,7 @@ static int seccomp_notify_serve(struct mm_id *mm_idp)struct seccomp_notif req = {};struct seccomp_notif_resp resp = {};int notify_fd = mm_idp->seccomp_notify_fd; -int ret; +int allowed, ret;CATCH_EINTR (ret =ioctl (notify_fd,SECCOMP_IOCTL_NOTIF_RECV , &req));if (ret <0 )
@@ -201,13 +201,19 @@ static int seccomp_notify_serve(struct mm_id *mm_idp) resp.id = req.id; -if (req.data.nr ==STUB_MMAP_NR && -stub_mmap_allowed (mm_idp, req.data.args[0 ], req.data.args[2 ], - req.data.args[5 ])) { +if (req.data.nr ==STUB_MMAP_NR ) + allowed =stub_mmap_allowed (mm_idp, req.data.args[0 ], + req.data.args[2 ], req.data.args[5 ]); +else if (req.data.nr == __NR_munmap) + allowed =stub_munmap_allowed (mm_idp, req.data.args[0 ], + req.data.args[1 ]); +else + allowed =0 ; + +if (allowed) resp.flags =SECCOMP_USER_NOTIF_FLAG_CONTINUE ; - }else { +else resp.error = -EPERM ; - }CATCH_EINTR (ret =ioctl (notify_fd,SECCOMP_IOCTL_NOTIF_SEND , &resp));if (ret <0 )
@@ -231,19 +237,20 @@ static void seccomp_notify_drain(struct mm_id *mm_idp)break ; } -static void seccomp_serve_mmaps (struct mm_id *mm_idp) +static void seccomp_serve_stub_syscalls (struct mm_id *mm_idp) {struct stub_data *data = (void *)mm_idp->stack; -int i, n_mmaps =0 ; +int i, n_notif =0 ;if (mm_idp->seccomp_notify_fd <0 )return ;for (i =0 ; i < data->syscall_data_len; i++) -if (data->syscall_data[i].syscall ==STUB_SYSCALL_MMAP ) - n_mmaps++; +if (data->syscall_data[i].syscall ==STUB_SYSCALL_MMAP || + data->syscall_data[i].syscall ==STUB_SYSCALL_MUNMAP ) + n_notif++; -for (i =0 ; i < n_mmaps; i++) +for (i =0 ; i < n_notif; i++)seccomp_notify_serve (mm_idp); }
@@ -255,7 +262,7 @@ void wait_stub_done_seccomp(struct mm_id *mm_idp, int running, int wait_sigsys) do { if (!running) { wake_seccomp_stub(mm_idp); - seccomp_serve_mmaps(mm_idp); + seccomp_serve_stub_syscalls(mm_idp); } do {
Signature · 1 line
2.43.0
PATCH 6/7 um: skas: kill stubs that block SIGALRM via a watchdog thread
In reply to Cong WangFrom: Cong Wang <[email protected]>
A hijacked stub can block SIGALRM with a crafted rt_sigreturn (the mask is restored from the stack it controls), so it is never preempted and never reports back. SIGALRM goes to the stub, not the monitor, so the monitor then blocks indefinitely in wait_stub_done_seccomp(). This is the one item ("blocking e.g. SIGALRM") of the stub.c "Known security issues" list that a seccomp filter cannot address, since rt_sigreturn is required and its effect lives on the stack rather than in register arguments.
Detect it out of band. A helper thread blocks on its own timerfd and watches a per-vCPU (pid, seq) pair the monitor updates around each wait:
pid is the stub being waited on, seq advances every time the stub reports. A stub that stops reporting leaves pid pinned and seq frozen; after SECCOMP_WD_STALL_TICKS ticks of no progress the watchdog SIGKILLs it, and the resulting SIGCHLD unblocks the monitor through the existing "pid < 0" teardown.
Each monitor writes only its own slot and the watchdog only reads, so the word-sized state needs just __READ_ONCE()/__WRITE_ONCE(), no lock; the watchdog scans every slot, covering all CPUs under SMP. A false kill cannot happen without the same pid and an unchanged seq across many ticks. The thread runs with all signals blocked (os_run_helper_thread()), uses write(2) rather than printk() from its non-kernel context, and is started once via a compare-and-swap guard. It is preferred over a bounded FUTEX_WAIT timeout: it costs one counter bump on the per-syscall hot path and catches a stall anywhere on stub input, not just the one futex.
Verified on UML (UP and 2-CPU SMP): heavy mmap/munmap churn and CPU-bound loops on every vCPU run with zero false kills; a stub with SIGALRM blocked is killed in ~5s and the monitor recovers, while syscall-making processes are untouched.
--- arch/um/os-Linux/skas/process.c | 129 ++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+)
arch/um/os-Linux/skas/process.c +129 -0
@@ -21,6 +21,7 @@#include <as-layout.h> #include <init.h> #include <kern_util.h> +#include <smp.h> #include <mem.h> #include <os.h> #include <ptrace_user.h>
@@ -33,6 +34,8 @@#include <linux/threads.h> #include <sys/ioctl.h> #include <poll.h> +#include <signal.h> +#include <sys/timerfd.h> #include <timetravel.h> #include <asm-generic/rwonce.h> #include "../internal.h"
@@ -254,11 +257,131 @@ static void seccomp_serve_stub_syscalls(struct mm_id *mm_idp)seccomp_notify_serve (mm_idp); } +#define SECCOMP_WD_TICK_SECS 1 +#define SECCOMP_WD_STALL_TICKS 5/* ~5s of no progress before killing */ + +static int seccomp_wd_pid[CONFIG_NR_CPUS ] = { [0 ...CONFIG_NR_CPUS -1 ] =-1 }; +static unsigned long seccomp_wd_seq[CONFIG_NR_CPUS ]; + +static inline void seccomp_wd_enter (int pid) +{ +int cpu =uml_curr_cpu (); + +__WRITE_ONCE (seccomp_wd_seq[cpu], seccomp_wd_seq[cpu] +1 ); +__WRITE_ONCE (seccomp_wd_pid[cpu], pid); +} + +static inline void seccomp_wd_progress (void ) +{ +int cpu =uml_curr_cpu (); + +__WRITE_ONCE (seccomp_wd_seq[cpu], seccomp_wd_seq[cpu] +1 ); +} + +static inline void seccomp_wd_exit (void ) +{ +__WRITE_ONCE (seccomp_wd_pid[uml_curr_cpu ()],-1 ); +} + +/* Per-CPU snapshot the watchdog compares against the next tick. */ +struct seccomp_wd_cpu { +int prev_pid; +unsigned long prev_seq; +int stall; +}; + +static void seccomp_wd_check_cpu (int cpu,struct seccomp_wd_cpu *st) +{ +static const char kill_msg[] = +"seccomp watchdog: killing unresponsive stub (SIGALRM blocked?)\n" ; +int pid =__READ_ONCE (seccomp_wd_pid[cpu]); +unsigned long seq =__READ_ONCE (seccomp_wd_seq[cpu]); + +if (pid >=0 && pid == st->prev_pid && seq == st->prev_seq) { +if (++st->stall >=SECCOMP_WD_STALL_TICKS ) { +/* printk() is unsafe from this thread. */ + (void )!write (2 , kill_msg,sizeof (kill_msg) -1 ); +kill (pid,SIGKILL ); + st->stall =0 ; + } + }else { + st->stall =0 ; + } + + st->prev_pid = pid; + st->prev_seq = seq; +} + +static void *seccomp_watchdog (void *arg) +{ +int tfd = (int )(long )arg; +struct seccomp_wd_cpu st[CONFIG_NR_CPUS ]; +int cpu; + +for (cpu =0 ; cpu <CONFIG_NR_CPUS ; cpu++) + st[cpu] = (struct seccomp_wd_cpu ){ .prev_pid =-1 }; + +for (;;) { +unsigned long long expirations; + +/* +* One check per wakeup; ignore the expiration count so a +* descheduled watchdog accrues stalls more slowly, never faster. +*/ +if (read (tfd, &expirations,sizeof (expirations)) != +sizeof (expirations)) +continue ; + +for (cpu =0 ; cpu <CONFIG_NR_CPUS ; cpu++) +seccomp_wd_check_cpu (cpu, &st[cpu]); + } + +return NULL ; +} + +static void start_seccomp_watchdog (void ) +{ +static int started; +static struct os_helper_thread *wd_td; +struct itimerspec its = { + .it_value = { .tv_sec =SECCOMP_WD_TICK_SECS }, + .it_interval = { .tv_sec =SECCOMP_WD_TICK_SECS }, + }; +int tfd; + +/* Several vCPU monitors may race here; only the first starts the thread. */ +if (!__sync_bool_compare_and_swap (&started,0 ,1 )) +return ; + + tfd =timerfd_create (CLOCK_MONOTONIC ,0 ); +if (tfd <0 ) { +printk (UM_KERN_ERR "%s : timerfd_create failed, errno = %d\n" , + __func__, errno); +return ; + } + +if (timerfd_settime (tfd,0 , &its,NULL ) <0 ) { +printk (UM_KERN_ERR "%s : timerfd_settime failed, errno = %d\n" , + __func__, errno); +close (tfd); +return ; + } + +if (os_run_helper_thread (&wd_td, seccomp_watchdog, (void *)(long )tfd) <0 ) { +printk (UM_KERN_ERR "%s : failed to start watchdog thread\n" , + __func__); +close (tfd); + wd_td =NULL ; + } +} +void wait_stub_done_seccomp (struct mm_id *mm_idp,int running,int wait_sigsys) {struct stub_data *data = (void *)mm_idp->stack;int ret; +seccomp_wd_enter (mm_idp->pid); + do { if (!running) { wake_seccomp_stub(mm_idp);
@@ -296,6 +419,9 @@ void wait_stub_done_seccomp(struct mm_id *mm_idp, int running, int wait_sigsys)if (__READ_ONCE (mm_idp->pid) <0 ) goto out_kill; +/* The stub reported back: record progress for the watchdog. */ +seccomp_wd_progress (); + running =0 ;/* We may receive a SIGALRM before SIGSYS, iterate again. */
@@ -312,9 +438,11 @@ void wait_stub_done_seccomp(struct mm_id *mm_idp, int running, int wait_sigsys) goto out_kill; } +seccomp_wd_exit ();return ; out_kill: +seccomp_wd_exit ();printk (UM_KERN_ERR "%s : failed to wait for stub, pid = %d, errno = %d\n" , __func__, mm_idp->pid, errno);/* This is not true inside start_userspace */
@@ -633,6 +761,7 @@ int start_userspace(struct mm_id *mm_id) err =get_stub_listener (mm_id);if (err) goto out_kill; +start_seccomp_watchdog (); }else {close (tramp_data.sockpair[1 ]); }
Signature · 1 line
2.43.0
PATCH 7/7 um: skas: refresh stub security notes after closing the known issues
In reply to Cong WangFrom: Cong Wang <[email protected]>
Drop the stale comment and update the seccomp= setup help text, which still warned about these exact issues and labelled the mode "insecure".
--- arch/um/kernel/skas/stub.c | 22 ---------------------- arch/um/os-Linux/start_up.c | 6 ------ 2 files changed, 28 deletions(-)
arch/um/kernel/skas/stub.c +0 -22
@@ -9,28 +9,6 @@#include <sys/socket.h> #include <errno.h> -/* -* Known security issues -* -* Userspace can jump to this address to execute *any* syscall that is -* permitted by the stub. As we will return afterwards, it can do -* whatever it likes, including: -* - Tricking the kernel into handing out the memory FD -* - Using this memory FD to read/write all physical memory -* - Running in parallel to the kernel processing a syscall -* (possibly creating data races?) -* - Blocking e.g. SIGALRM to avoid time based scheduling -* -* To avoid this, the permitted location for each syscall needs to be -* checked for in the SECCOMP filter (which is reasonably simple). Also, -* more care will need to go into considerations how the code might be -* tricked by using a prepared stack (or even modifying the stack from -* another thread in case SMP support is added). -* -* As for the SIGALRM, the best counter measure will be to check in the -* kernel that the process is reporting back the SIGALRM in a timely -* fashion. -*/ static __always_inline int syscall_handler(int fd_map[STUB_MAX_FDS]) {struct stub_data *d =get_stub_data ();
arch/um/os-Linux/start_up.c +0 -6
@@ -452,12 +452,6 @@ __uml_setup("seccomp=", uml_seccomp_config," This method is overall faster than the ptrace based userspace, primarily\n" " because it reduces the number of context switches for (minor) page faults.\n" "\n" -" However, the SECCOMP filter is not (yet) restrictive enough to prevent\n" -" userspace from reading and writing all physical memory. Userspace\n" -" processes could also trick the stub into disabling SIGALRM which\n" -" prevents it from being interrupted for scheduling purposes.\n" -"\n" -" This is insecure and should only be used with a trusted userspace\n\n" );void __init os_early_checks (void )
Signature · 1 line
2.43.0
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined belowSTUB_START).
That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Quoted 3 lines
6: add a watchdog thread that detects a stub which stops reporting back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.
That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
Quoted 2 lines
7: drop the now-resolved "Known security issues" note and refresh the seccomp= help text.
Benjamin
Quoted 33 lines
After the series a hijacked stub is confined to the frames its own page tables reference and can no longer reach arbitrary guest/host memory; one that evades preemption is detected out of band and killed rather than wedging the monitor.
Verified on UML (UP and 2-CPU SMP): boots and survives fork/exec storms and heavy mmap/munmap churn with zero false denials or false kills; an artificially SIGALRM-blocked busy loop is killed in ~5s and the monitor recovers, while syscall-making processes are untouched. Each patch builds and the series is bisectable.
---
Cong Wang (7):
um: skas: create a seccompUSER_NOTIFlistener and hand it to the monitor
um: skas: gate stubmmap()through theUSER_NOTIFmonitor
um: skas: validate stubmmap()against the guest page table
um: skas: handle out-of-batch stub mmap notifications
um: skas: validate stubmunmap()against the guest address range
um: skas: kill stubs that block SIGALRM via a watchdog thread
um: skas: refresh stub security notes after closing the known issuesarch/um/include/shared/skas/mm_id.h | 1 + arch/um/include/shared/skas/skas.h | 5 + arch/um/kernel/skas/stub.c | 22 -- arch/um/kernel/skas/stub_exe.c | 19 +- arch/um/kernel/skas/uaccess.c | 48 +++++ arch/um/os-Linux/skas/process.c | 315 ++++++++++++++++++++++++---- arch/um/os-Linux/start_up.c | 6 -
7 files changed, 344 insertions(+), 72 deletions(-)base-commit:
1a3746ccbb0a97bed3c06ccde6b880013b1dddc1
Quoted 35 lines · Benjamin Berg
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener
owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the
PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined below
STUB_START).That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than
ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.
Quoted 4 lines
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Sorry, I thought I included them in patch 4/7, but I missed them.
Here they are:
mmap() -- ns/call
-----------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 13056 10368 +2688 (+26%) 17408 15104 16K 13824 11008 +2816 (+26%) 15616 12544 64K 19840 15616 +4224 (+27%) 21760 17408 256K 33152 28928 +4224 (+15%) 36352 31744 1M 95616 84608 +11008 (+13%) 117504 108032
munmap() -- ns/call
-------------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 11008 8448 +2560 (+30%) 13568 11264 16K 10752 8448 +2304 (+27%) 12288 9472 64K 12032 9728 +2304 (+24%) 13824 10752 256K 15360 12800 +2560 (+20%) 17152 14592 1M 27648 24832 +2816 (+11%) 30976 26880
Since this is a clear security-vs-performance balance, I picked security over performance. Please let me know if you prefer otherwise.
Meanwhile, after a second thought, there is a zero-overhead solution. pidfd_mmap(), which can be built on top of another patch of mine:
https://lore.kernel.org/all/[email protected]/
Something like:
int map(struct mm_id *mm_idp, unsigned long virt, unsigned long len, int prot,
int phys_fd, unsigned long long offset)
{
return pidfd_mmap(mm_idp->stub_pidfd, (void *)virt, len,
prot_to_mmap(prot), MAP_FIXED | MAP_SHARED,
phys_fd, offset);
}
int unmap(struct mm_id *mm_idp, unsigned long addr, unsigned long len)
{
return pidfd_munmap(mm_idp->stub_pidfd, (void *)addr, len);
}Obviously, this one requires a new syscall for the host kernel first.
Quoted 10 lines
Quoted 4 lines
6: add a watchdog thread that detects a stub which stops reporting
back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
The flag-and-recheck scheme matches the ptrace path (wait_stub_done), where the monitor is the tracer: it sees the stub's SIGALRM as a waitpid() signal-stop and PTRACE_CONTs it, so "monitor receives, then deals with the child" is accurate there. This patch is on the seccomp path (wait_stub_done_seccomp), which is architecturally different.
Or am I missing anything?
Thanks for your review.
Regards,
Cong
Quoted 63 lines · Cong Wang
Quoted 35 lines · Benjamin Berg
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined belowSTUB_START).That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than
ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.Quoted 4 lines
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Sorry, I thought I included them in patch 4/7, but I missed them.
Here they are:
mmap()-- ns/call
-----------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 13056 10368 +2688 (+26%) 17408 15104 16K 13824 11008 +2816 (+26%) 15616 12544 64K 19840 15616 +4224 (+27%) 21760 17408 256K 33152 28928 +4224 (+15%) 36352 31744 1M 95616 84608 +11008 (+13%) 117504 108032
munmap()-- ns/call
-------------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 11008 8448 +2560 (+30%) 13568 11264 16K 10752 8448 +2304 (+27%) 12288 9472 64K 12032 9728 +2304 (+24%) 13824 10752 256K 15360 12800 +2560 (+20%) 17152 14592 1M 27648 24832 +2816 (+11%) 30976 26880
Quoted 2 lines
Since this is a clear security-vs-performance balance, I picked security over performance. Please let me know if you prefer otherwise.
So, I still believe that (almost) zero-overhead is possible in the current framework. Maybe one could add a non-zero overhead option, but then I would like to be able to disable it as there are plenty of scenarios where security is of no concern.
Also, I am unsure what you measured there. When you write "mmap" here, is that an "mmap" call made by the userspace process (which might not actually change the page mappings), or is that an "mmap" call that was queued for the stub to process?
A large amount of the overhead that UML tends to have happens when handling both minor and major page faults. Simple tests for this is for example a relatively tight fork/exec loop or just applications starting up. In our test runs, we saw major speed improvements (>10%) just by fixing bugs in the TLB handling code that resulted in fewer page faults happening at application runtime.
Quoted 3 lines
Meanwhile, after a second thought, there is a zero-overhead solution.
pidfd_mmap(), which can be built on top of another patch of mine:
https://lore.kernel.org/all/[email protected]/
Sure, should some sort of method land that allows directly modifying the mappings of the remote process, that would be rather convenient.
Quoted 27 lines
Something like:
int map(struct
mm_id*mm_idp, unsigned long virt, unsigned long len, int prot, intphys_fd, unsigned long long offset)
{
returnpidfd_mmap(mm_idp->stub_pidfd, (void *)virt, len, prot_to_mmap(prot), MAP_FIXED | MAP_SHARED, phys_fd, offset);
}
intunmap(struct mm_id *mm_idp, unsigned long addr, unsigned long len){
returnpidfd_munmap(mm_idp->stub_pidfd, (void *)addr, len); }Obviously, this one requires a new syscall for the host kernel first.
Quoted 10 lines
Quoted 4 lines
6: add a watchdog thread that detects a stub which stops reporting back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.
That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
Quoted 7 lines
The flag-and-recheck scheme matches the ptrace path (
wait_stub_done), where the monitor is the tracer: it sees the stub's SIGALRM as awaitpid()signal-stop andPTRACE_CONTsit, so "monitor receives, then deals with the child" is accurate there. This patch is on the seccomp path (wait_stub_done_seccomp), which is architecturally different.Or am I missing anything?
The fact that it is not actually different from an architectural standpoint. In both cases SIGALRM is reported back to the kernel. It obviously has to be reported back, otherwise scheduling would not work.
Benjamin
Quoted 6 lines
Thanks for your review.
Regards,
Cong
Quoted 65 lines · Benjamin Berg
Quoted 63 lines · Cong Wang
Quoted 35 lines · Benjamin Berg
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener
owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the
PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined below
STUB_START).That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than
ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.Quoted 4 lines
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Sorry, I thought I included them in patch 4/7, but I missed them.
Here they are:
mmap()-- ns/call
-----------------size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 13056 10368 +2688 (+26%) 17408 15104 16K 13824 11008 +2816 (+26%) 15616 12544 64K 19840 15616 +4224 (+27%) 21760 17408 256K 33152 28928 +4224 (+15%) 36352 31744 1M 95616 84608 +11008 (+13%) 117504 108032
munmap()-- ns/call
-------------------size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 11008 8448 +2560 (+30%) 13568 11264 16K 10752 8448 +2304 (+27%) 12288 9472 64K 12032 9728 +2304 (+24%) 13824 10752 256K 15360 12800 +2560 (+20%) 17152 14592 1M 27648 24832 +2816 (+11%) 30976 26880
Quoted 7 lines
Quoted 2 lines
Since this is a clear security-vs-performance balance, I picked security over performance. Please let me know if you prefer otherwise.
So, I still believe that (almost) zero-overhead is possible in the current framework. Maybe one could add a non-zero overhead option, but then I would like to be able to disable it as there are plenty of scenarios where security is of no concern.
This is fair, I can add an option to keep it disabled by default.
Quoted 6 lines
Also, I am unsure what you measured there. When you write "mmap" here, is that an "mmap" call made by the userspace process (which might not actually change the page mappings), or is that an "mmap" call that was queued for the stub to process?
Quoted 6 lines
A large amount of the overhead that UML tends to have happens when handling both minor and major page faults. Simple tests for this is for example a relatively tight fork/exec loop or just applications starting up. In our test runs, we saw major speed improvements (>10%) just by fixing bugs in the TLB handling code that resulted in fewer page faults happening at application runtime.
It is just a simple user-space mmap() loop just for benchmarking this patchset.
Quoted 8 lines
Quoted 4 lines
Meanwhile, after a second thought, there is a zero-overhead solution.
pidfd_mmap(), which can be built on top of another patch of mine:
https://lore.kernel.org/all/[email protected]/Sure, should some sort of method land that allows directly modifying the mappings of the remote process, that would be rather convenient.
Great. I will work on it. One question: should I drop this one or we can use this one as the short term solution before the pidfd_mmap() solution gets in? Even if we had pidfd_mmap(), UML might still want to keep compatibility with older host kernels?
Quoted 33 lines
Quoted 33 lines
Something like:
int map(struct
mm_id*mm_idp, unsigned long virt, unsigned long len, int prot,
intphys_fd, unsigned long long offset){ return pidfd_mmap(mm_idp->stub_pidfd, (void *)virt, len, prot_to_mmap(prot), MAP_FIXED | MAP_SHARED, phys_fd, offset); } int unmap(struct mm_id *mm_idp, unsigned long addr, unsigned long len) { return pidfd_munmap(mm_idp->stub_pidfd, (void *)addr, len); }Obviously, this one requires a new syscall for the host kernel first.
Quoted 10 lines
Quoted 4 lines
6: add a watchdog thread that detects a stub which stops reporting
back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
The flag-and-recheck scheme matches the ptrace path (
wait_stub_done), where the monitor is the tracer: it sees the stub's SIGALRM as awaitpid()signal-stop andPTRACE_CONTsit, so "monitor receives, then deals with the child" is accurate there. This patch is on the seccomp path (wait_stub_done_seccomp), which is architecturally different.
Quoted 5 lines
Quoted 1 line
Or am I missing anything?
The fact that it is not actually different from an architectural standpoint. In both cases SIGALRM is reported back to the kernel. It obviously has to be reported back, otherwise scheduling would not work.
You're right.
In the seccomp path the monitor also receives the SIGALRM and forwards it to the stub: the per-vCPU POSIX timer targets the monitor thread and um_timer() -> os_alarm_process() -> kill(stub, SIGALRM) forwards the tick. ptrace vs. seccomp differ only in how the monitor learns the stub stopped (waitpid vs. futex), not in the point you raised.
So the watchdog thread is redundant. The inner FUTEX_WAIT loop in wait_stub_done_seccomp already wakes on every forwarded tick via EINTR; responsive stub acks by flipping data->futex out of FUTEX_IN_CHILD, a SIGALRM-blocking one doesn't, exactly your flag scheme. I'll drop the helper thread and timerfd and detect the stall inline: count consecutive ticks where the stub didn't report and goto out_kill past a small threshold.
Thanks!
Hi,
Quoted 70 lines · Cong Wang
Quoted 68 lines · Benjamin Berg
Quoted 65 lines · Cong Wang
Quoted 35 lines · Benjamin Berg
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined belowSTUB_START).That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than
ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.Quoted 4 lines
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Sorry, I thought I included them in patch 4/7, but I missed them.
Here they are:
mmap()-- ns/call
-----------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 13056 10368 +2688 (+26%) 17408 15104 16K 13824 11008 +2816 (+26%) 15616 12544 64K 19840 15616 +4224 (+27%) 21760 17408 256K 33152 28928 +4224 (+15%) 36352 31744 1M 95616 84608 +11008 (+13%) 117504 108032
munmap()-- ns/call
-------------------
size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 11008 8448 +2560 (+30%) 13568 11264 16K 10752 8448 +2304 (+27%) 12288 9472 64K 12032 9728 +2304 (+24%) 13824 10752 256K 15360 12800 +2560 (+20%) 17152 14592 1M 27648 24832 +2816 (+11%) 30976 26880Since this is a clear security-vs-performance balance, I picked security over performance. Please let me know if you prefer otherwise.
Quoted 6 lines
Quoted 4 lines
So, I still believe that (almost) zero-overhead is possible in the current framework. Maybe one could add a non-zero overhead option, but then I would like to be able to disable it as there are plenty of scenarios where security is of no concern.
This is fair, I can add an option to keep it disabled by default.
Sure. But I still think that a discussion is needed on whether another option is viable. I would really prefer not adding your proposed solution if another low/zero overhead option is viable.
And I still absolutely believe that it is viable to modify the current code and make it secure. I just never finished that work because it requires a lot of care and it is not that important to us.
Quoted 12 lines
Quoted 11 lines
Also, I am unsure what you measured there. When you write "mmap" here, is that an "mmap" call made by the userspace process (which might not actually change the page mappings), or is that an "mmap" call that was queued for the stub to process?
A large amount of the overhead that UML tends to have happens when handling both minor and major page faults. Simple tests for this is for example a relatively tight fork/exec loop or just applications starting up. In our test runs, we saw major speed improvements (>10%) just by fixing bugs in the TLB handling code that resulted in fewer page faults happening at application runtime.
Quoted 2 lines
It is just a simple user-space
mmap()loop just for benchmarking this patchset.
If you just say "mmap() loop" I think that you are purely doing an mmap/unmap syscall of anonymous memory in a loop. However, the kernel may not map any of those pages until they are actually accessed. I would not have been surprised if you did not see any performance difference at all in such a test.
Are you actually touching (writing) each of the mapped pages?
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Meanwhile, after a second thought, there is a zero-overhead solution.
pidfd_mmap(), which can be built on top of another patch of mine:
https://lore.kernel.org/all/[email protected]/
Quoted 8 lines
Quoted 3 lines
Sure, should some sort of method land that allows directly modifying the mappings of the remote process, that would be rather convenient.
Great. I will work on it. One question: should I drop this one or we can use this one as the short term solution before the
pidfd_mmap()solution gets in? Even if we hadpidfd_mmap(), UML might still want to keep compatibility with older host kernels?
Of course, UML should not regress on older kernels. Also, any new API like that will probably still take a while. That said, I am not sure there is a pressing need to fix the security problem as long as we have ptrace mode available.
Quoted 48 lines
Quoted 34 lines
Quoted 34 lines
Something like:
int map(struct
mm_id*mm_idp, unsigned long virt, unsigned long len, int prot, intphys_fd, unsigned long long offset)
{
returnpidfd_mmap(mm_idp->stub_pidfd, (void *)virt, len, prot_to_mmap(prot), MAP_FIXED | MAP_SHARED, phys_fd, offset);
}
intunmap(struct mm_id *mm_idp, unsigned long addr, unsigned long len){
returnpidfd_munmap(mm_idp->stub_pidfd, (void *)addr, len); }Obviously, this one requires a new syscall for the host kernel first.
Quoted 10 lines
Quoted 4 lines
6: add a watchdog thread that detects a stub which stops reporting back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.
That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
The flag-and-recheck scheme matches the ptrace path (
wait_stub_done), where the monitor is the tracer: it sees the stub's SIGALRM as awaitpid()signal-stop andPTRACE_CONTsit, so "monitor receives, then deals with the child" is accurate there. This patch is on the seccomp path (wait_stub_done_seccomp), which is architecturally different.Quoted 5 lines
Quoted 1 line
Or am I missing anything?
The fact that it is not actually different from an architectural standpoint. In both cases SIGALRM is reported back to the kernel. It obviously has to be reported back, otherwise scheduling would not work.
You're right.
In the seccomp path the monitor also receives the SIGALRM and forwards it to the stub: the per-vCPU POSIX timer targets the monitor thread and
um_timer()->os_alarm_process()->kill(stub, SIGALRM)forwards the tick. ptrace vs. seccomp differ only in how the monitor learns the stub stopped (waitpid vs. futex), not in the point you raised.
Quoted 6 lines
So the watchdog thread is redundant. The inner
FUTEX_WAITloop inwait_stub_done_seccompalready wakes on every forwarded tick via EINTR; responsive stub acks by flippingdata->futexout ofFUTEX_IN_CHILD, a SIGALRM-blocking one doesn't, exactly your flag scheme. I'll drop the helper thread and timerfd and detect the stall inline: count consecutive ticks where the stub didn't report and gotoout_killpast a small threshold.
Not sure I follow and the description makes me wonder if it was written by an LLM. To me it seems overly complicated and incorrect in subtle ways.
Benjamin
Quoted 81 lines · Benjamin Berg
Hi,
Quoted 70 lines · Cong Wang
Quoted 68 lines · Benjamin Berg
Quoted 65 lines · Cong Wang
Quoted 35 lines · Benjamin Berg
Quoted 16 lines · Cong Wang
From: Cong Wang <[email protected]>
In the seccomp ("SECCOMP") userspace mode, each guest userspace process runs in a stub under a seccomp filter and traps to the monitor (the UML kernel) on every syscall. Two items on the
stub.c"Known security issues" list could not be addressed by the filter alone:- a hijacked stub could
mmap()arbitrary physmem offsets, which is an intra-guest disclosure and, on this base (single physmem fd, no kernel/user split), a host escape; and- a hijacked stub could block SIGALRM via a crafted
rt_sigreturnto evade preemption and wedge the monitor indefinitely.This series closes both:
Quoted 8 lines
1-2: route the stub's
mmap()through aSECCOMP_RET_USER_NOTIFlistener
owned by the monitor (no behavioural change yet).
3-4: validate eachmmap()against the mm's page table -- allowed iff the
PTE already maps the requested frame with no more access than it grants -- including out-of-batch mmaps a hijacked stub issues on its own.
5: route and validatemunmap()the same way (range-confined below
STUB_START).That approach seems odd to me. Adding an explicit out-of-band check means you require two extra context switches per mmap syscall. I would expect that this makes the SECCOMP approach a lot slower than
ptrace(). My take is still that it is possible to carefully craft a SECCOMP filter as well as stub/kernel code that makes exploitation impossible for non-SMP.Quoted 4 lines
The true SMP case is more complicated, but we do not have that anyway, so I would not worry about it for now.
Did you run any performance tests?
Sorry, I thought I included them in patch 4/7, but I missed them.
Here they are:
mmap()-- ns/call
-----------------size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 13056 10368 +2688 (+26%) 17408 15104 16K 13824 11008 +2816 (+26%) 15616 12544 64K 19840 15616 +4224 (+27%) 21760 17408 256K 33152 28928 +4224 (+15%) 36352 31744 1M 95616 84608 +11008 (+13%) 117504 108032
munmap()-- ns/call
-------------------size WITH med WITHOUT med delta WITH p99 WITHOUT p99 4K 11008 8448 +2560 (+30%) 13568 11264 16K 10752 8448 +2304 (+27%) 12288 9472 64K 12032 9728 +2304 (+24%) 13824 10752 256K 15360 12800 +2560 (+20%) 17152 14592 1M 27648 24832 +2816 (+11%) 30976 26880Since this is a clear security-vs-performance balance, I picked security over performance. Please let me know if you prefer otherwise.
Quoted 6 lines
Quoted 4 lines
So, I still believe that (almost) zero-overhead is possible in the current framework. Maybe one could add a non-zero overhead option, but then I would like to be able to disable it as there are plenty of scenarios where security is of no concern.
This is fair, I can add an option to keep it disabled by default.
Quoted 7 lines
Sure. But I still think that a discussion is needed on whether another option is viable. I would really prefer not adding your proposed solution if another low/zero overhead option is viable.
And I still absolutely believe that it is viable to modify the current code and make it secure. I just never finished that work because it requires a lot of care and it is not that important to us.
OK. Just to provide more context: I am evaluating using UML for multi-tenant cloud, security is a must there, not just optional.
I guess I might be the first one doing so, otherwise these security issues would have been resolved. :)
Quoted 16 lines
Quoted 13 lines
Quoted 6 lines
Also, I am unsure what you measured there. When you write "mmap" here, is that an "mmap" call made by the userspace process (which might not actually change the page mappings), or is that an "mmap" call that was queued for the stub to process?
Quoted 6 lines
A large amount of the overhead that UML tends to have happens when handling both minor and major page faults. Simple tests for this is for example a relatively tight fork/exec loop or just applications starting up. In our test runs, we saw major speed improvements (>10%) just by fixing bugs in the TLB handling code that resulted in fewer page faults happening at application runtime.
Quoted 2 lines
It is just a simple user-space
mmap()loop just for benchmarking this patchset.
Quoted 7 lines
If you just say "
mmap()loop" I think that you are purely doing an mmap/unmap syscall of anonymous memory in a loop. However, the kernel may not map any of those pages until they are actually accessed. I would not have been surprised if you did not see any performance difference at all in such a test.Are you actually touching (writing) each of the mapped pages?
Yes, with MAP_POPULATE. Sorry for forgetting to mention it.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Meanwhile, after a second thought, there is a zero-overhead solution.
pidfd_mmap(), which can be built on top of another patch of mine:
https://lore.kernel.org/all/[email protected]/Quoted 8 lines
Quoted 3 lines
Sure, should some sort of method land that allows directly modifying the mappings of the remote process, that would be rather convenient.
Great. I will work on it. One question: should I drop this one or we can use this one as the short term solution before the
pidfd_mmap()solution gets in? Even if we hadpidfd_mmap(), UML might still want to keep compatibility with older host kernels?
Quoted 4 lines
Of course, UML should not regress on older kernels. Also, any new API like that will probably still take a while. That said, I am not sure there is a pressing need to fix the security problem as long as we have ptrace mode available.
Maybe I am greedy, I want both security and performance. I ruled out ptrace model due to low performance and try to improve seccomp mode for better security.
Anyway, I agree to drop this patchset and work on pidfd_mmap(), I will send them out shortly.
Quoted 56 lines
Quoted 49 lines
Quoted 35 lines
Quoted 35 lines
Something like:
int map(struct
mm_id*mm_idp, unsigned long virt, unsigned long len, int prot,
intphys_fd, unsigned long long offset){ return pidfd_mmap(mm_idp->stub_pidfd, (void *)virt, len, prot_to_mmap(prot), MAP_FIXED | MAP_SHARED, phys_fd, offset); } int unmap(struct mm_id *mm_idp, unsigned long addr, unsigned long len) { return pidfd_munmap(mm_idp->stub_pidfd, (void *)addr, len); }Obviously, this one requires a new syscall for the host kernel first.
Quoted 10 lines
Quoted 4 lines
6: add a watchdog thread that detects a stub which stops reporting
back (e.g. blocked SIGALRM) and SIGKILLs it, letting the monitor recover via the existing teardown.That also seems like an odd solution to me. Architecturally, UML first receives the SIGALRM and forwards it to the child. It would seem much easier to set a flag and clear it again when the process reports back that it received the SIGALRM. Then, when the kernel receives the next SIGALRM, just kill the child immediately if the flag is still set.
The flag-and-recheck scheme matches the ptrace path (
wait_stub_done), where the monitor is the tracer: it sees the stub's SIGALRM as awaitpid()signal-stop andPTRACE_CONTsit, so "monitor receives, then deals with the child" is accurate there. This patch is on the seccomp path (wait_stub_done_seccomp), which is architecturally different.Quoted 5 lines
Quoted 1 line
Or am I missing anything?
The fact that it is not actually different from an architectural standpoint. In both cases SIGALRM is reported back to the kernel. It obviously has to be reported back, otherwise scheduling would not work.
You're right.
In the seccomp path the monitor also receives the SIGALRM and forwards it to the stub: the per-vCPU POSIX timer targets the monitor thread and
um_timer()->os_alarm_process()->kill(stub, SIGALRM)forwards the tick. ptrace vs. seccomp differ only in how the monitor learns the stub stopped (waitpid vs. futex), not in the point you raised.Quoted 6 lines
So the watchdog thread is redundant. The inner
FUTEX_WAITloop inwait_stub_done_seccompalready wakes on every forwarded tick via EINTR; responsive stub acks by flippingdata->futexout ofFUTEX_IN_CHILD, a SIGALRM-blocking one doesn't, exactly your flag scheme. I'll drop the helper thread and timerfd and detect the stall inline: count consecutive ticks where the stub didn't report and gotoout_killpast a small threshold.
Quoted 3 lines
Not sure I follow and the description makes me wonder if it was written by an LLM. To me it seems overly complicated and incorrect in subtle ways.
Yes, I marked it with Assisted-by: Claude:claude-opus-4.8.
Thanks.