Skip to contentSkip to conversation

Add support for SafeStack

PATCH Applied Daniele Buono 2 participants 7 messages May 29, 2020 – Jun 17, 2020

lore
Share & tools
lore
#0 lore raw

v2 cover Add support for SafeStack

LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7

From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."

Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).

This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.

Changes since v1:
- CONFIG_SAFESTACK is now set up in configure, and not in the code
- Added option for a --disable-safe-stack in configure
- Configure checks if SafeStack is enabled by default in the compiler, and set the CONFIG_SAFESTACK accordingly
- Updated some comments in the code and the commit log

NOTE: I kept configure as Patch #3.
The reason is that the code changes will not be enabled without the configure, making the code fully functional if only Patches #1 or #2 are applied.
On the other hand, the configure patch will produce incorrect code if we request SafeStack and the other patches are not applied.

Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: add check for SafeStack in sigaltstack
configure: add flags to support SafeStack
check-block: enable iotests with SafeStack

 configure                    | 73 ++++++++++++++++++++++++++++++++++++
 include/qemu/coroutine_int.h |  5 +++
 tests/check-block.sh         | 12 +++++-
 util/coroutine-sigaltstack.c |  4 ++
 util/coroutine-ucontext.c    | 26 +++++++++++++
 5 files changed, 119 insertions(+), 1 deletion(-)
Signature · 1 line
2.26.2
#2 lore raw

PATCH v2 2/4 coroutine: add check for SafeStack in sigaltstack

Current implementation of LLVM's SafeStack is not compatible with code that uses an alternate stack created with sigaltstack(). Since coroutine-sigaltstack relies on sigaltstack(), it is not compatible with SafeStack. The resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.

In the future LLVM may provide a SafeStack implementation compatible with sigaltstack(). In the meantime, if SafeStack is desired, the coroutine implementation from coroutine-ucontext should be used.
As a safety check, add a control in coroutine-sigaltstack to throw a preprocessor #error if SafeStack is enabled and we are trying to use coroutine-sigaltstack to implement coroutines.

---
 util/coroutine-sigaltstack.c | 4 ++++
 1 file changed, 4 insertions(+)
util/coroutine-sigaltstack.c +4 -0
index f6fc49a0e5..aade82afb8 100644
@@ -30,6 +30,10 @@
 #include "qemu-common.h"
 #include "qemu/coroutine_int.h"
 
+#ifdef CONFIG_SAFESTACK
+#error "SafeStack is not compatible with code run in alternate signal stacks"
+#endif
+
 typedef struct {
     Coroutine base;
     void *stack;
Signature · 1 line
2.26.2
#1 lore raw

PATCH v2 1/4 coroutine: support SafeStack in ucontext backend

In reply to Daniele Buono

LLVM's SafeStack instrumentation does not yet support programs that make use of the APIs in ucontext.h
With the current implementation of coroutine-ucontext, the resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
This fix allocates an additional unsafe stack area for each coroutine, and sets the new unsafe stack pointer before calling swapcontext() in qemu_coroutine_new.
This is the only place where the pointer needs to be manually updated, since sigsetjmp/siglongjmp are already instrumented by LLVM to properly support SafeStack.
The additional stack is then freed in qemu_coroutine_delete.

---
 include/qemu/coroutine_int.h |  5 +++++
 util/coroutine-ucontext.c    | 26 ++++++++++++++++++++++++++
 2 files changed, 31 insertions(+)
include/qemu/coroutine_int.h +5 -0
index bd6b0468e1..1da148552f 100644
@@ -28,6 +28,11 @@
 #include "qemu/queue.h"
 #include "qemu/coroutine.h"
 
+#ifdef CONFIG_SAFESTACK
+/* Pointer to the unsafe stack, defined by the compiler */
+extern __thread void *__safestack_unsafe_stack_ptr;
+#endif
+
 #define COROUTINE_STACK_SIZE (1 << 20)
 
 typedef enum {
util/coroutine-ucontext.c +26 -0
index bd593e61bc..9108eb1294 100644
@@ -41,6 +41,11 @@ typedef struct {
     Coroutine base;
     void *stack;
     size_t stack_size;
+#ifdef CONFIG_SAFESTACK
+    /* Need an unsafe stack for each coroutine */
+    void *unsafe_stack;
+    size_t unsafe_stack_size;
+#endif
     sigjmp_buf env;
 
 #ifdef CONFIG_VALGRIND_H
@@ -140,6 +145,10 @@ Coroutine *qemu_coroutine_new(void)
     co = g_malloc0(sizeof(*co));
     co->stack_size = COROUTINE_STACK_SIZE;
     co->stack = qemu_alloc_stack(&co->stack_size);
+#ifdef CONFIG_SAFESTACK
+    co->unsafe_stack_size = COROUTINE_STACK_SIZE;
+    co->unsafe_stack = qemu_alloc_stack(&co->unsafe_stack_size);
+#endif
     co->base.entry_arg = &old_env; /* stash away our jmp_buf */
 
     uc.uc_link = &old_uc;
@@ -160,6 +169,20 @@ Coroutine *qemu_coroutine_new(void)
     /* swapcontext() in, siglongjmp() back out */
     if (!sigsetjmp(old_env, 0)) {
         start_switch_fiber(&fake_stack_save, co->stack, co->stack_size);
+#ifdef CONFIG_SAFESTACK
+        /*
+         * Before we swap the context, set the new unsafe stack
+         * The unsafe stack grows just like the normal stack, so start from
+         * the last usable location of the memory area.
+         * NOTE: we don't have to re-set the usp afterwards because we are
+         * coming back to this context through a siglongjmp.
+         * The compiler already wrapped the corresponding sigsetjmp call with
+         * code that saves the usp on the (safe) stack before the call, and
+         * restores it right after (which is where we return with siglongjmp).
+         */
+        void *usp = co->unsafe_stack + co->unsafe_stack_size;
+        __safestack_unsafe_stack_ptr = usp;
+#endif
         swapcontext(&old_uc, &uc);
     }
 
@@ -192,6 +215,9 @@ void qemu_coroutine_delete(Coroutine *co_)
 #endif
 
     qemu_free_stack(co->stack, co->stack_size);
+#ifdef CONFIG_SAFESTACK
+    qemu_free_stack(co->unsafe_stack, co->unsafe_stack_size);
+#endif
     g_free(co);
 }
 
Signature · 1 line
2.26.2
#3 lore raw

PATCH v2 3/4 configure: add flags to support SafeStack

In reply to Daniele Buono

This patch adds a flag to enable/disable the SafeStack instrumentation provided by LLVM.

On enable, make sure that the compiler supports the flags, and that we are using the proper coroutine implementation (coroutine-ucontext). On disable, explicitly disable the option if it was enabled by default.

While SafeStack is supported only on Linux, NetBSD, FreeBSD and macOS, we are not checking for the O.S. since this is already done by LLVM.

---
 configure | 73 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
configure +73 -0
index b969dee675..260772b2d5 100755
@@ -302,6 +302,7 @@ audio_win_int=""
 libs_qga=""
 debug_info="yes"
 stack_protector=""
+safe_stack=""
 use_containers="yes"
 gdb_bin=$(command -v "gdb-multiarch" || command -v "gdb")
 
@@ -1275,6 +1276,10 @@ for opt do
   ;;
   --disable-stack-protector) stack_protector="no"
   ;;
+  --enable-safe-stack) safe_stack="yes"
+  ;;
+  --disable-safe-stack) safe_stack="no"
+  ;;
   --disable-curses) curses="no"
   ;;
   --enable-curses) curses="yes"
@@ -1804,6 +1809,8 @@ disabled with --disable-FEATURE, default is enabled if available:
   debug-tcg       TCG debugging (default is disabled)
   debug-info      debugging information
   sparse          sparse checker
+  safe-stack      SafeStack Stack Smash Protection. Depends on
+                  clang/llvm >= 3.7 and requires coroutine backend ucontext.
 
   gnutls          GNUTLS cryptography support
   nettle          nettle cryptography support
@@ -5517,6 +5524,67 @@ if test "$debug_stack_usage" = "yes"; then
   fi
 fi
 
+##################################################
+# SafeStack
+
+
+if test "$safe_stack" = "yes"; then
+cat > $TMPC << EOF
+int main(int argc, char *argv[])
+{
+#if ! __has_feature(safe_stack)
+#error SafeStack Disabled
+#endif
+    return 0;
+}
+EOF
+  flag="-fsanitize=safe-stack"
+  # Check that safe-stack is supported and enabled.
+  if compile_prog "-Werror $flag" "$flag"; then
+    # Flag needed both at compilation and at linking
+    QEMU_CFLAGS="$QEMU_CFLAGS $flag"
+    QEMU_LDFLAGS="$QEMU_LDFLAGS $flag"
+  else
+    error_exit "SafeStack not supported by your compiler"
+  fi
+  if test "$coroutine" != "ucontext"; then
+    error_exit "SafeStack is only supported by the coroutine backend ucontext"
+  fi
+else
+cat > $TMPC << EOF
+int main(int argc, char *argv[])
+{
+#if defined(__has_feature)
+#if __has_feature(safe_stack)
+#error SafeStack Enabled
+#endif
+#endif
+    return 0;
+}
+EOF
+if test "$safe_stack" = "no"; then
+  # Make sure that safe-stack is disabled
+  if ! compile_prog "-Werror" ""; then
+    # SafeStack was already enabled, try to explicitly remove the feature
+    flag="-fno-sanitize=safe-stack"
+    if ! compile_prog "-Werror $flag" "$flag"; then
+      error_exit "Configure cannot disable SafeStack"
+    fi
+    QEMU_CFLAGS="$QEMU_CFLAGS $flag"
+    QEMU_LDFLAGS="$QEMU_LDFLAGS $flag"
+  fi
+else # "$safe_stack" = ""
+  # Set safe_stack to yes or no based on pre-existing flags
+  if compile_prog "-Werror" ""; then
+    safe_stack="no"
+  else
+    safe_stack="yes"
+    if test "$coroutine" != "ucontext"; then
+      error_exit "SafeStack is only supported by the coroutine backend ucontext"
+    fi
+  fi
+fi
+fi
 
 ##########################################
 # check if we have open_by_handle_at
@@ -6611,6 +6679,7 @@ echo "sparse enabled    $sparse"
 echo "strip binaries    $strip_opt"
 echo "profiler          $profiler"
 echo "static build      $static"
+echo "safe stack        $safe_stack"
 if test "$darwin" = "yes" ; then
     echo "Cocoa support     $cocoa"
 fi
@@ -8195,6 +8264,10 @@ if test "$ccache_cpp2" = "yes"; then
   echo "export CCACHE_CPP2=y" >> $config_host_mak
 fi
 
+if test "$safe_stack" = "yes"; then
+  echo "CONFIG_SAFESTACK=y" >> $config_host_mak
+fi
+
 # If we're using a separate build tree, set it up now.
 # DIRS are directories which we simply mkdir in the build tree;
 # LINKS are things to symlink back into the source tree
Signature · 1 line
2.26.2
#4 lore raw

PATCH v2 4/4 check-block: enable iotests with SafeStack

In reply to Daniele Buono

SafeStack is a stack protection technique implemented in llvm. It is enabled with a -fsanitize flag.
iotests are currently disabled when any -fsanitize option is used, because such options tend to produce additional warnings and false positives.

While common -fsanitize options are used to verify the code and not added in production, SafeStack's main use is in production environments to protect against stack smashing.

Since SafeStack does not print any warning or false positive, enable iotests when SafeStack is the only -fsanitize option used. This is likely going to be a production binary and we want to make sure it works correctly.

---
 tests/check-block.sh | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
tests/check-block.sh +11 -1
index ad320c21ba..8e29c868e5 100755
@@ -21,7 +21,17 @@ if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then
     exit 0
 fi
 
-if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then
+# Disable tests with any sanitizer except for SafeStack
+CFLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null )
+SANITIZE_FLAGS=""
+#Remove all occurrencies of -fsanitize=safe-stack
+for i in ${CFLAGS}; do
+        if [ "${i}" != "-fsanitize=safe-stack" ]; then
+                SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}"
+        fi
+done
+if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then
+    # Have a sanitize flag that is not allowed, stop
     echo "Sanitizers are enabled ==> Not running the qemu-iotests."
     exit 0
 fi
Signature · 1 line
2.26.2
#5 lore raw
In reply to Daniele Buono

Ping?

Quoted 49 lines · Daniele Buono

LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7

Quoted 1 line

From https://clang.llvm.org/docs/SafeStack.html:

"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."

Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).

This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.

Changes since v1:
- CONFIG_SAFESTACK is now set up in configure, and not in the code
- Added option for a --disable-safe-stack in configure
- Configure checks if SafeStack is enabled by default in the compiler, and set the CONFIG_SAFESTACK accordingly
- Updated some comments in the code and the commit log

NOTE: I kept configure as Patch #3.
The reason is that the code changes will not be enabled without the configure, making the code fully functional if only Patches #1 or #2 are applied.
On the other hand, the configure patch will produce incorrect code if we request SafeStack and the other patches are not applied.

Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: add check for SafeStack in sigaltstack
configure: add flags to support SafeStack
check-block: enable iotests with SafeStack

  configure                    | 73 ++++++++++++++++++++++++++++++++++++
  include/qemu/coroutine_int.h |  5 +++
  tests/check-block.sh         | 12 +++++-
  util/coroutine-sigaltstack.c |  4 ++
  util/coroutine-ucontext.c    | 26 +++++++++++++
  5 files changed, 119 insertions(+), 1 deletion(-)
#6 lore raw
In reply to Daniele Buono
Quoted 42 lines · Daniele Buono

LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7

From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."

Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).

This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.

Changes since v1:
- CONFIG_SAFESTACK is now set up in configure, and not in the code
- Added option for a --disable-safe-stack in configure
- Configure checks if SafeStack is enabled by default in the compiler, and set the CONFIG_SAFESTACK accordingly
- Updated some comments in the code and the commit log

NOTE: I kept configure as Patch #3.
The reason is that the code changes will not be enabled without the configure, making the code fully functional if only Patches #1 or #2 are applied.
On the other hand, the configure patch will produce incorrect code if we request SafeStack and the other patches are not applied.

Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: add check for SafeStack in sigaltstack
configure: add flags to support SafeStack
check-block: enable iotests with SafeStack

Quoted 11 lines
 configure                    | 73 ++++++++++++++++++++++++++++++++++++
 include/qemu/coroutine_int.h |  5 +++
 tests/check-block.sh         | 12 +++++-
 util/coroutine-sigaltstack.c |  4 ++
 util/coroutine-ucontext.c    | 26 +++++++++++++
 5 files changed, 119 insertions(+), 1 deletion(-)
Signature · 1 line
2.26.2

Thanks, applied to my block tree:
https://github.com/stefanha/qemu/commits/block

Stefan