Add support for SafeStack
cover Add support for SafeStack
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."
Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When running swapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0
Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: Add check for SafeStack in sigalstack
configure: add flag to enable SafeStack
check-block: Enable iotests with SafeStack
configure | 29 +++++++++++++++++++++++++++++ include/qemu/coroutine_int.h | 6 ++++++ tests/check-block.sh | 12 +++++++++++- util/coroutine-sigaltstack.c | 4 ++++ util/coroutine-ucontext.c | 25 +++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 1 deletion(-)
Signature · 1 line
2.26.2
PATCH 1/4 coroutine: support SafeStack in ucontext backend
LLVM's SafeStack instrumentation does not yet support programs that make use of the APIs in ucontext.h
With the current implementation of coroutine-ucontext, the resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
This fix allocates an additional unsafe stack area for each coroutine, and sets the new unsafe stack pointer before calling swapcontext() in qemu_coroutine_new.
This is the only place where the pointer needs to be manually updated, since sigsetjmp/siglongjmp are already instrumented by LLVM to properly support SafeStack.
The additional stack is then freed in qemu_coroutine_delete.
--- include/qemu/coroutine_int.h | 6 ++++++ util/coroutine-ucontext.c | 25 +++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
include/qemu/coroutine_int.h +6 -0
@@ -28,6 +28,12 @@#include "qemu/queue.h" #include "qemu/coroutine.h" +#if defined(__has_feature) &&__has_feature (safe_stack) +#define CONFIG_SAFESTACK 1 +/* Pointer to the unsafe stack, defined by the compiler */ +extern __threadvoid *__safestack_unsafe_stack_ptr; +#endif +#define COROUTINE_STACK_SIZE (1 << 20) typedef enum {
util/coroutine-ucontext.c +25 -0
@@ -41,6 +41,11 @@ typedef struct {Coroutine base;void *stack;size_t stack_size; +#ifdef CONFIG_SAFESTACK +/* Need an unsafe stack for each coroutine */ +void *unsafe_stack; +size_t unsafe_stack_size; +#endif sigjmp_buf env;#ifdef CONFIG_VALGRIND_H
@@ -140,6 +145,10 @@ Coroutine *qemu_coroutine_new(void) co =g_malloc0 (sizeof (*co)); co->stack_size =COROUTINE_STACK_SIZE ; co->stack =qemu_alloc_stack (&co->stack_size); +#ifdef CONFIG_SAFESTACK + co->unsafe_stack_size =COROUTINE_STACK_SIZE ; + co->unsafe_stack =qemu_alloc_stack (&co->unsafe_stack_size); +#endif co->base.entry_arg = &old_env;/* stash away our jmp_buf */ uc.uc_link = &old_uc;
@@ -160,6 +169,19 @@ Coroutine *qemu_coroutine_new(void)/* swapcontext() in, siglongjmp() back out */ if (!sigsetjmp (old_env,0 )) {start_switch_fiber (&fake_stack_save, co->stack, co->stack_size); +#ifdef CONFIG_SAFESTACK +/* +* Before we swap the context, set the new unsafe stack +* The unsafe stack grows just like the normal stack, so start from +* the last usable location of the memory area. +* NOTE: we don't have to re-set it afterwards because sigsetjmp was +* called with the original usp. Since we are not coming back with a +* swapcontext, but with a siglongjmp, when we are back here we +* already have usp restored to the valid one for this function +*/ +void *usp = co->unsafe_stack + co->unsafe_stack_size; + __safestack_unsafe_stack_ptr = usp; +#endif swapcontext (&old_uc, &uc); }
@@ -192,6 +214,9 @@ void qemu_coroutine_delete(Coroutine *co_)#endif qemu_free_stack (co->stack, co->stack_size); +#ifdef CONFIG_SAFESTACK +qemu_free_stack (co->unsafe_stack, co->unsafe_stack_size); +#endif g_free (co); }
Signature · 1 line
2.26.2
Re: [PATCH 1/4] coroutine: support SafeStack in ucontext backend
include/qemu/coroutine_int.h line 32
Quoted 10 lines · Daniele Buono
include/qemu/coroutine_int.h +2
-0@@ -28,6 +28,12 @@#include "qemu/queue.h" #include "qemu/coroutine.h" +#if defined(__has_feature) &&__has_feature (safe_stack) +#define CONFIG_SAFESTACK 1
Please perform this feature check in ./configure. That way CONFIG_SAFESTACK will be defined alongside all the other CONFIG_* values and be available to C and Makefiles via config-host.h and config-host.mak.
include/qemu/coroutine_int.h Coroutine *qemu_coroutine_new(void) · line 180
Quoted 13 lines
include/qemu/coroutine_int.h +9
-0@@ -160,6 +169,19 @@ Coroutine *qemu_coroutine_new(void)/* swapcontext() in, siglongjmp() back out */ if (!sigsetjmp (old_env,0 )) {start_switch_fiber (&fake_stack_save, co->stack, co->stack_size); +#ifdef CONFIG_SAFESTACK +/* +* Before we swap the context, set the new unsafe stack +* The unsafe stack grows just like the normal stack, so start from +* the last usable location of the memory area. +* NOTE: we don't have to re-set it afterwards because sigsetjmp was +* called with the original usp. Since we are not coming back with a +* swapcontext, but with a siglongjmp, when we are back here we +* already have usp restored to the valid one for this function
I don't understand this comment. __safestack_unsafe_stack_ptr is a thread-local variable, not a CPU register. How will siglongjmp() automatically restore it?
Quoted 7 lines
include/qemu/coroutine_int.h +4
-0+*/ +void *usp = co->unsafe_stack + co->unsafe_stack_size; + __safestack_unsafe_stack_ptr = usp; +#endif swapcontext (&old_uc, &uc); }
Re: [PATCH 1/4] coroutine: support SafeStack in ucontext backend
Hi Stefan,
thank you so much for reviewing this.
See my answers below:
Quoted 9 lines · Stefan Hajnoczi
Quoted 8 lines · Daniele Buono
include/qemu/coroutine_int.h +0
-0@@ -28,6 +28,12 @@#include "qemu/queue.h" #include "qemu/coroutine.h"
Quoted 8 lines
include/qemu/coroutine_int.h line 32
Quoted 2 lines
include/qemu/coroutine_int.h +2
-0+#if defined(__has_feature) &&__has_feature (safe_stack) +#define CONFIG_SAFESTACK 1Please perform this feature check in ./configure. That way
CONFIG_SAFESTACKwill be defined alongside all the other CONFIG_* values and be available to C and Makefiles viaconfig-host.hand config-host.mak.
Sure, will do this in v2.
Quoted 9 lines
Quoted 9 lines
include/qemu/coroutine_int.h +5
-0@@ -160,6 +169,19 @@ Coroutine *qemu_coroutine_new(void)/* swapcontext() in, siglongjmp() back out */ if (!sigsetjmp (old_env,0 )) {start_switch_fiber (&fake_stack_save, co->stack, co->stack_size); +#ifdef CONFIG_SAFESTACK +/* +* Before we swap the context, set the new unsafe stack +* The unsafe stack grows just like the normal stack, so start from +* the last usable location of the memory area.
Quoted 9 lines
include/qemu/coroutine_int.h Coroutine *qemu_coroutine_new(void) · line 180
Quoted 4 lines
include/qemu/coroutine_int.h +4
-0+* NOTE: we don't have to re-set it afterwards because sigsetjmp was +* called with the original usp. Since we are not coming back with a +* swapcontext, but with a siglongjmp, when we are back here we +* already have usp restored to the valid one for this function I don't understand this comment.
__safestack_unsafe_stack_ptris a thread-local variable, not a CPU register. How willsiglongjmp()automatically restore it?
Correct, setjmp/longjmp have no visibility of the unsafe stack. What I meant is that it is not automatically restored by the longjmp itself, but by code the compiler adds around the sigsetjmp.
Specifically, every sigsetjmp/sigjmp is intercepted by the compiler, the current value of __safestack_unsafe_stack_ptr is saved on the normal (safe) stack.
Right after the sigsetjmp call it is then restored.
I will change the comment to make it clearer.
In practice, this is what happens:
Original clang implementation in qemu_coroutine_new:
----
40130c: callq 4008d0 <__sigsetjmp@plt> 401311: cmp $0x0,%eax 401314: jne 40132d <qemu_coroutine_new+0x12d>
----
Clang Implementation with safestack:
----
4027a7: mov %rdx,-0x38(%rbp) <- Save unsafe ptr onto the safe stack
[...]
40289c: callq 401410 <__sigsetjmp@plt> 4028a1: mov 0x201738(%rip),%rdi # 603fe0 <__safestack_unsafe_stack_ptr@@Base+0x603fe0> 4028a8: mov -0x38(%rbp),%rbx 4028ac: mov %rbx,%fs:(%rdi) <- Restore the unsafe ptr 4028b0: cmp $0x0,%eax 4028b3: jne 4028d9 <qemu_coroutine_new+0x179>
Quoted 7 lines
Quoted 7 lines
include/qemu/coroutine_int.h +4
-0+*/ +void *usp = co->unsafe_stack + co->unsafe_stack_size; + __safestack_unsafe_stack_ptr = usp; +#endif swapcontext (&old_uc, &uc); }
Re: [PATCH 1/4] coroutine: support SafeStack in ucontext backend
Quoted 43 lines · Daniele Buono
Quoted 10 lines · Stefan Hajnoczi
Quoted 9 lines · Daniele Buono
util/coroutine-ucontext.c +5
-0@@ -160,6 +169,19 @@ Coroutine *qemu_coroutine_new(void)/* swapcontext() in, siglongjmp() back out */ if (!sigsetjmp (old_env,0 )) {start_switch_fiber (&fake_stack_save, co->stack, co->stack_size); +#ifdef CONFIG_SAFESTACK +/* +* Before we swap the context, set the new unsafe stack +* The unsafe stack grows just like the normal stack, so start from +* the last usable location of the memory area. Quoted 9 lines
util/coroutine-ucontext.c Coroutine *qemu_coroutine_new(void) · line 180
Quoted 4 lines
util/coroutine-ucontext.c +4
-0+* NOTE: we don't have to re-set it afterwards because sigsetjmp was +* called with the original usp. Since we are not coming back with a +* swapcontext, but with a siglongjmp, when we are back here we +* already have usp restored to the valid one for this function I don't understand this comment.
__safestack_unsafe_stack_ptris a thread-local variable, not a CPU register. How willsiglongjmp()automatically restore it?Correct, setjmp/longjmp have no visibility of the unsafe stack. What I meant is that it is not automatically restored by the longjmp itself, but by code the compiler adds around the sigsetjmp.
Specifically, every sigsetjmp/sigjmp is intercepted by the compiler, the current value of
__safestack_unsafe_stack_ptris saved on the normal (safe) stack.
Right after the sigsetjmp call it is then restored.I will change the comment to make it clearer.
In practice, this is what happens:
Original clang implementation in
qemu_coroutine_new:
----40130c: callq 4008d0 <__sigsetjmp@plt> 401311: cmp $0x0,%eax 401314: jne 40132d <qemu_coroutine_new+0x12d>----
Clang Implementation with safestack:
----
4027a7: mov %rdx,-0x38(%rbp) <- Save unsafe ptr onto the safe stack
[...]
Quoted 7 lines
40289c: callq 401410 <__sigsetjmp@plt> 4028a1: mov 0x201738(%rip),%rdi # 603fe0 <__safestack_unsafe_stack_ptr@@Base+0x603fe0> 4028a8: mov -0x38(%rbp),%rbx 4028ac: mov %rbx,%fs:(%rdi) <- Restore the unsafe ptr 4028b0: cmp $0x0,%eax 4028b3: jne 4028d9 <qemu_coroutine_new+0x179>
Oh, that's interesting. Thanks for explaining and updating the comment.
Stefan
PATCH 2/4 coroutine: Add check for SafeStack in sigalstack
In reply to Daniele BuonoLLVM's SafeStack instrumentation cannot be used inside signal handlers that make use of sigaltstack().
Since coroutine-sigaltstack relies on sigaltstack(), it is not compatible with SafeStack. The resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
To avoid this, we add a check in coroutine-sigaltstack that throws a preprocessor #error and interrupt the compilation if SafeStack is enabled.
--- util/coroutine-sigaltstack.c | 4 ++++ 1 file changed, 4 insertions(+)
util/coroutine-sigaltstack.c +4 -0
@@ -30,6 +30,10 @@#include "qemu-common.h" #include "qemu/coroutine_int.h" +#ifdef CONFIG_SAFESTACK +#error "SafeStack does not work with sigaltstack's implementation" +#endif + typedef struct { Coroutine base; void *stack;
Signature · 1 line
2.26.2
Re: [PATCH 2/4] coroutine: Add check for SafeStack in sigalstack
Quoted 21 lines · Daniele Buono
LLVM's SafeStack instrumentation cannot be used inside signal handlers that make use of
sigaltstack().
Since coroutine-sigaltstack relies onsigaltstack(), it is not compatible with SafeStack. The resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
To avoid this, we add a check in coroutine-sigaltstack that throws a preprocessor #error and interrupt the compilation if SafeStack is enabled.Signed-off-by Daniele Buono <[email protected]>--- util/coroutine-sigaltstack.c | 4 ++++ 1 file changed, 4 insertions(+)util/coroutine-sigaltstack.c +0
-0
util/coroutine-sigaltstack.c line 39
Quoted 10 lines
util/coroutine-sigaltstack.c +4
-0@@ -30,6 +30,10 @@#include "qemu-common.h" #include "qemu/coroutine_int.h" +#ifdef CONFIG_SAFESTACK +#error "SafeStack does not work with sigaltstack's implementation" +#endif + typedef struct { Coroutine base; void *stack;
Re: [PATCH 2/4] coroutine: Add check for SafeStack in sigalstack
In reply to Daniele BuonoOn Wed, Apr 29, 2020 at 03:44:18PM -0400, Daniele Buono wrote:
s/sigalstack/sigaltstack/ in the commit message.
Quoted 20 lines
LLVM's SafeStack instrumentation cannot be used inside signal handlers that make use of
sigaltstack().
Since coroutine-sigaltstack relies onsigaltstack(), it is not compatible with SafeStack. The resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
To avoid this, we add a check in coroutine-sigaltstack that throws a preprocessor #error and interrupt the compilation if SafeStack is enabled.Signed-off-by Daniele Buono <[email protected]>--- util/coroutine-sigaltstack.c | 4 ++++ 1 file changed, 4 insertions(+)util/coroutine-sigaltstack.c +0
-0
util/coroutine-sigaltstack.c line 35
Quoted 6 lines
util/coroutine-sigaltstack.c +3
-0@@ -30,6 +30,10 @@#include "qemu-common.h" #include "qemu/coroutine_int.h" +#ifdef CONFIG_SAFESTACK +#error "SafeStack does not work with sigaltstack's implementation" +#endif
Neither the commit description nor the #error message explain why it doesn't work. Could it work in the future or is there a fundamental reason why it will never work?
Stefan
Re: [PATCH 2/4] coroutine: Add check for SafeStack in sigalstack
Sorry, missed the question at the end of the email. Will change the commit and error message to explain better in v2.
Similar to the ucontext, case, sigaltstack does not work out of the box because it requires a stack to be allocated by the user.
I'll be honest, I didn't check the details of how sigaltstack is used in coroutine-sigaltstack. It is very possible that this coroutine version can also be adapted to run properly with SafeStack. coroutine_trampoline is probably the place where we can set the usp so that, when coroutine_bootstrap is called, it is done with the new unsafe stack.
My initial focus was on the ucontext implementation since that is the default one and mostly used. For now, I am just blocking the user from using coroutine-sigaltstack with SafeStack.
However, if you are interested, Ican try to add support to sigaltstack in the future.
Quoted 37 lines · Stefan Hajnoczi
On Wed, Apr 29, 2020 at 03:44:18PM -0400, Daniele Buono wrote:
s/sigalstack/sigaltstack/ in the commit message.
Quoted 20 lines
LLVM's SafeStack instrumentation cannot be used inside signal handlers that make use of
sigaltstack().
Since coroutine-sigaltstack relies onsigaltstack(), it is not compatible with SafeStack. The resulting binary is incorrect, with different coroutines sharing the same unsafe stack and producing undefined behavior at runtime.
To avoid this, we add a check in coroutine-sigaltstack that throws a preprocessor #error and interrupt the compilation if SafeStack is enabled.Signed-off-by Daniele Buono <[email protected]>--- util/coroutine-sigaltstack.c | 4 ++++ 1 file changed, 4 insertions(+)util/coroutine-sigaltstack.c +0
-0util/coroutine-sigaltstack.c line 35
Quoted 6 lines
util/coroutine-sigaltstack.c +3
-0@@ -30,6 +30,10 @@#include "qemu-common.h" #include "qemu/coroutine_int.h" +#ifdef CONFIG_SAFESTACK +#error "SafeStack does not work with sigaltstack's implementation" +#endif Neither the commit description nor the #error message explain why it doesn't work. Could it work in the future or is there a fundamental reason why it will never work?
Stefan
PATCH 3/4 configure: add flag to enable SafeStack
In reply to Daniele BuonoThis patch adds a flag to enable the SafeStack instrumentation provided by LLVM.
The checks make sure that the compiler supports the flags, and that we are using the proper coroutine implementation (coroutine-ucontext). While SafeStack is supported only on Linux, NetBSD, FreeBSD and macOS, we are not checking for the O.S. since this is already done by LLVM.
--- configure | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
configure +29 -0
@@ -302,6 +302,7 @@ audio_win_int="" libs_qga="" debug_info="yes" stack_protector="" +safe_stack="no" use_containers="yes" gdb_bin=$(command -v "gdb")
@@ -1275,6 +1276,8 @@ for opt do ;; --disable-stack-protector) stack_protector="no" ;; + --enable-safe-stack) safe_stack="yes" + ;; --disable-curses) curses="no" ;; --enable-curses) curses="yes"
@@ -1774,6 +1777,8 @@ Advanced options (experts only): --with-coroutine=BACKEND coroutine backend. Supported options: ucontext, sigaltstack, windows --enable-gcov enable test coverage analysis with gcov + --enable-safe-stack enable the SafeStack stack protection. Depends on + clang/llvm >= 3.7 and coroutine backend ucontext. --gcov=GCOV use specified gcov [$gcov_tool] --disable-blobs disable installing provided firmware blobs --with-vss-sdk=SDK-path enable Windows VSS support in QEMU Guest Agent
@@ -5501,6 +5506,29 @@ if test "$debug_stack_usage" = "yes"; then fi fi +################################################## +# Check if SafeStack is enabled and supported + +if test "$safe_stack" = "yes"; then + cat > $TMPC << EOF +int main(int argc, char *argv[]) +{ + return 0; +} +EOF + flag="-fsanitize=safe-stack" + # Check that safe-stack is supported. + if compile_prog "-Werror $flag" ""; then + # Flag needed both at compilation and at linking + QEMU_CFLAGS="$QEMU_CFLAGS $flag" + QEMU_LDFLAGS="$QEMU_LDFLAGS $flag" + else + error_exit "SafeStack not supported by your compiler" + fi + if test "$coroutine" != "ucontext"; then + error_exit "SafeStack is only supported by the coroutine backend ucontext" + fi +fi ########################################## # check if we have open_by_handle_at
@@ -6595,6 +6623,7 @@ echo "sparse enabled $sparse" echo "strip binaries $strip_opt" echo "profiler $profiler" echo "static build $static" +echo "safe stack $safe_stack" if test "$darwin" = "yes" ; then echo "Cocoa support $cocoa" fi
Signature · 1 line
2.26.2
Re: [PATCH 3/4] configure: add flag to enable SafeStack
Quoted 11 lines · Daniele Buono
This patch adds a flag to enable the SafeStack instrumentation provided by LLVM.
The checks make sure that the compiler supports the flags, and that we are using the proper coroutine implementation (coroutine-ucontext). While SafeStack is supported only on Linux, NetBSD, FreeBSD and macOS, we are not checking for theO.S. since this is already done by LLVM.Signed-off-by Daniele Buono <[email protected]>--- configure | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)
Great, this can become Patch 1 and it can set CONFIG_SAFESTACK as mentioned in my earlier reply.
configure audio_win_int="" · line 305
Quoted 9 lines
configure +1
-0@@ -302,6 +302,7 @@ audio_win_int="" libs_qga="" debug_info="yes" stack_protector="" +safe_stack="no"
The comment above this says:
# Always add --enable-foo and --disable-foo command line args.
Please add --disable-safe-stack.
Re: [PATCH 3/4] configure: add flag to enable SafeStack
Hi Stefan,
I would feel more confident by adding another check in configure to make sure that the user didn't enable SafeStack manually through other means, like manually setting the option through extra_cflags.
What do you think?
Quoted 32 lines · Stefan Hajnoczi
Quoted 11 lines · Daniele Buono
This patch adds a flag to enable the SafeStack instrumentation provided by LLVM.
The checks make sure that the compiler supports the flags, and that we are using the proper coroutine implementation (coroutine-ucontext). While SafeStack is supported only on Linux, NetBSD, FreeBSD and macOS, we are not checking for theO.S. since this is already done by LLVM.Signed-off-by Daniele Buono <[email protected]>--- configure | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+)Great, this can become Patch 1 and it can set
CONFIG_SAFESTACKas mentioned in my earlier reply.configure audio_win_int="" · line 305
Quoted 9 lines
configure +1
-0@@ -302,6 +302,7 @@ audio_win_int="" libs_qga="" debug_info="yes" stack_protector="" +safe_stack="no"The comment above this says:
# Always add --enable-foo and --disable-foo command line args.
Please add --disable-safe-stack.
Re: [PATCH 3/4] configure: add flag to enable SafeStack
Quoted 4 lines · Daniele Buono
I would feel more confident by adding another check in configure to make sure that the user didn't enable SafeStack manually through other means, like manually setting the option through
extra_cflags.
What do you think?
Sure, a compile_prog call could check if SafeStack is enable when it shouldn't be.
This can be done together with a --disable option.
Stefan
Re: [PATCH 3/4] configure: add flag to enable SafeStack
Hi Stefan, a quick clarification on configure:
right now, in configure, there's
* "Advanced Options (experts only)"
which usually don't have both enable and disable for each option, and
* "Optional features, enabled with --enable-FEATURE and disabled with --disable-FEATURE, default is enabled if available:"
How do you think SafeStack should be classified?
* If we do it as Advanced option, we should probably force it disabled unless --enable-safe-stack is provided. In this case --disable-safe-stack is not really necessary.
* If we do it as optional feature, I have two ways to handle the default:
1. enable/disable based on the compilation flags given to configure
2. enable every time the provided compiler supports it
Quoted 13 lines · Stefan Hajnoczi
Quoted 4 lines · Daniele Buono
I would feel more confident by adding another check in configure to make sure that the user didn't enable SafeStack manually through other means, like manually setting the option through
extra_cflags.
What do you think?Sure, a
compile_progcall could check if SafeStack is enable when it shouldn't be.This can be done together with a --disable option.
Stefan
PATCH 4/4 check-block: Enable iotests with SafeStack
In reply to Daniele BuonoSafeStack is a stack protection technique implemented in llvm. It is enabled with a -fsanitize flag.
iotests are currently disabled when any -fsanitize option is used. Since SafeStack is useful on production environments, and its implementation may break the binary, filter it out when the check is performed, so that if SafeStack was the only -fsanitize option, iotests are still performed.
--- tests/check-block.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
tests/check-block.sh +11 -1
@@ -21,7 +21,17 @@ if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then exit 0 fi -if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then +# Disable tests with any sanitizer except for SafeStack +CFLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ) +SANITIZE_FLAGS="" +#Remove all occurrencies of -fsanitize=safe-stack +for i in ${CFLAGS}; do + if [ "${i}" != "-fsanitize=safe-stack" ]; then + SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}" + fi +done +if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then + # Have a sanitize flag that is not allowed, stop echo "Sanitizers are enabled ==> Not running the qemu-iotests." exit 0 fi
Signature · 1 line
2.26.2
Re: [PATCH 4/4] check-block: Enable iotests with SafeStack
Quoted 7 lines · Daniele Buono
SafeStack is a stack protection technique implemented in llvm. It is enabled with a -fsanitize flag.
iotests are currently disabled when any -fsanitize option is used. Since SafeStack is useful on production environments, and its implementation may break the binary, filter it out when the check is performed, so that if SafeStack was the only -fsanitize option, iotests are still performed.
I can't parse this sentence. What does "its implementation may break the binary" mean? Do you mean it's worth running tests with SafeStack enabled because it exposes failures that go unnoticed without SafeStack?
Quoted 19 lines
Signed-off-by Daniele Buono <[email protected]>--- tests/check-block.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)tests/check-block.sh +4
-1@@ -21,7 +21,17 @@ if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then exit 0 fi -if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then +# Disable tests with any sanitizer except for SafeStack +CFLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ) +SANITIZE_FLAGS="" +#Remove all occurrencies of -fsanitize=safe-stack
tests/check-block.sh if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then · line 37
Quoted 10 lines
tests/check-block.sh +7
-0+for i in ${CFLAGS}; do + if [ "${i}" != "-fsanitize=safe-stack" ]; then + SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}" + fi +done +if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then + # Have a sanitize flag that is not allowed, stop echo "Sanitizers are enabled ==> Not running the qemu-iotests." exit 0 fi
The commit that disabled check-block.sh with sanitizers said:
The sanitizers (especially the address sanitizer from Clang) are sometimes printing out warnings or false positives - this spoils the output of the iotests, causing some of the tests to fail.
It seems fine to allow SafeStack if check-block.sh currently passes with it enabled. Does it pass and produce no extra output?
Stefan
Re: [PATCH 4/4] check-block: Enable iotests with SafeStack
Quoted 9 lines · Stefan Hajnoczi
Quoted 7 lines · Daniele Buono
SafeStack is a stack protection technique implemented in llvm. It is enabled with a -fsanitize flag.
iotests are currently disabled when any -fsanitize option is used. Since SafeStack is useful on production environments, and its implementation may break the binary, filter it out when the check is performed, so that if SafeStack was the only -fsanitize option, iotests are still performed.
Quoted 3 lines
I can't parse this sentence. What does "its implementation may break the binary" mean? Do you mean it's worth running tests with SafeStack enabled because it exposes failures that go unnoticed without SafeStack?
What I meant is that, without proper changes, SafeStack breaks co-routines. Since they are heavily used in the io subsystem, this is probably the best class of tests to make sure co-routines are working fine with SafeStack.
I initially re-enabled the iotests for my internal testing.
Since I wasn't seeing any issue, I thought it would be useful to keep running this to make sure future implementations of SafeStack won't break co-routines again.
Quoted 33 lines
Quoted 20 lines
Signed-off-by Daniele Buono <[email protected]>--- tests/check-block.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)tests/check-block.sh +4
-1@@ -21,7 +21,17 @@ if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then exit 0 fi -if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then +# Disable tests with any sanitizer except for SafeStack +CFLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ) +SANITIZE_FLAGS="" +#Remove all occurrencies of -fsanitize=safe-stacktests/check-block.sh if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then · line 37
Quoted 10 lines
tests/check-block.sh +7
-0+for i in ${CFLAGS}; do + if [ "${i}" != "-fsanitize=safe-stack" ]; then + SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}" + fi +done +if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then + # Have a sanitize flag that is not allowed, stop echo "Sanitizers are enabled ==> Not running the qemu-iotests." exit 0 fiThe commit that disabled
check-block.shwith sanitizers said:
Quoted 7 lines
The sanitizers (especially the address sanitizer from Clang) are sometimes printing out warnings or false positives - this spoils the output of the iotests, causing some of the tests to fail.
It seems fine to allow SafeStack if
check-block.shcurrently passes with it enabled. Does it pass and produce no extra output?
Yes, that was the idea. SafeStack should pass the tests without extra output.
It did (pass) on my testing machine. However I don't remember if I did the full (slow) check or only the partial one.
Will check again before I submit v2
Quoted 2 lines
Stefan
Re: [PATCH 4/4] check-block: Enable iotests with SafeStack
Quoted 41 lines · Daniele Buono
Quoted 33 lines · Stefan Hajnoczi
Quoted 18 lines · Daniele Buono
Signed-off-by Daniele Buono <[email protected]>--- tests/check-block.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)tests/check-block.sh +4
-1@@ -21,7 +21,17 @@ if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then exit 0 fi -if grep -q "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ; then +# Disable tests with any sanitizer except for SafeStack +CFLAGS=$( grep "CFLAGS.*-fsanitize" config-host.mak 2>/dev/null ) +SANITIZE_FLAGS="" +#Remove all occurrencies of -fsanitize=safe-stacktests/check-block.sh if grep -q "CONFIG_GPROF=y" config-host.mak 2>/dev/null ; then · line 36
Quoted 10 lines
tests/check-block.sh +7
-0+for i in ${CFLAGS}; do + if [ "${i}" != "-fsanitize=safe-stack" ]; then + SANITIZE_FLAGS="${SANITIZE_FLAGS} ${i}" + fi +done +if echo ${SANITIZE_FLAGS} | grep -q "\-fsanitize" 2>/dev/null; then + # Have a sanitize flag that is not allowed, stop echo "Sanitizers are enabled ==> Not running the qemu-iotests." exit 0 fiThe commit that disabled
check-block.shwith sanitizers said:Quoted 7 lines
The sanitizers (especially the address sanitizer from Clang) are sometimes printing out warnings or false positives - this spoils the output of the iotests, causing some of the tests to fail.
It seems fine to allow SafeStack if
check-block.shcurrently passes with it enabled. Does it pass and produce no extra output?
Quoted 7 lines
Yes, that was the idea. SafeStack should pass the tests without extra output.
It did (pass) on my testing machine. However I don't remember if I did the full (slow) check or only the partial one.
Will check again before I submit v2
Great, thanks!
Stefan
Quoted 22 lines · Daniele Buono
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
Quoted 10 lines
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When runningswapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0
Clang10:
Quoted 14 lines
Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: Add check for SafeStack in sigalstack
configure: add flag to enable SafeStack
check-block: Enable iotests with SafeStackconfigure | 29 +++++++++++++++++++++++++++++ include/qemu/coroutine_int.h | 6 ++++++ tests/check-block.sh | 12 +++++++++++- util/coroutine-sigaltstack.c | 4 ++++ util/coroutine-ucontext.c | 25 +++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 1 deletion(-)
+Alex & Daniel who keep track on CI stuff.
Quoted 22 lines · Daniele Buono
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
Quoted 10 lines
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When runningswapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0
I noticed building using SafeStack is slower, and running with it is even sloooower. It makes sense to have this integrated if we use it regularly. Do you have plan for this? Using public CI doesn't seem reasonable.
Quoted 14 lines
Daniele Buono (4):
coroutine: support SafeStack in ucontext backend
coroutine: Add check for SafeStack in sigalstack
configure: add flag to enable SafeStack
check-block: Enable iotests with SafeStackconfigure | 29 +++++++++++++++++++++++++++++ include/qemu/coroutine_int.h | 6 ++++++ tests/check-block.sh | 12 +++++++++++- util/coroutine-sigaltstack.c | 4 ++++ util/coroutine-ucontext.c | 25 +++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 1 deletion(-)
Quoted 36 lines · Philippe Mathieu-Daudé
+Alex & Daniel who keep track on CI stuff.
Quoted 22 lines · Daniele Buono
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
Quoted 10 lines
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When runningswapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0
Quoted 3 lines
I noticed building using SafeStack is slower, and running with it is even sloooower. It makes sense to have this integrated if we use it regularly. Do you have plan for this? Using public CI doesn't seem reasonable.
The runtime behaviour is rather odd, given the docs they provide:
"The performance overhead of the SafeStack instrumentation is
less than 0.1% on average across a variety of benchmarks This is mainly because most small functions do not have any variables that require the unsafe stack and, hence, do not need unsafe stack frames to be created. The cost of creating unsafe stack frames for large functions is amortized by the cost of executing the function.
In some cases, SafeStack actually improves the performance"
Regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 51 lines · Daniel P. Berrangé
Quoted 36 lines · Philippe Mathieu-Daudé
+Alex & Daniel who keep track on CI stuff.
Quoted 22 lines · Daniele Buono
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
Quoted 10 lines
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When runningswapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0Quoted 3 lines
I noticed building using SafeStack is slower, and running with it is even sloooower. It makes sense to have this integrated if we use it regularly. Do you have plan for this? Using public CI doesn't seem reasonable.
The runtime behaviour is rather odd, given the docs they provide:
"The performance overhead of the SafeStack instrumentation is
less than 0.1% on average across a variety of benchmarks This is mainly because most small functions do not have any variables that require the unsafe stack and, hence, do not need unsafe stack frames to be created. The cost of creating unsafe stack frames for large functions is amortized by the cost of executing the function.
Quoted 1 line
In some cases, SafeStack actually improves the performance"
I'm sorry I was testing this with a single core instead of all of them... Thanks for looking at the doc.
Quoted 4 lines
Regards,
Daniel
Hello everybody, just pinging since it it's been a few days.
Quoted 62 lines · Philippe Mathieu-Daudé
Quoted 51 lines · Daniel P. Berrangé
Quoted 36 lines · Philippe Mathieu-Daudé
+Alex & Daniel who keep track on CI stuff.
Quoted 22 lines · Daniele Buono
LLVM supports SafeStack instrumentation to protect against stack buffer overflows, since version 3.7
From https://clang.llvm.org/docs/SafeStack.html:
"It works by separating the program stack into two distinct regions: the safe stack and the unsafe stack. The safe stack stores return addresses, register spills, and local variables that are always accessed in a safe way, while the unsafe stack stores everything else. This separation ensures that buffer overflows on the unsafe stack cannot be used to overwrite anything on the safe stack."Unfortunately, the use of two stack regions does not cope well with QEMU's coroutines. The second stack region is not properly set up with both ucontext and sigaltstack, so multiple coroutines end up sharing the same memory area for the unsafe stack, causing undefined behaviors at runtime (and most iochecks to fail).
This patch series fixes the implementation of the ucontext backend and make sure that sigaltstack is never used if the compiler is applying the SafeStack instrumentation. It also adds a configure flag to enable SafeStack, and enables iotests when SafeStack is used.
Quoted 10 lines
This is an RFC mainly because of the low-level use of the SafeStack runtime.
When runningswapcontext(), we have to manually set the unsafe stack pointer to the new area allocated for the coroutine. LLVM does not allow this by using builtin, so we have to use implementation details that may change in the future.
This patch has been tested briefly ( make check on an x86 system ) with clang v3.9, v4.0, v5.0, v6.0
Heavier testing, with make check-acceptance has been performed with clang v7.0Quoted 3 lines
I noticed building using SafeStack is slower, and running with it is even sloooower. It makes sense to have this integrated if we use it regularly. Do you have plan for this? Using public CI doesn't seem reasonable.
The runtime behaviour is rather odd, given the docs they provide:
"The performance overhead of the SafeStack instrumentation is less than 0.1% on average across a variety of benchmarks This is mainly because most small functions do not have any variables that require the unsafe stack and, hence, do not need unsafe stack frames to be created. The cost of creating unsafe stack frames for large functions is amortized by the cost of executing the function.
Quoted 1 line
In some cases, SafeStack actually improves the performance"
I'm sorry I was testing this with a single core instead of all of them... Thanks for looking at the doc.
Quoted 4 lines
Regards,
Daniel
Quoted 1 line · Daniele Buono
Hello everybody, just pinging since it it's been a few days.
Hi Daniele,
Sorry I'm late to the party. This looks useful, the patches are not invasive and provide the option of enabling extra security.
I have left comments on specific patches.
Stefan