Add Thread Sanitizer support to QEMU
v3 cover Add Thread Sanitizer support to QEMU
Changes in v3:
- Fixed issue in tsan changes to start_switch_fiber(), found by --enable-sanitizers testing.
- Removed the UC_TRACE() code.
- Removed the tb_destroy callback.
v2: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg01534.html
This patch series continues the work done by Emilio Cota and others to add Thread Sanitizer (TSan) support to QEMU.
The starting point for this work was Emilio's branch here:
https://github.com/cota/qemu/commits/tsan
specifically this commit: 0be125fc0afd47218b34d2019abdd19b644f3199
The main purpose of this patch is to enable TSan support so that QEMU developers can start using the tool.
We found this tool useful and even ran it on our recent changes in the cpu-locks series, which fixes many warnings.
Clearly there is work to do here to clean up all the warnings. :) We have also made an effort to introduce enough of the TSan suppression mechanisms, so that others can continue this work.
This series adds support for:
- configure option for --enable-tsan.
- testing.rst has the full details on how to use TSan with or without docker, including all the suppression mechanisms.
- We added an Ubuntu 20.04 docker that supports TSan builds.
- test-tsan is a new docker test that builds and runs make check under TSan.
- We added an example blacklist file for files or functions TSan should ignore at compile time. This can now be specified manually.
- Added a suppression file for TSan to suppress certain warnings at run time.
- Added tsan.h with annotations which also can be used to suppress warnings.
Emilio G. Cota (7):
cpu: convert queued work to a QSIMPLEQ
thread: add qemu_spin_destroy
cputlb: destroy CPUTLB with tlb_destroy
qht: call qemu_spin_destroy for head buckets
tcg: call qemu_spin_destroy for tb->jmp_lock
translate-all: call qemu_spin_destroy for PageDesc
thread: add tsan annotations to QemuSpin
Lingfeng Yang (1):
configure: add --enable-tsan flag + fiber annotations for
coroutine-ucontext
Robert Foley (5):
tests/docker: Added docker build support for TSan. include/qemu: Added tsan.h for annotations.
util: Added tsan annotate for thread name.
docs: Added details on TSan to testing.rst
tests: Disable select tests under TSan, which hit TSan issue.
accel/tcg/cputlb.c | 15 +++ accel/tcg/translate-all.c | 17 ++++ configure | 47 ++++++++- cpus-common.c | 25 ++--- cpus.c | 14 ++- docs/devel/testing.rst | 107 +++++++++++++++++++++ exec.c | 1 + hw/core/cpu.c | 1 + include/exec/exec-all.h | 8 ++ include/hw/core/cpu.h | 6 +- include/qemu/thread.h | 38 +++++++- include/qemu/tsan.h | 71 ++++++++++++++ include/tcg/tcg.h | 1 + tcg/tcg.c | 9 ++ tests/Makefile.include | 9 +- tests/docker/dockerfiles/ubuntu2004.docker | 65 +++++++++++++ tests/docker/test-tsan | 44 +++++++++ tests/qtest/Makefile.include | 7 +- tests/tsan/blacklist.tsan | 10 ++ tests/tsan/suppressions.tsan | 14 +++ util/coroutine-ucontext.c | 66 +++++++++++-- util/qemu-thread-posix.c | 2 + util/qht.c | 1 + 23 files changed, 541 insertions(+), 37 deletions(-) create mode 100644 include/qemu/tsan.h create mode 100644 tests/docker/dockerfiles/ubuntu2004.docker create mode 100755 tests/docker/test-tsan create mode 100644 tests/tsan/blacklist.tsan create mode 100644 tests/tsan/suppressions.tsan
Signature · 1 line
2.17.1
PATCH v3 1/13 configure: add --enable-tsan flag + fiber annotations for coroutine-ucontext
From: Lingfeng Yang <[email protected]>
We tried running QEMU under tsan in 2016, but tsan's lack of support for longjmp-based fibers was a blocker:
https://groups.google.com/forum/#!topic/thread-sanitizer/se0YuzfWazw
Fortunately, thread sanitizer gained fiber support in early 2019:
https://reviews.llvm.org/D54889
This patch brings tsan support upstream by importing the patch that annotated QEMU's coroutines as tsan fibers in Android's QEMU fork:
https://android-review.googlesource.com/c/platform/external/qemu/+/844675
Tested with '--enable-tsan --cc=clang-9 --cxx=clang++-9 --disable-werror' configure flags.
--- configure | 47 +++++++++++++++++++++++++++- util/coroutine-ucontext.c | 66 +++++++++++++++++++++++++++++++++------ 2 files changed, 103 insertions(+), 10 deletions(-)
configure +46 -1
@@ -395,6 +395,7 @@ gprof="no" debug_tcg="no" debug="no" sanitizers="no" +tsan="no" fortify_source="" strip_opt="yes" tcg_interpreter="no"
@@ -1150,6 +1151,10 @@ for opt do ;; --disable-sanitizers) sanitizers="no" ;; + --enable-tsan) tsan="yes" + ;; + --disable-tsan) tsan="no" + ;; --enable-sparse) sparse="yes" ;; --disable-sparse) sparse="no"
@@ -1754,6 +1759,7 @@ Advanced options (experts only): --with-pkgversion=VERS use specified string as sub-version of the package --enable-debug enable common debug build options --enable-sanitizers enable default sanitizers + --enable-tsan enable thread sanitizer --disable-strip disable stripping binaries --disable-werror disable compilation abort on warning --disable-stack-protector disable compiler-provided stack protection
@@ -6196,6 +6202,30 @@ if test "$fuzzing" = "yes" ; then fi fi +# Thread sanitizer is, for now, much noisier than the other sanitizers; +# keep it separate until that is not the case. +if test "$tsan" = "yes" && test "$sanitizers" = "yes"; then + error_exit "TSAN is not supported with other sanitiziers." +fi +have_tsan=no +have_tsan_iface_fiber=no +if test "$tsan" = "yes" ; then + write_c_skeleton + if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then + have_tsan=yes + fi + cat > $TMPC << EOF +#include <sanitizer/tsan_interface.h> +int main(void) { + __tsan_create_fiber(0); + return 0; +} +EOF + if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then + have_tsan_iface_fiber=yes + fi +fi + ########################################## # check for libpmem
@@ -6297,6 +6327,16 @@ if test "$have_asan" = "yes"; then "Without code annotation, the report may be inferior." fi fi +if test "$have_tsan" = "yes" ; then + if test "$have_tsan_iface_fiber" = "yes" ; then + QEMU_CFLAGS="-fsanitize=thread $QEMU_CFLAGS" + QEMU_LDFLAGS="-fsanitize=thread $QEMU_LDFLAGS" + else + error_exit "Cannot enable TSAN due to missing fiber annotation interface." + fi +elif test "$tsan" = "yes" ; then + error_exit "Cannot enable TSAN due to missing sanitize thread interface." +fi if test "$have_ubsan" = "yes"; then QEMU_CFLAGS="-fsanitize=undefined $QEMU_CFLAGS" QEMU_LDFLAGS="-fsanitize=undefined $QEMU_LDFLAGS"
@@ -6332,7 +6372,8 @@ if test "$werror" = "yes"; then QEMU_CFLAGS="-Werror $QEMU_CFLAGS" fi -if test "$solaris" = "no" ; then +# Exclude --warn-common with TSan to suppress warnings from the TSan libraries. +if test "$solaris" = "no" && test "$tsan" = "no"; then if $ld --version 2>/dev/null | grep "GNU ld" >/dev/null 2>/dev/null ; then QEMU_LDFLAGS="-Wl,--warn-common $QEMU_LDFLAGS" fi
@@ -7386,6 +7427,10 @@ if test "$have_asan_iface_fiber" = "yes" ; then echo "CONFIG_ASAN_IFACE_FIBER=y" >> $config_host_mak fi +if test "$have_tsan" = "yes" && test "$have_tsan_iface_fiber" = "yes" ; then + echo "CONFIG_TSAN=y" >> $config_host_mak +fi + if test "$has_environ" = "yes" ; then echo "CONFIG_HAS_ENVIRON=y" >> $config_host_mak fi
util/coroutine-ucontext.c +57 -9
123 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 2/13 cpu: convert queued work to a QSIMPLEQ
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
We convert queued work to a QSIMPLEQ, instead of
open-coding it.
While at it, make sure that all accesses to the list are performed while holding the list's lock.
--- cpus-common.c | 25 ++++++++----------------- cpus.c | 14 ++++++++++++-- hw/core/cpu.c | 1 + include/hw/core/cpu.h | 6 +++--- 4 files changed, 24 insertions(+), 22 deletions(-)
cpus-common.c +8 -17
@@ -97,7 +97,7 @@ void cpu_list_remove(CPUState *cpu) }struct qemu_work_item { -struct qemu_work_item *next; +QSIMPLEQ_ENTRY (qemu_work_item) node;run_on_cpu_func func;run_on_cpu_data data;bool free, exclusive, done;
@@ -106,13 +106,7 @@ struct qemu_work_item {static void queue_work_on_cpu (CPUState *cpu,struct qemu_work_item *wi) {qemu_mutex_lock (&cpu->work_mutex); -if (cpu->queued_work_first ==NULL ) { - cpu->queued_work_first = wi; - }else { - cpu->queued_work_last->next = wi; - } - cpu->queued_work_last = wi; - wi->next =NULL ; +QSIMPLEQ_INSERT_TAIL (&cpu->work_list, wi, node); wi->done = false;qemu_mutex_unlock (&cpu->work_mutex);
@@ -306,17 +300,14 @@ void process_queued_cpu_work(CPUState *cpu) {struct qemu_work_item *wi; -if (cpu->queued_work_first ==NULL ) { +qemu_mutex_lock (&cpu->work_mutex); +if (QSIMPLEQ_EMPTY (&cpu->work_list)) { +qemu_mutex_unlock (&cpu->work_mutex);return ; } - -qemu_mutex_lock (&cpu->work_mutex); -while (cpu->queued_work_first !=NULL ) { - wi = cpu->queued_work_first; - cpu->queued_work_first = wi->next; -if (!cpu->queued_work_first) { - cpu->queued_work_last =NULL ; - } +while (!QSIMPLEQ_EMPTY (&cpu->work_list)) { + wi =QSIMPLEQ_FIRST (&cpu->work_list); +QSIMPLEQ_REMOVE_HEAD (&cpu->work_list, node);qemu_mutex_unlock (&cpu->work_mutex);if (wi->exclusive) {/* Running work items outside the BQL avoids the following deadlock:
cpus.c +12 -2
@@ -97,9 +97,19 @@ bool cpu_is_stopped(CPUState *cpu)return cpu->stopped || !runstate_is_running (); } +static inline bool cpu_work_list_empty (CPUState *cpu) +{ +bool ret; + +qemu_mutex_lock (&cpu->work_mutex); + ret =QSIMPLEQ_EMPTY (&cpu->work_list); +qemu_mutex_unlock (&cpu->work_mutex); +return ret; +} +static bool cpu_thread_is_idle (CPUState *cpu) { -if (cpu->stop || cpu->queued_work_first) { +if (cpu->stop || !cpu_work_list_empty (cpu)) {return false; }if (cpu_is_stopped (cpu)) {
@@ -1498,7 +1508,7 @@ static void *qemu_tcg_rr_cpu_thread_fn(void *arg) cpu = first_cpu; } -while (cpu && !cpu->queued_work_first && !cpu->exit_request) { +while (cpu &&cpu_work_list_empty (cpu) && !cpu->exit_request) {atomic_mb_set (&tcg_current_rr_cpu, cpu); current_cpu = cpu;
hw/core/cpu.c +1 -0
@@ -370,6 +370,7 @@ static void cpu_common_initfn(Object *obj) cpu->nr_threads =1 ;qemu_mutex_init (&cpu->work_mutex); +QSIMPLEQ_INIT (&cpu->work_list);QTAILQ_INIT (&cpu->breakpoints);QTAILQ_INIT (&cpu->watchpoints);
include/hw/core/cpu.h +3 -3
@@ -331,8 +331,8 @@ struct qemu_work_item;* @opaque: User data. * @mem_io_pc: Host Program Counter at which the memory was accessed. * @kvm_fd: vCPU file descriptor for KVM. -* @work_mutex: Lock to prevent multiple access to queued_work_*. -* @queued_work_first: First asynchronous work pending. +* @work_mutex: Lock to prevent multiple access to @work_list. +* @work_list: List of pending asynchronous work. * @trace_dstate_delayed: Delayed changes to trace_dstate (includes all changes * to @trace_dstate). * @trace_dstate: Dynamic tracing state of events for this vCPU (bitmask).
@@ -376,7 +376,7 @@ struct CPUState {sigjmp_buf jmp_env;QemuMutex work_mutex; -struct qemu_work_item *queued_work_first, *queued_work_last; + QSIMPLEQ_HEAD(, qemu_work_item) work_list;CPUAddressSpace *cpu_ases;int num_ases;
Signature · 1 line
2.17.1
PATCH v3 3/13 thread: add qemu_spin_destroy
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
It will be used for TSAN annotations.
--- include/qemu/thread.h | 3 +++ 1 file changed, 3 insertions(+)
include/qemu/thread.h +3 -0
@@ -215,6 +215,9 @@ static inline void qemu_spin_init(QemuSpin *spin)__sync_lock_release (&spin->value); } +static inline void qemu_spin_destroy (QemuSpin *spin) +{ } +static inline void qemu_spin_lock (QemuSpin *spin) {while (unlikely (__sync_lock_test_and_set (&spin->value, true))) {
Signature · 1 line
2.17.1
PATCH v3 4/13 cputlb: destroy CPUTLB with tlb_destroy
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
I was after adding qemu_spin_destroy calls, but while at it I noticed that we are leaking some memory.
--- accel/tcg/cputlb.c | 15 +++++++++++++++ exec.c | 1 + include/exec/exec-all.h | 8 ++++++++ 3 files changed, 24 insertions(+)
accel/tcg/cputlb.c +15 -0
@@ -270,6 +270,21 @@ void tlb_init(CPUState *cpu) } } +void tlb_destroy (CPUState *cpu) +{ +CPUArchState *env = cpu->env_ptr; +int i; + +qemu_spin_destroy (&env_tlb (env)->c.lock); +for (i =0 ; i <NB_MMU_MODES ; i++) { +CPUTLBDesc *desc = &env_tlb (env)->d[i]; +CPUTLBDescFast *fast = &env_tlb (env)->f[i]; + +g_free (fast->table); +g_free (desc->iotlb); + } +} +/* flush_all_helper: run fn across all cpus * * If the wait flag is set then the src cpu's helper will be queued as
exec.c +1 -0
@@ -892,6 +892,7 @@ void cpu_exec_unrealizefn(CPUState *cpu) {CPUClass *cc =CPU_GET_CLASS (cpu); +tlb_destroy (cpu);cpu_list_remove (cpu);if (cc->vmsd !=NULL ) {
include/exec/exec-all.h +8 -0
@@ -124,6 +124,11 @@ void cpu_address_space_init(CPUState *cpu, int asidx,* @cpu: CPU whose TLB should be initialized */ void tlb_init (CPUState *cpu); +/** +* tlb_destroy - destroy a CPU's TLB +* @cpu: CPU whose TLB should be destroyed +*/ +void tlb_destroy (CPUState *cpu);/** * tlb_flush_page: * @cpu: CPU whose TLB should be flushed
@@ -284,6 +289,9 @@ void tlb_set_page(CPUState *cpu, target_ulong vaddr,static inline void tlb_init (CPUState *cpu) { } +static inline void tlb_destroy (CPUState *cpu) +{ +}static inline void tlb_flush_page (CPUState *cpu,target_ulong addr) { }
Signature · 1 line
2.17.1
PATCH v3 5/13 qht: call qemu_spin_destroy for head buckets
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
--- util/qht.c | 1 + 1 file changed, 1 insertion(+)
util/qht.c +1 -0
@@ -348,6 +348,7 @@ static inline void qht_chain_destroy(const struct qht_bucket *head)struct qht_bucket *curr = head->next;struct qht_bucket *prev; +qemu_spin_destroy (&head->lock);while (curr) { prev = curr; curr = curr->next;
Signature · 1 line
2.17.1
PATCH v3 6/13 tcg: call qemu_spin_destroy for tb->jmp_lock
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
--- accel/tcg/translate-all.c | 8 ++++++++ include/tcg/tcg.h | 1 + tcg/tcg.c | 9 +++++++++ 3 files changed, 18 insertions(+)
accel/tcg/translate-all.c +8 -0
@@ -384,6 +384,11 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,return 0 ; } +void tb_destroy (TranslationBlock *tb) +{ +qemu_spin_destroy (&tb->jmp_lock); +} +bool cpu_restore_state (CPUState *cpu,uintptr_t host_pc,bool will_exit) {TranslationBlock *tb;
@@ -413,6 +418,7 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t host_pc, bool will_exit)/* one-shot translation, invalidate it immediately */ tb_phys_invalidate (tb,-1 );tcg_tb_remove (tb); +tb_destroy (tb); } r = true; }
@@ -1886,6 +1892,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu, orig_aligned -=ROUND_UP (sizeof (*tb), qemu_icache_linesize);atomic_set (&tcg_ctx->code_gen_ptr, (void *)orig_aligned); +tb_destroy (tb);return existing_tb; }tcg_tb_insert (tb);
@@ -2235,6 +2242,7 @@ void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr)tb_phys_invalidate (tb->orig_tb,-1 ); }tcg_tb_remove (tb); +tb_destroy (tb); }/* TODO: If env->pc != tb->pc (i.e. the faulting instruction was not
include/tcg/tcg.h +1 -0
@@ -819,6 +819,7 @@ void tcg_pool_reset(TCGContext *s);TranslationBlock *tcg_tb_alloc (TCGContext *s);void tcg_region_init (void ); +void tb_destroy (TranslationBlock *tb);void tcg_region_reset_all (void );size_t tcg_code_size (void );
tcg/tcg.c +9 -0
@@ -502,6 +502,14 @@ size_t tcg_nb_tbs(void)return nb_tbs; } +static gboolean tcg_region_tree_traverse (gpointer k,gpointer v,gpointer data) +{ +TranslationBlock *tb = v; + +tb_destroy (tb); +return FALSE; +} +static void tcg_region_tree_reset_all (void ) {size_t i;
@@ -510,6 +518,7 @@ static void tcg_region_tree_reset_all(void)for (i =0 ; i < region.n; i++) {struct tcg_region_tree *rt = region_trees + i * tree_size; +g_tree_foreach (rt->tree, tcg_region_tree_traverse,NULL );/* Increment the refcount first so that destroy acts as a reset */ g_tree_ref (rt->tree);g_tree_destroy (rt->tree);
Signature · 1 line
2.17.1
PATCH v3 7/13 translate-all: call qemu_spin_destroy for PageDesc
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
The radix tree is append-only, but we can fail to insert a PageDesc if the insertion races with another thread.
--- accel/tcg/translate-all.c | 9 +++++++++ 1 file changed, 9 insertions(+)
accel/tcg/translate-all.c +9 -0
@@ -547,6 +547,15 @@ static PageDesc *page_find_alloc(tb_page_addr_t index, int alloc)#endif existing =atomic_cmpxchg (lp,NULL , pd);if (unlikely (existing)) { +#ifndef CONFIG_USER_ONLY + { +int i; + +for (i =0 ; i <V_L2_SIZE ; i++) { +qemu_spin_destroy (&pd[i].lock); + } + } +#endif g_free (pd); pd = existing; }
Signature · 1 line
2.17.1
PATCH v3 8/13 thread: add tsan annotations to QemuSpin
In reply to Robert FoleyFrom: "Emilio G. Cota" <[email protected]>
--- include/qemu/thread.h | 39 ++++++++++++++++++++++++++++++++++++--- 1 file changed, 36 insertions(+), 3 deletions(-)
include/qemu/thread.h +36 -3
@@ -206,6 +206,10 @@ void qemu_thread_atexit_add(struct Notifier *notifier);*/ void qemu_thread_atexit_remove (struct Notifier *notifier); +#ifdef CONFIG_TSAN +#include <sanitizer/tsan_interface.h> +#endif +struct QemuSpin {int value; };
@@ -213,23 +217,46 @@ struct QemuSpin {static inline void qemu_spin_init (QemuSpin *spin) {__sync_lock_release (&spin->value); +#ifdef CONFIG_TSAN +__tsan_mutex_create (spin, __tsan_mutex_not_static); +#endif } -static inline void qemu_spin_destroy (QemuSpin *spin) -{ } +/* const parameter because the only purpose here is the TSAN annotation */ +static inline void qemu_spin_destroy (const QemuSpin *spin) +{ +#ifdef CONFIG_TSAN +__tsan_mutex_destroy ((void *)spin, __tsan_mutex_not_static); +#endif +}static inline void qemu_spin_lock (QemuSpin *spin) { +#ifdef CONFIG_TSAN +__tsan_mutex_pre_lock (spin,0 ); +#endif while (unlikely (__sync_lock_test_and_set (&spin->value, true))) {while (atomic_read (&spin->value)) {cpu_relax (); } } +#ifdef CONFIG_TSAN +__tsan_mutex_post_lock (spin,0 ,0 ); +#endif }static inline bool qemu_spin_trylock (QemuSpin *spin) { -return __sync_lock_test_and_set (&spin->value, true); +#ifdef CONFIG_TSAN +__tsan_mutex_pre_lock (spin, __tsan_mutex_try_lock); +#endif +bool busy =__sync_lock_test_and_set (&spin->value, true); +#ifdef CONFIG_TSAN +unsigned flags = __tsan_mutex_try_lock; + flags |= busy ? __tsan_mutex_try_lock_failed :0 ; +__tsan_mutex_post_lock (spin, flags,0 ); +#endif +return busy; }static inline bool qemu_spin_locked (QemuSpin *spin)
@@ -239,7 +266,13 @@ static inline bool qemu_spin_locked(QemuSpin *spin)static inline void qemu_spin_unlock (QemuSpin *spin) { +#ifdef CONFIG_TSAN +__tsan_mutex_pre_unlock (spin,0 ); +#endif __sync_lock_release (&spin->value); +#ifdef CONFIG_TSAN +__tsan_mutex_post_unlock (spin,0 ); +#endif }struct QemuLockCnt {
Signature · 1 line
2.17.1
PATCH v3 9/13 tests/docker: Added docker build support for TSan.
In reply to Robert FoleyAdded a new docker for ubuntu 20.04.
This docker has support for Thread Sanitizer including one patch we need in one of the header files.
https://github.com/llvm/llvm-project/commit/a72dc86cd
This command will build with tsan enabled:
make docker-test-tsan-ubuntu2004 V=1
Also added the TSAN suppresion file to disable certain cases of TSAN warnings.
--- tests/docker/dockerfiles/ubuntu2004.docker | 65 ++++++++++++++++++++++ tests/docker/test-tsan | 44 +++++++++++++++ tests/tsan/blacklist.tsan | 10 ++++ tests/tsan/suppressions.tsan | 14 +++++ 4 files changed, 133 insertions(+) create mode 100644 tests/docker/dockerfiles/ubuntu2004.docker create mode 100755 tests/docker/test-tsan create mode 100644 tests/tsan/blacklist.tsan create mode 100644 tests/tsan/suppressions.tsan
tests/docker/dockerfiles/ubuntu2004.docker new file +65 -0
@@ -0,0 +1,65 @@ +FROM ubuntu:20.04 +ENV PACKAGES flex bison \ + ccache \ + clang-10\ + gcc \ + gettext \ + git \ + glusterfs-common \ + libaio-dev \ + libattr1-dev \ + libbrlapi-dev \ + libbz2-dev \ + libcacard-dev \ + libcap-ng-dev \ + libcurl4-gnutls-dev \ + libdrm-dev \ + libepoxy-dev \ + libfdt-dev \ + libgbm-dev \ + libgtk-3-dev \ + libibverbs-dev \ + libiscsi-dev \ + libjemalloc-dev \ + libjpeg-turbo8-dev \ + liblzo2-dev \ + libncurses5-dev \ + libncursesw5-dev \ + libnfs-dev \ + libnss3-dev \ + libnuma-dev \ + libpixman-1-dev \ + librados-dev \ + librbd-dev \ + librdmacm-dev \ + libsasl2-dev \ + libsdl2-dev \ + libseccomp-dev \ + libsnappy-dev \ + libspice-protocol-dev \ + libspice-server-dev \ + libssh-dev \ + libusb-1.0-0-dev \ + libusbredirhost-dev \ + libvdeplug-dev \ + libvte-2.91-dev \ + libxen-dev \ + libzstd-dev \ + make \ + python3-yaml \ + python3-sphinx \ + sparse \ + texinfo \ + xfslibs-dev\ + vim +RUN apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get -y install $PACKAGES +RUN dpkg -l $PACKAGES | sort > /packages.txt +ENV FEATURES clang tsan pyyaml sdl2 + +# https://bugs.launchpad.net/qemu/+bug/1838763 +ENV QEMU_CONFIGURE_OPTS --disable-libssh + +# Apply patch https://reviews.llvm.org/D75820 +# This is required for TSan in clang-10 to compile with QEMU. +RUN sed -i 's/^const/static const/g' /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
tests/docker/test-tsan new file +44 -0
44 lines not on this page · raw message
tests/tsan/blacklist.tsan new file +10 -0
@@ -0,0 +1,10 @@ +# This is an example blacklist. +# To enable use of the blacklist add this to configure: +# "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" +# The eventual goal would be to fix these warnings. + +# TSan is not happy about setting/getting of dirty bits, +# for example, cpu_physical_memory_set_dirty_range, +# and cpu_physical_memory_get_dirty. +src:bitops.c +src:bitmap.c
tests/tsan/suppressions.tsan new file +14 -0
@@ -0,0 +1,14 @@ +# This is the set of runtime suppressions of TSan warnings. +# The goal would be to have here only items we do not +# plan to fix, and to explain why for each item. + +# TSan reports a double lock on RECURSIVE mutexes. +# Since the recursive lock is intentional, we choose to ignore it. +mutex:aio_context_acquire +mutex:pthread_mutex_lock + +# TSan reports a race betwen pthread_mutex_init() and +# pthread_mutex_lock(). Since this is outside of QEMU, +# we choose to ignore it. +race:pthread_mutex_init +race:pthread_mutex_lock
Signature · 1 line
2.17.1
PATCH v3 10/13 include/qemu: Added tsan.h for annotations.
In reply to Robert FoleyThese annotations will allow us to give tsan additional hints. For example, we can inform tsan about reads/writes to ignore to silence certain classes of warnings.
We can also annotate threads so that the proper thread naming shows up in tsan warning results.
--- include/qemu/tsan.h | 71 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 include/qemu/tsan.h
include/qemu/tsan.h new file +71 -0
@@ -0,0 +1,71 @@ +#ifndef QEMU_TSAN_H +#define QEMU_TSAN_H +/* +* tsan.h +* +* This file defines macros used to give ThreadSanitizer +* additional information to help suppress warnings. +* This is necessary since TSan does not provide a header file +* for these annotations. The standard way to include these +* is via the below macros. +* +* Annotation examples can be found here: +* https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan +* annotate_happens_before.cpp or ignore_race.cpp are good places to start. +* +* The full set of annotations can be found here in tsan_interface_ann.cpp. +* https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/ +* +* This work is licensed under the terms of the GNU GPL, version 2 or later. +* See the COPYING file in the top-level directory. +*/ + +#ifdef CONFIG_TSAN +/* +* Informs TSan of a happens before/after relationship. +*/ +#define QEMU_TSAN_ANNOTATE_HAPPENS_BEFORE (addr) \ + AnnotateHappensBefore(__FILE__, __LINE__, (void *)(addr)) +#define QEMU_TSAN_ANNOTATE_HAPPENS_AFTER (addr) \ + AnnotateHappensAfter(__FILE__, __LINE__, (void *)(addr)) +/* +* Gives TSan more information about thread names it can report the +* name of the thread in the warning report. +*/ +#define QEMU_TSAN_ANNOTATE_THREAD_NAME (name) \ + AnnotateThreadName(__FILE__, __LINE__, (void *)(name)) +/* +* Allows defining a region of code on which TSan will not record memory READS. +* This has the effect of disabling race detection for this section of code. +*/ +#define QEMU_TSAN_ANNOTATE_IGNORE_READS_BEGIN () \ + AnnotateIgnoreReadsBegin(__FILE__, __LINE__) +#define QEMU_TSAN_ANNOTATE_IGNORE_READS_END () \ + AnnotateIgnoreReadsEnd(__FILE__, __LINE__) +/* +* Allows defining a region of code on which TSan will not record memory +* WRITES. This has the effect of disabling race detection for this +* section of code. +*/ +#define QEMU_TSAN_ANNOTATE_IGNORE_WRITES_BEGIN () \ + AnnotateIgnoreWritesBegin(__FILE__, __LINE__) +#define QEMU_TSAN_ANNOTATE_IGNORE_WRITES_END () \ + AnnotateIgnoreWritesEnd(__FILE__, __LINE__) +#else +#define QEMU_TSAN_ANNOTATE_HAPPENS_BEFORE (addr) +#define QEMU_TSAN_ANNOTATE_HAPPENS_AFTER (addr) +#define QEMU_TSAN_ANNOTATE_THREAD_NAME (name) +#define QEMU_TSAN_ANNOTATE_IGNORE_READS_BEGIN () +#define QEMU_TSAN_ANNOTATE_IGNORE_READS_END () +#define QEMU_TSAN_ANNOTATE_IGNORE_WRITES_BEGIN () +#define QEMU_TSAN_ANNOTATE_IGNORE_WRITES_END () +#endif + +void AnnotateHappensBefore (const char *f,int l,void *addr); +void AnnotateHappensAfter (const char *f,int l,void *addr); +void AnnotateThreadName (const char *f,int l,char *name); +void AnnotateIgnoreReadsBegin (const char *f,int l); +void AnnotateIgnoreReadsEnd (const char *f,int l); +void AnnotateIgnoreWritesBegin (const char *f,int l); +void AnnotateIgnoreWritesEnd (const char *f,int l); +#endif
Signature · 1 line
2.17.1
PATCH v3 11/13 util: Added tsan annotate for thread name.
In reply to Robert FoleyThis allows us to see the name of the thread in tsan warning reports such as this:
Thread T7 'CPU 1/TCG' (tid=24317, running) created by main thread at:
--- util/qemu-thread-posix.c | 2 ++ 1 file changed, 2 insertions(+)
util/qemu-thread-posix.c +2 -0
@@ -15,6 +15,7 @@#include "qemu/atomic.h" #include "qemu/notify.h" #include "qemu-thread-common.h" +#include "qemu/tsan.h" static bool name_threads;
@@ -513,6 +514,7 @@ static void *qemu_thread_start(void *args) # endif }#endif +QEMU_TSAN_ANNOTATE_THREAD_NAME (qemu_thread_args->name);g_free (qemu_thread_args->name);g_free (qemu_thread_args);pthread_cleanup_push (qemu_thread_atexit_notify,NULL );
Signature · 1 line
2.17.1
PATCH v3 12/13 docs: Added details on TSan to testing.rst
In reply to Robert FoleyAdds TSan details to testing.rst.
This includes background and reference details on TSan, and details on how to build and test with TSan both with and without docker.
--- docs/devel/testing.rst | 107 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+)
docs/devel/testing.rst +107 -0
@@ -397,6 +397,113 @@ list is in the ``make docker`` help text. The frequently used ones are: * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test failure" section. +Thread Sanitizer +================ + +Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports +building and testing with this tool. + +For more information on TSan: + +https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual + +Thread Sanitizer in Docker +--------------------------- +TSan is currently supported in the ubuntu2004 docker. + +The test-tsan test will build using TSan and then run make check. + +.. code:: + + make docker-test-tsan@ubuntu2004 + +TSan warnings under docker are placed in files located at build/tsan/. + +We recommend using DEBUG=1 to allow launching the test from inside the docker, +and to allow review of the warnings generated by TSan. + +Building and Testing with TSan +------------------------------ + +It is possible to build and test with TSan, with a few additional steps. +These steps are normally done automatically in the docker. + +There is a one time patch needed in clang-9 or clang-10 at this time: + +.. code:: + + sed -i 's/^const/static const/g' \ + /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h + +To configure the build for TSan: + +.. code:: + + ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ + --disable-werror --extra-cflags="-O0" + +The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment +variable. + +More information on the TSAN_OPTIONS can be found here: + +https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags + +For example: + +.. code:: + + export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ + detect_deadlocks=false history_size=7 exitcode=0 \ + log_path=<build path>/tsan/tsan_warning + +The above exitcode=0 has TSan continue without error if any warnings are found. +This allows for running the test and then checking the warnings afterwards. +If you want TSan to stop and exit with error on warnings, use exitcode=66. + +TSan Suppressions +----------------- +Keep in mind that for any data race warning, although there might be a data race +detected by TSan, there might be no actual bug here. TSan provides several +different mechanisms for suppressing warnings. In general it is recommended +to fix the code if possible to eliminate the data race rather than suppress +the warning. + +A few important files for suppressing warnings are: + +tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. +The comment on each supression will typically indicate why we are +suppressing it. More information on the file format can be found here: + +https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions + +tests/tsan/blacklist.tsan - Has TSan warnings we wish to disable +at compile time for test or debug. +Add flags to configure to enable: + +"--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/blacklist.tsan" + +More information on the file format can be found here under "Blacklist Format": + +https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags + +TSan Annotations +---------------- +include/qemu/tsan.h defines annotations. See this file for more descriptions +of the annotations themselves. Annotations can be used to suppress +TSan warnings or give TSan more information so that it can detect proper +relationships between accesses of data. + +Annotation examples can be found here: + +https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ + +Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp + +The full set of annotations can be found here: + +https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp + VM testing ==========
Signature · 1 line
2.17.1
PATCH v3 13/13 tests: Disable select tests under TSan, which hit TSan issue.
In reply to Robert FoleyDisable a few tests under CONFIG_TSAN, which run into a known TSan issue that results in a hang.
https://github.com/google/sanitizers/issues/1116
The disabled tests under TSan include all the qtests as well as the test-char, test-qga, and test-qdev-global-props.
--- tests/Makefile.include | 9 +++++++-- tests/qtest/Makefile.include | 7 +++++-- 2 files changed, 12 insertions(+), 4 deletions(-)
tests/Makefile.include +7 -2
@@ -55,7 +55,6 @@ SYSEMU_TARGET_LIST := $(subst -softmmu.mak,,$(notdir \ check-unit-y += tests/check-qdict$(EXESUF) check-unit-y += tests/check-block-qdict$(EXESUF) -check-unit-$(CONFIG_SOFTMMU) += tests/test-char$(EXESUF) check-unit-y += tests/check-qnum$(EXESUF) check-unit-y += tests/check-qstring$(EXESUF) check-unit-y += tests/check-qlist$(EXESUF)
@@ -108,7 +107,6 @@ check-unit-y += tests/test-qht$(EXESUF) check-unit-y += tests/test-qht-par$(EXESUF) check-unit-y += tests/test-bitops$(EXESUF) check-unit-y += tests/test-bitcnt$(EXESUF) -check-unit-$(CONFIG_SOFTMMU) += tests/test-qdev-global-props$(EXESUF) check-unit-y += tests/check-qom-interface$(EXESUF) check-unit-y += tests/check-qom-proplist$(EXESUF) check-unit-y += tests/test-qemu-opts$(EXESUF)
@@ -123,9 +121,16 @@ check-speed-$(CONFIG_BLOCK) += tests/benchmark-crypto-cipher$(EXESUF) check-unit-$(CONFIG_BLOCK) += tests/test-crypto-secret$(EXESUF) check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_GNUTLS)) += tests/test-crypto-tlscredsx509$(EXESUF) check-unit-$(call land,$(CONFIG_BLOCK),$(CONFIG_GNUTLS)) += tests/test-crypto-tlssession$(EXESUF) +ifndef CONFIG_TSAN +# Some tests: test-char, test-qdev-global-props, and test-qga, +# are not runnable under TSan due to a known issue. +# https://github.com/google/sanitizers/issues/1116 +check-unit-$(CONFIG_SOFTMMU) += tests/test-char$(EXESUF) +check-unit-$(CONFIG_SOFTMMU) += tests/test-qdev-global-props$(EXESUF) ifneq (,$(findstring qemu-ga,$(TOOLS))) check-unit-$(call land,$(CONFIG_LINUX),$(CONFIG_VIRTIO_SERIAL)) += tests/test-qga$(EXESUF) endif +endif check-unit-$(CONFIG_SOFTMMU) += tests/test-timed-average$(EXESUF) check-unit-$(call land,$(CONFIG_SOFTMMU),$(CONFIG_INOTIFY1)) += tests/test-util-filemonitor$(EXESUF) check-unit-$(CONFIG_SOFTMMU) += tests/test-util-sockets$(EXESUF)
tests/qtest/Makefile.include +5 -2
@@ -313,12 +313,15 @@ tests/qtest/tpm-tis-device-test$(EXESUF): tests/qtest/tpm-tis-device-test.o test # QTest rules TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGET_DIRS))) +QTEST_TARGETS = +# The qtests are not runnable (yet) under TSan due to a known issue. +# https://github.com/google/sanitizers/issues/1116 +ifndef CONFIG_TSAN ifeq ($(CONFIG_POSIX),y) QTEST_TARGETS = $(TARGETS) check-qtest-y=$(foreach TARGET,$(TARGETS), $(check-qtest-$(TARGET)-y:%=tests/qtest/%$(EXESUF))) check-qtest-y += $(check-qtest-generic-y:%=tests/qtest/%$(EXESUF)) -else -QTEST_TARGETS = +endif endif qtest-obj-y = tests/qtest/libqtest.o $(test-util-obj-y)
Signature · 1 line
2.17.1
Quoted 23 lines · Robert Foley
Changes in v3:
- Fixed issue in tsan changes tostart_switch_fiber(), found by --enable-sanitizers testing.
- Removed theUC_TRACE()code.
- Removed thetb_destroycallback.v2: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg01534.html
This patch series continues the work done by Emilio Cota and others to add Thread Sanitizer (TSan) support to QEMU.
The starting point for this work was Emilio's branch here:
https://github.com/cota/qemu/commits/tsan
specifically this commit:0be125fc0afd47218b34d2019abdd19b644f3199The main purpose of this patch is to enable TSan support so that QEMU developers can start using the tool.
We found this tool useful and even ran it on our recent changes in the cpu-locks series, which fixes many warnings.
Clearly there is work to do here to clean up all the warnings. :) We have also made an effort to introduce enough of the TSan suppression mechanisms, so that others can continue this work.
Quoted 11 lines
This series adds support for:
- configure option for --enable-tsan.
-testing.rsthas the full details on how to use TSan with or without docker, including all the suppression mechanisms.
- We added an Ubuntu 20.04 docker that supports TSan builds.
- test-tsan is a new docker test that builds and runs make check under TSan.
- We added an example blacklist file for files or functions TSan should ignore at compile time. This can now be specified manually.
- Added a suppression file for TSan to suppress certain warnings at run time.
- Addedtsan.hwith annotations which also can be used to suppress warnings.
Queued to testing/next, thanks.
Signature · 1 line
Alex Bennée