Skip to contentSkip to conversation

gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

PATCH In review Thomas Huth 3 participants 7 messages Aug 31, 2020 – Sep 1, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 gitlab-ci.yml: Run check-qtest and check-unit at the end of the fuzzer job

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

---
 .gitlab-ci.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
.gitlab-ci.yml +3 -2
index 17f1f8fad9..417fda6909 100644
@@ -256,13 +256,14 @@ build-oss-fuzz:
     - mkdir build-oss-fuzz
     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
       ./scripts/oss-fuzz/build.sh
+    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
     - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
                       | grep -v slirp); do
         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
         echo Testing ${fuzzer} ... ;
-        ASAN_OPTIONS="fast_unwind_on_malloc=0"
-         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
       done
+    - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
   <<: *native_build_job_definition
Signature · 1 line
2.18.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 3 lines · Thomas Huth

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

Sounds good to me, though its too bad that this is limited to i386 and we aren't using the --enable-sanitizers configure argument due to the strange oss-fuzz build script.

Speaking of testing with sanitizers, does it make sense to have a job that does check-qtest with --enable-tsan, now that we have ThreadSanitizer support?

Can --enable-sanitizers --enable-tsan be added to some existing job, without severely increasing the pipeline's exeuction time?

Quoted 29 lines
---
 .gitlab-ci.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
.gitlab-ci.yml +3 -2
index 17f1f8fad9..417fda6909 100644
@@ -256,13 +256,14 @@ build-oss-fuzz:
     - mkdir build-oss-fuzz
     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
       ./scripts/oss-fuzz/build.sh
+    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
     - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f
                       | grep -v slirp); do
         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 || continue ;
         echo Testing ${fuzzer} ... ;
-        ASAN_OPTIONS="fast_unwind_on_malloc=0"
-         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
       done
+    - cd build-oss-fuzz && make check-qtest-i386 check-unit
 
 build-tci:
   <<: *native_build_job_definition
Signature · 1 line
2.18.2
#2 lore raw
Quoted 9 lines · Alexander Bulekov
Quoted 3 lines · Thomas Huth

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

Sounds good to me, though its too bad that this is limited to i386 and we aren't using the --enable-sanitizers configure argument due to the strange oss-fuzz build script.

Quoted 6 lines

Speaking of testing with sanitizers, does it make sense to have a job that does check-qtest with --enable-tsan, now that we have ThreadSanitizer support?

Can --enable-sanitizers --enable-tsan be added to some existing job, without severely increasing the pipeline's exeuction time?

Good idea, I'll have a try when I've got some spare time again (probably not this week, though, it's a pretty busy one for me).

Quoted 1 line

Thanks!

Thomas

#3 lore raw
In reply to Thomas Huth

Hi Thomas,

Quoted 18 lines · Thomas Huth

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

---
 .gitlab-ci.yml | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
.gitlab-ci.yml +1 -0
index 17f1f8fad9..417fda6909 100644
@@ -256,13 +256,14 @@ build-oss-fuzz:
     - mkdir build-oss-fuzz
     - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
       ./scripts/oss-fuzz/build.sh
+    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
     - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable -type f

.gitlab-ci.yml build-oss-fuzz: · line 268

Quoted 10 lines
.gitlab-ci.yml +2 -2
                       | grep -v slirp); do
         grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
 continue ;
         echo Testing ${fuzzer} ... ;
-        ASAN_OPTIONS="fast_unwind_on_malloc=0"
-         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
       done
+    - cd build-oss-fuzz && make check-qtest-i386 check-unit
 

As this does not use the fuzzer main entry point, what is the point of running that?

Quoted 7 lines
.gitlab-ci.yml +0 -0
 build-tci:

<<: *native_build_job_definition
--
2.18.2

#4 lore raw
Quoted 36 lines · Philippe Mathieu-Daudé

Hi Thomas,

Le lun. 31 août 2020 17:33, Thomas Huth <[email protected] <mailto:[email protected]>> a écrit :

    The fuzzer job finishes quite early, so we can run the unit tests and
    qtests with -fsanitize=address here without extending the total test
    time.

    Signed-off-by: Thomas Huth <[email protected] <mailto:[email protected]>>
    ---
     .gitlab-ci.yml | 5 +++--
     1 file changed, 3 insertions(+), 2 deletions(-)

    diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
    index 17f1f8fad9..417fda6909 100644
    --- a/.gitlab-ci.yml
    +++ b/.gitlab-ci.yml
    @@ -256,13 +256,14 @@ build-oss-fuzz:
         - mkdir build-oss-fuzz
         - CC="clang" CXX="clang++" CFLAGS="-fsanitize=address"
           ./scripts/oss-fuzz/build.sh
    +    - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
         - for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
    -type f
                           | grep -v slirp); do
             grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
    continue ;
             echo Testing ${fuzzer} ... ;
    -        ASAN_OPTIONS="fast_unwind_on_malloc=0"
    -         "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
    +        "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
           done
    +    - cd build-oss-fuzz && make check-qtest-i386 check-unit
Quoted 2 lines

As this does not use the fuzzer main entry point, what is the point of running that?

Read the friendly patch description ;-) - the idea here is that we run the qtests and unit tests with -fsanitize=address in one of the pipelines. We could also use a different task for this, but since the fuzzer tests finish quite early and we already compiled the code with -fsanitize=address here anyway, it's IMHO simply a good fit here.

Thomas

#5 lore raw
Quoted 38 lines · Thomas Huth
Quoted 36 lines · Philippe Mathieu-Daudé

Hi Thomas,

Le lun. 31 août 2020 17:33, Thomas Huth <[email protected] <mailto:[email protected]>> a écrit :

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

Signed-off-by Thomas Huth <[email protected] <mailto:[email protected]>>

---
.gitlab-ci.yml | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 17f1f8fad9..417fda6909 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -256,13 +256,14 @@ build-oss-fuzz:
- mkdir build-oss-fuzz
- CC="clang" CXX="clang++" CFLAGS="-fsanitize=address" ./scripts/oss-fuzz/build.sh
+ - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
- for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
-type f
| grep -v slirp); do
grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
continue ;
echo Testing ${fuzzer} ... ;
- ASAN_OPTIONS="fast_unwind_on_malloc=0"
- "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+ "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
done
+ - cd build-oss-fuzz && make check-qtest-i386 check-unit

Quoted 6 lines
Quoted 2 lines

As this does not use the fuzzer main entry point, what is the point of running that?

Read the friendly patch description ;-) - the idea here is that we run the qtests and unit tests with -fsanitize=address in one of the pipelines.

Sorry, the description was not that obvious to me. Can you add a comment before the 'make check*' line?

# Unrelated to fuzzer: run tests with -fsanitize=address

Quoted 7 lines

We could also use a different task for this, but since the fuzzer tests finish quite early and we already compiled the code with -fsanitize=address here anyway, it's IMHO simply a good fit here.

Thomas

#6 lore raw
Quoted 42 lines · Philippe Mathieu-Daudé
Quoted 41 lines · Thomas Huth
Quoted 38 lines · Philippe Mathieu-Daudé

Hi Thomas,

Le lun. 31 août 2020 17:33, Thomas Huth <[email protected] <mailto:[email protected]>> a écrit :

The fuzzer job finishes quite early, so we can run the unit tests and qtests with -fsanitize=address here without extending the total test time.

Signed-off-by Thomas Huth <[email protected] <mailto:[email protected]>>

---
.gitlab-ci.yml | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 17f1f8fad9..417fda6909 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -256,13 +256,14 @@ build-oss-fuzz:
- mkdir build-oss-fuzz
- CC="clang" CXX="clang++" CFLAGS="-fsanitize=address" ./scripts/oss-fuzz/build.sh
+ - export ASAN_OPTIONS="fast_unwind_on_malloc=0"
- for fuzzer in $(find ./build-oss-fuzz/DEST_DIR/ -executable
-type f
| grep -v slirp); do
grep "LLVMFuzzerTestOneInput" ${fuzzer} > /dev/null 2>&1 ||
continue ;
echo Testing ${fuzzer} ... ;
- ASAN_OPTIONS="fast_unwind_on_malloc=0"
- "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
+ "${fuzzer}" -runs=1000 -seed=1 || exit 1 ;
done
+ - cd build-oss-fuzz && make check-qtest-i386 check-unit

As this does not use the fuzzer main entry point, what is the point of running that?

Quoted 8 lines
Quoted 3 lines

Read the friendly patch description ;-) - the idea here is that we run the qtests and unit tests with -fsanitize=address in one of the pipelines.

Sorry, the description was not that obvious to me. Can you add a comment before the 'make check*' line?

# Unrelated to fuzzer: run tests with -fsanitize=address

Sure, will do.

Thomas