Skip to contentSkip to conversation

Add option to disable building tests

PATCH In review Florian Schmidt 9 participants 12 messages Jul 24 – Jul 27 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Add option to disable building tests

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64
real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
 meson.build                   | 2 +-
 meson_options.txt             | 2 ++
 scripts/meson-buildoptions.sh | 3 +++
 3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
 subdir('pyvenv')
 # Tests are disabled on emscripten because they rely on host features that aren't
 # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
   subdir('tests')
 endif
 if gtk.found()
meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
 # on the configure script command line.  After adding an option
 # here make sure to run "make update-buildoptions".
 
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
 option('docs', type : 'feature', value : 'auto',
        description: 'Documentations build support')
 option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +3 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {
   printf "%s\n" '  spice-protocol  Spice protocol support'
   printf "%s\n" '  stack-protector compiler-provided stack protection'
   printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
   printf "%s\n" '  tools           build support utilities that come with QEMU'
   printf "%s\n" '  tpm             TPM support'
   printf "%s\n" '  u2f             U2F emulation support'
@@ -515,6 +516,8 @@ _meson_option_parse() {
     --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
     --disable-tcg-interpreter) printf "%s" -Dtcg_interpreter=false ;;
     --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
     --enable-tools) printf "%s" -Dtools=enabled ;;
     --disable-tools) printf "%s" -Dtools=disabled ;;
     --enable-tpm) printf "%s" -Dtpm=enabled ;;
Signature · 1 line
2.47.3
Reviewed-by Tested-by on v1 1/1
#1 lore raw
Quoted 72 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64
real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
 meson.build                   | 2 +-
 meson_options.txt             | 2 ++
 scripts/meson-buildoptions.sh | 3 +++
 3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
 subdir('pyvenv')
 # Tests are disabled on emscripten because they rely on host features that aren't
 # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
   subdir('tests')
 endif
 if gtk.found()
meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
 # on the configure script command line.  After adding an option
 # here make sure to run "make update-buildoptions".
 
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
 option('docs', type : 'feature', value : 'auto',
        description: 'Documentations build support')
 option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +1 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {
   printf "%s\n" '  spice-protocol  Spice protocol support'
   printf "%s\n" '  stack-protector compiler-provided stack protection'
   printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
   printf "%s\n" '  tools           build support utilities that come with QEMU'
   printf "%s\n" '  tpm             TPM support'
   printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 523

Quoted 8 lines
scripts/meson-buildoptions.sh +2 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
     --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
     --disable-tcg-interpreter) printf "%s" -Dtcg_interpreter=false ;;
     --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
     --enable-tools) printf "%s" -Dtools=enabled ;;
     --disable-tools) printf "%s" -Dtools=disabled ;;
     --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably too error prone and much less simple than this patch. So I don't think it's a good idea.

Regards,
Pierrick

#2 lore raw
Quoted 87 lines · Pierrick Bouvier
Quoted 72 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64
real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
  meson.build                   | 2 +-
  meson_options.txt             | 2 ++
  scripts/meson-buildoptions.sh | 3 +++
  3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
  subdir('pyvenv')
  # Tests are disabled on emscripten because they rely on host features that aren't
  # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
    subdir('tests')
  endif
  if gtk.found()
meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
  # on the configure script command line.  After adding an option
  # here make sure to run "make update-buildoptions".
 
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
  option('docs', type : 'feature', value : 'auto',
         description: 'Documentations build support')
  option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +1 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {
    printf "%s\n" '  spice-protocol  Spice protocol support'
    printf "%s\n" '  stack-protector compiler-provided stack protection'
    printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
    printf "%s\n" '  tools           build support utilities that come with QEMU'
    printf "%s\n" '  tpm             TPM support'
    printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 523

Quoted 8 lines
scripts/meson-buildoptions.sh +2 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
      --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
      --disable-tcg-interpreter) printf "%s" -Dtcg_interpreter=false ;;
      --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
      --enable-tools) printf "%s" -Dtools=enabled ;;
      --disable-tools) printf "%s" -Dtools=disabled ;;
      --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

Quoted 8 lines

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably too error prone and much less simple than this patch. So I don't think it's a good idea.

Regards,
Pierrick

When is it useful to build without the provided test suite? When another one is better? Why not contribute it then?

Or is is just that you don't want to test your changes?

The problem might be we build the tests by default but don't run them by default.

#3 lore raw
Quoted 4 lines · Philippe Mathieu-Daudé

When is it useful to build without the provided test suite? When another one is better? Why not contribute it then?

Or is is just that you don't want to test your changes?

In our case, we have pipelines that rebuild/repackage QEMU after a full build+test, without running the test suite again. In those cases, building the tests is wasted effort, and considerably increases the overall build times.

Quoted 2 lines

The problem might be we build the tests by default but don't run them by default.

That's fair, but I would argue that in that case, there should still be some way to build the binaries without building and running the tests.

Cheers,
Florian

#4 lore raw
Quoted 10 lines · Florian Schmidt
Quoted 4 lines · Philippe Mathieu-Daudé

When is it useful to build without the provided test suite? When another one is better? Why not contribute it then?

Or is is just that you don't want to test your changes?

In our case, we have pipelines that rebuild/repackage QEMU after a full build+test, without running the test suite again. In those cases, building the tests is wasted effort, and considerably increases the overall build times.

I would suggest that building a second time is wasted effort too. If the 2nd build is byte-for-byte identical to the 1st build, then it is pointless. If it is not byte-for-byte identical, then you'll be taking binaries that have not been tested.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
#5 lore raw
Quoted 4 lines · Daniel P. Berrangé

I would suggest that building a second time is wasted effort too. If the 2nd build is byte-for-byte identical to the 1st build, then it is pointless. If it is not byte-for-byte identical, then you'll be taking binaries that have not been tested.

You're absolutely right. They are bit-to-bit-identical, and it is wasted effort, too. That is something we'll look at internally.

However, it felt like a genuinely useful feature to split build and test phase more strongly, by having the ability to only build the product binaries if desired.

There are other options, of course. A new build target such as "make bin" or similar could only build the product binaries, and a following "make test" would then build + run the tests. This could be combined with Philippe's idea to run (and not only build) the tests by default, though I'm always wary of changing the behaviour of default targets. That's obviously a maintainers' decision.

Cheers,
Florian

#6 lore raw
In reply to Philippe Mathieu-Daudé
Quoted 92 lines · Philippe Mathieu-Daudé
Quoted 91 lines · Pierrick Bouvier
Quoted 71 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64- softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64

real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
  meson.build                   | 2 +-
  meson_options.txt             | 2 ++
  scripts/meson-buildoptions.sh | 3 +++

  3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
   subdir('pyvenv')
   # Tests are disabled on emscripten because they rely on host features that aren't
   # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
     subdir('tests')

  endif
  if gtk.found()

meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
   # on the configure script command line.  After adding an option
   # here make sure to run "make update-buildoptions".
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
   option('docs', type : 'feature', value : 'auto',
          description: 'Documentations build support')
   option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +0 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {

    printf "%s\n" '  spice-protocol  Spice protocol support'
    printf "%s\n" '  stack-protector compiler-provided stack protection' printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
    printf "%s\n" '  tools           build support utilities that come with QEMU' printf "%s\n" '  tpm             TPM support'
    printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 523

Quoted 8 lines
scripts/meson-buildoptions.sh +2 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
       --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
       --disable-tcg-interpreter) printf "%s" -Dtcg_interpreter=false ;;
       --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
       --enable-tools) printf "%s" -Dtools=enabled ;;
       --disable-tools) printf "%s" -Dtools=disabled ;;
       --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably too error prone and much less simple than this patch. So I don't think it's a good idea.

Quoted 5 lines
Quoted 3 lines

Regards,
Pierrick

When is it useful to build without the provided test suite?

I sometimes wished indeed for a --disable-tests switch in the past already (was just too lazy to contribute a patch): I'm sometimes building QEMU in a separate directory, just for a very special case like testing a patch with an --enable-asan build, or for doing a "git bisect". In such cases, I never want to run the normal tests since it's simply not necessary. I run the tests from my main build directory instead once I have a final patch and want to contribute it to upstream. So IMHO this patch is a good idea.

Thomas

#7 lore raw
Quoted 94 lines · Thomas Huth
Quoted 93 lines · Philippe Mathieu-Daudé
Quoted 92 lines · Pierrick Bouvier
Quoted 72 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' 
--target-list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto-objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target-list=x86_64- softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64

real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
  meson.build                   | 2 +-
  meson_options.txt             | 2 ++
  scripts/meson-buildoptions.sh | 3 +++

  3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
   subdir('pyvenv')
   # Tests are disabled on emscripten because they rely on host features that aren't
   # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
     subdir('tests')

  endif
  if gtk.found()

meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
   # on the configure script command line.  After adding an option
   # here make sure to run "make update-buildoptions".
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
   option('docs', type : 'feature', value : 'auto',
          description: 'Documentations build support')
   option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +0 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {

    printf "%s\n" '  spice-protocol  Spice protocol support'
    printf "%s\n" '  stack-protector compiler-provided stack protection' printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
    printf "%s\n" '  tools           build support utilities that come with QEMU' printf "%s\n" '  tpm             TPM support'
    printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 523

Quoted 8 lines
scripts/meson-buildoptions.sh +2 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
       --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
       --disable-tcg-interpreter) printf "%s" -Dtcg_interpreter=false ;;
       --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
       --enable-tools) printf "%s" -Dtools=enabled ;;
       --disable-tools) printf "%s" -Dtools=disabled ;;
       --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably too error prone and much less simple than this patch. So I don't think it's a good idea.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Regards,
Pierrick

When is it useful to build without the provided test suite?

I sometimes wished indeed for a --disable-tests switch in the past already (was just too lazy to contribute a patch): I'm sometimes building QEMU in a separate directory, just for a very special case like testing a patch with an --enable-asan build, or for doing a "git bisect". In such cases, I never want
to run the normal tests since it's simply not necessary. I run the tests from my main build directory instead once I have a final patch and want to contribute it to upstream. So IMHO this patch is a good idea.

I similarly have a local patch for this for such usage, that I never cleaned up to submit but I think it's a good idea.

Regards,
BALATON Zoltan

#8 lore raw
Quoted 94 lines · BALATON Zoltan
Quoted 93 lines · Thomas Huth
Quoted 92 lines · Philippe Mathieu-Daudé
Quoted 91 lines · Pierrick Bouvier
Quoted 71 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto- objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target- list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto- objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target- list=x86_64- softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64

real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
  meson.build                   | 2 +-
  meson_options.txt             | 2 ++
  scripts/meson-buildoptions.sh | 3 +++

  3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +1 -1
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
   subdir('pyvenv')
   # Tests are disabled on emscripten because they rely on host features that aren't
   # supported by emscripten (e.g. fork and unix socket).
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'
     subdir('tests')

  endif
  if gtk.found()

meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
   # on the configure script command line.  After adding an option
   # here make sure to run "make update-buildoptions".
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
   option('docs', type : 'feature', value : 'auto',
          description: 'Documentations build support')
   option('fuzzing', type : 'boolean', value: false,
scripts/meson-buildoptions.sh +0 -0
index c003985047..3fec13a336 100644
@@ -194,6 +194,7 @@ meson_options_help() {

    printf "%s\n" '  spice-protocol  Spice protocol support'
    printf "%s\n" '  stack-protector compiler-provided stack protection' printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite'
    printf "%s\n" '  tools           build support utilities that come with QEMU' printf "%s\n" '  tpm             TPM support'
    printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh _meson_option_parse() { · line 523

Quoted 8 lines
scripts/meson-buildoptions.sh +2 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
       --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
       --disable-tcg-interpreter) printf "%s" - Dtcg_interpreter=false ;;
       --tls-priority=*) quote_sh "-Dtls_priority=$2" ;;
+    --enable-tests) printf "%s" -Dtests=enabled ;;
+    --disable-tests) printf "%s" -Dtests=disabled ;;
       --enable-tools) printf "%s" -Dtools=enabled ;;
       --disable-tools) printf "%s" -Dtools=disabled ;;
       --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably too error prone and much less simple than this patch. So I don't think it's a good idea.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Regards,
Pierrick

When is it useful to build without the provided test suite?

I sometimes wished indeed for a --disable-tests switch in the past already (was just too lazy to contribute a patch): I'm sometimes building QEMU in a separate directory, just for a very special case like testing a patch with an --enable-asan build, or for doing a "git bisect". In such cases, I never want to run the normal tests since it's simply not necessary. I run the tests from my main build directory instead once I have a final patch and want to contribute it to upstream. So IMHO this patch is a good idea.

I similarly have a local patch for this for such usage, that I never cleaned up to submit but I think it's a good idea.

I'm not saying this patch is a bad idea, I'll likely use it too; I want to clarify what are the valid use cases the community sees here. Building pointless things clearly has a negative impact on resources and our time, but not testing changes also has. IOW in somes cases this change is acceptable, but I'm a little wary on massive use by default.

Anyway I didn't wanted to start yet another endless discussion so I won't intervene further in this thread, sorry for the usual noise.

#9 lore raw
Quoted 120 lines · Philippe Mathieu-Daudé
Quoted 119 lines · BALATON Zoltan
Quoted 117 lines · Thomas Huth
Quoted 103 lines · Philippe Mathieu-Daudé
Quoted 102 lines · Pierrick Bouvier
Quoted 81 lines · Florian Schmidt

There are situations in which you might want to build QEMU without building the full test suite, which in some configurations can take a considerable amount of time to build. This is especially true when using
LTO with clang. For example:

$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto- 
objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target- 
list=x86_64-softmmu
[...]
$ time make -j8
[...]
[3078/3078] Linking target tests/qtest/qos-test
real    6m43.250s
user    101m15.967s
sys     4m3.813s
$ ../configure --cc=clang '--extra-ldflags=-flto=thin -ffat-lto- 
objects' '--extra-cflags=-flto=thin -ffat-lto-objects' --target- 
list=x86_64- softmmu --disable-tests
[...]
$ time make -j8
[...]
[2024/2024] Linking target qemu-system-x86_64

real    3m14.277s
user    33m13.174s
sys     1m36.642s

Add a toggle to optionally disable building tests.

---
  meson.build                   | 2 +-
  meson_options.txt             | 2 ++
  scripts/meson-buildoptions.sh | 3 +++

  3 files changed, 6 insertions(+), 1 deletion(-)
meson.build +0 -0
index 49a5baf5b5..ad728c26ec 100644
@@ -4605,7 +4605,7 @@ subdir('docs')
   subdir('pyvenv')
   # Tests are disabled on emscripten because they rely on host 

features that aren't

  # supported by emscripten (e.g. fork and unix socket).
meson.build +1 -1
-if host_os != 'emscripten'
+if get_option('tests').allowed() and host_os != 'emscripten'

    subdir('tests')
  endif
  if gtk.found()

meson_options.txt +2 -0
index a07cb47d35..68333ca402 100644
@@ -43,6 +43,8 @@ option('gdb', type: 'string', value: '',
   # on the configure script command line.  After adding an option
   # here make sure to run "make update-buildoptions".
+option('tests', type: 'feature', value: 'auto',
+       description: 'Build the test suite')
   option('docs', type : 'feature', value : 'auto',
          description: 'Documentations build support')
   option('fuzzing', type : 'boolean', value: false,
scripts/meson- renamed from scripts/meson-buildoptions.sh +0 -0

buildoptions.sh
index c003985047..3fec13a336 100644

scripts/meson-buildoptions.sh +0 -0
@@ -194,6 +194,7 @@ meson_options_help() {

    printf "%s\n" '  spice-protocol  Spice protocol support' printf "%s\n" '  stack-protector compiler-provided stack protection'
    printf "%s\n" '  tcg             TCG support'
+  printf "%s\n" '  tests           build test suite' printf "%s\n" '  tools           build support utilities that come with QEMU'
    printf "%s\n" '  tpm             TPM support' printf "%s\n" '  u2f             U2F emulation support'

scripts/meson-buildoptions.sh +0 -0
@@ -515,6 +516,8 @@ _meson_option_parse() {
       --enable-tcg-interpreter) printf "%s" -Dtcg_interpreter=true ;;
Quoted 8 lines

      --disable-tcg-interpreter) printf "%s" - Dtcg_interpreter=false ;;
      --tls-priority=*) quote_sh "-Dtls_priority=$2" ;; +    --enable-tests) printf "%s" -Dtests=enabled ;; +    --disable-tests) printf "%s" -Dtests=disabled ;; --enable-tools) printf "%s" -Dtools=enabled ;; --disable-tools) printf "%s" -Dtools=disabled ;; --enable-tpm) printf "%s" -Dtpm=enabled ;;

That's a great addition, including just for speeding up normal builds.

If we want to overengineer the thing, it could be possible to still declare all tests, but not build them by default. However, it's probably
too error prone and much less simple than this patch. So I don't think it's a good idea.

Quoted 5 lines
Quoted 3 lines

Regards,
Pierrick

When is it useful to build without the provided test suite?

I sometimes wished indeed for a --disable-tests switch in the past already (was just too lazy to contribute a patch): I'm sometimes building QEMU in a separate directory, just for a very special case like testing a patch with an --enable-asan build, or for doing a "git bisect". In such cases, I never want to run the normal tests since it's simply not necessary. I run the tests from my main build directory instead once I have a final patch and want to contribute it to upstream. So IMHO this patch is a good idea.

Quoted 7 lines
Quoted 2 lines

I similarly have a local patch for this for such usage, that I never cleaned up to submit but I think it's a good idea.

I'm not saying this patch is a bad idea, I'll likely use it too; I want to clarify what are the valid use cases the community sees here. Building pointless things clearly has a negative impact on resources and our time, but not testing changes also has.

Perhaps we could add something to checkpatch that emits an error if config-status has --disable-tests.

Quoted 5 lines

IOW in somes cases this change is acceptable, but I'm a little wary on massive use by default.

Anyway I didn't wanted to start yet another endless discussion so I won't intervene further in this thread, sorry for the usual noise.

I think we'd rather have your comments and everyone else's, there's no deadline for anything.

#10 lore raw
Quoted 9 lines · Fabiano Rosas
Quoted 4 lines · Philippe Mathieu-Daudé

I'm not saying this patch is a bad idea, I'll likely use it too; I want to clarify what are the valid use cases the community sees here. Building pointless things clearly has a negative impact on resources and our time, but not testing changes also has.

Perhaps we could add something to checkpatch that emits an error if config-status has --disable-tests.

I think that's probably overkill. People who don't run tests are much more likely to be doing it by simply not running "make check" and "make check-functional" than by using this new --disable-tests configure option :-)

I think if --disable-tests isn't the default then that's enough to mean that it only gets used for the situations where it matters to cut down that build time (like the bisection-test cases etc).

(Also, checkpatch runs in a source tree and can't know about configure options, which are per-build-tree.)

thanks
-- PMM

#11 lore raw
Quoted 11 lines · Peter Maydell
Quoted 9 lines · Fabiano Rosas
Quoted 4 lines · Philippe Mathieu-Daudé

I'm not saying this patch is a bad idea, I'll likely use it too; I want to clarify what are the valid use cases the community sees here. Building pointless things clearly has a negative impact on resources and our time, but not testing changes also has.

Perhaps we could add something to checkpatch that emits an error if config-status has --disable-tests.

Quoted 8 lines

I think that's probably overkill. People who don't run tests are much more likely to be doing it by simply not running "make check" and "make check-functional" than by using this new --disable-tests configure option :-)

I think if --disable-tests isn't the default then that's enough to mean that it only gets used for the situations where it matters to cut down that build time (like the bisection-test cases etc).

+1 from me:
Add "--disable-tests", but don't make it default.

Those tests just stole lots of my time while running git-bisect.....

Helge