Skip to contentSkip to conversation

meson: Clean up some dependencies regarding qemu-system

PATCH In review Helge Deller 6 participants 12 messages Dec 16, 2022 – Dec 22, 2022 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 meson: Clean up some dependencies regarding qemu-system

Reduce amount of compiled code and installed binaries if just the qemu utilities such as qemu-img, qemu-io and qemu-nbd have to be built. This helps to make those tools easier available on hosts where qemu-system or qemu-user isn't supported.

Reason for this patch is that qemu fail to build on debian for some seconday non-release architectures (e.g. hppa, sh4, ia64), as can be seen here:
https://buildd.debian.org/status/package.php?p=qemu&suite=sid
This patch helps to clean up the build so that these tools can be provided on those platforms in future.

docs/meson.build +1 -1
index 9136fed3b7..cfbde168b7 100644
@@ -49,7 +49,7 @@ if build_docs
         'qemu-trace-stap.1': (stap.found() ? 'man1' : ''),
         'virtfs-proxy-helper.1': (have_virtfs_proxy_helper ? 'man1' : ''),
         'virtiofsd.1': (have_virtiofsd ? 'man1' : ''),
-        'qemu.1': 'man1',
+        'qemu.1': (have_system ? 'man1' : ''),
         'qemu-block-drivers.7': 'man7',
         'qemu-cpu-models.7': 'man7'
   }
meson.build +7 -5
index 5c6b5a1c75..fe4c9c83bb 100644
@@ -3021,7 +3021,6 @@ subdir('util')
 subdir('qom')
 subdir('authz')
 subdir('crypto')
-subdir('ui')
 subdir('hw')
 subdir('gdbstub')
 
@@ -3060,9 +3059,10 @@ if have_system or have_user
                          arguments: ['@INPUT@', '@EXTRA_ARGS@', '-o', '@OUTPUT@'])
   subdir('libdecnumber')
   subdir('target')
+  subdir('ui')
+  subdir('audio')
 endif
 
-subdir('audio')
 subdir('io')
 subdir('chardev')
 subdir('fsdev')
@@ -3581,7 +3581,7 @@ subdir('qga')
 
 # Don't build qemu-keymap if xkbcommon is not explicitly enabled
 # when we don't build tools or system
-if xkbcommon.found()
+if xkbcommon.found() and have_system
   # used for the update-keymaps target, so include rules even if !have_tools
   qemu_keymap = executable('qemu-keymap', files('qemu-keymap.c', 'ui/input-keymap.c') + genh,
                            dependencies: [qemuutil, xkbcommon], install: have_tools)
@@ -3596,7 +3596,9 @@ if have_tools
                dependencies: [blockdev, qemuutil, gnutls, selinux],
                install: true)
 
-  subdir('storage-daemon')
+  if have_system
+    subdir('storage-daemon')
+  endif
   subdir('contrib/rdmacm-mux')
   subdir('contrib/elf2dmp')
 
@@ -3611,7 +3613,7 @@ if have_tools
     subdir('contrib/vhost-user-scsi')
   endif
 
-  if targetos == 'linux'
+  if targetos == 'linux' and have_system
     executable('qemu-bridge-helper', files('qemu-bridge-helper.c'),
                dependencies: [qemuutil, libcap_ng],
                install: true,
qapi/meson.build +1 -1
index fbdb442fdf..4a499db441 100644
@@ -51,12 +51,12 @@ qapi_all_modules = [
   'transaction',
   'virtio',
   'yank',
+  'qdev',
 ]
 if have_system
   qapi_all_modules += [
     'acpi',
     'audio',
-    'qdev',
     'pci',
     'rdma',
     'rocker',
tests/meson.build +3 -1
index 8e318ec513..a63a7de9f0 100644
@@ -1,5 +1,7 @@
 subdir('bench')
-subdir('qemu-iotests')
+if have_system
+  subdir('qemu-iotests')
+endif
 
 test_qapi_outputs = [
   'qapi-builtin-types.c',
tools/meson.build +1 -1
index 10eb3a043f..740d572a94 100644
@@ -5,7 +5,7 @@ have_virtiofsd = get_option('virtiofsd') \
              error_message: 'virtiofsd requires libcap-ng-devel and seccomp-devel') \
     .require(have_vhost_user,
              error_message: 'virtiofsd needs vhost-user-support') \
-    .disable_auto_if(not have_tools and not have_system) \
+    .disable_auto_if(not have_system) \
     .allowed()
 
 if have_virtiofsd
#1 lore raw
Quoted 35 lines · Helge Deller
meson.build +5 -3
@@ -3581,7 +3581,7 @@ subdir('qga')
 
  # Don't build qemu-keymap if xkbcommon is not explicitly enabled
  # when we don't build tools or system
-if xkbcommon.found()
+if xkbcommon.found() and have_system
    # used for the update-keymaps target, so include rules even if !have_tools
    qemu_keymap = executable('qemu-keymap', files('qemu-keymap.c', 'ui/input-keymap.c') + genh,
                             dependencies: [qemuutil, xkbcommon], install: have_tools)
@@ -3596,7 +3596,9 @@ if have_tools
                 dependencies: [blockdev, qemuutil, gnutls, selinux],
                 install: true)
 
-  subdir('storage-daemon')
+  if have_system
+    subdir('storage-daemon')
+  endif
    subdir('contrib/rdmacm-mux')
    subdir('contrib/elf2dmp')
 
@@ -3611,7 +3613,7 @@ if have_tools
      subdir('contrib/vhost-user-scsi')
    endif
 
-  if targetos == 'linux'
+  if targetos == 'linux' and have_system
      executable('qemu-bridge-helper', files('qemu-bridge-helper.c'),
                 dependencies: [qemuutil, libcap_ng],
                 install: true,
tools/meson.build +0 -0
index 10eb3a043f..740d572a94 100644

tools/meson.build have_virtiofsd = get_option('virtiofsd') \ · line 11

Quoted 8 lines
tools/meson.build +1 -1
@@ -5,7 +5,7 @@ have_virtiofsd = get_option('virtiofsd') \
               error_message: 'virtiofsd requires libcap-ng-devel and seccomp-devel') \
      .require(have_vhost_user,
               error_message: 'virtiofsd needs vhost-user-support') \
-    .disable_auto_if(not have_tools and not have_system) \
+    .disable_auto_if(not have_system) \
      .allowed()
 
  if have_virtiofsd

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

The rest is okay.

Paolo

#2 lore raw

Hi Paolo,

Quoted 42 lines · Paolo Bonzini
Quoted 41 lines · Helge Deller
meson.build +4 -2
@@ -3581,7 +3581,7 @@ subdir('qga')
 
   # Don't build qemu-keymap if xkbcommon is not explicitly enabled
   # when we don't build tools or system
-if xkbcommon.found()
+if xkbcommon.found() and have_system
     # used for the update-keymaps target, so include rules even if !have_tools
     qemu_keymap = executable('qemu-keymap', files('qemu-keymap.c', 'ui/input-keymap.c') + genh,
                              dependencies: [qemuutil, xkbcommon], install: have_tools)
@@ -3596,7 +3596,9 @@ if have_tools
                  dependencies: [blockdev, qemuutil, gnutls, selinux],
                  install: true)
 
-  subdir('storage-daemon')
+  if have_system
+    subdir('storage-daemon')
+  endif
     subdir('contrib/rdmacm-mux')
     subdir('contrib/elf2dmp')
 
@@ -3611,7 +3613,7 @@ if have_tools
       subdir('contrib/vhost-user-scsi')

    endif

-  if targetos == 'linux'
+  if targetos == 'linux' and have_system executable('qemu-bridge-helper', files('qemu-bridge-helper.c'),
                 dependencies: [qemuutil, libcap_ng],
                 install: true,

tools/meson.build +1 -1
index 10eb3a043f..740d572a94 100644
@@ -5,7 +5,7 @@ have_virtiofsd = get_option('virtiofsd') \
                error_message: 'virtiofsd requires libcap-ng-devel and seccomp-devel') \
       .require(have_vhost_user,
                error_message: 'virtiofsd needs vhost-user-support') \
-    .disable_auto_if(not have_tools and not have_system) \
+    .disable_auto_if(not have_system) \
       .allowed()
 
Quoted 7 lines
Quoted 2 lines

  if have_virtiofsd

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.
My patch was targetting at making "qemu-utils" possible to build and that's why I think the tools you mention should depend on "have_system".

Quoted 1 line

The rest is okay.

Ok, thanks!

Helge

#3 lore raw
Quoted 14 lines · Helge Deller

Hi Paolo,

Quoted 4 lines · Paolo Bonzini

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

Quoted 7 lines

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.

Possibly, but it's not the categorization we use upstream, which splits our binaries into three groups:
* system-emulation binaries (qemu-system-arm, etc)
* usermode-emulation binaries (qemu-arm, etc)
* tools (everything else)

(I think the guest-agent may be a fourth group.)

thanks
-- PMM

#4 lore raw
Quoted 18 lines · Peter Maydell
Quoted 17 lines · Helge Deller

Hi Paolo,

Quoted 4 lines · Paolo Bonzini

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

Quoted 8 lines
Quoted 4 lines

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.

Possibly, but it's not the categorization we use upstream, which splits our binaries into three groups:
* system-emulation binaries (qemu-system-arm, etc)

Subcategory:

* helpers (required to use system-emulation binaries), apparently provided by 'qemu-system-common' on Debian -- except virtiofsd which is not a helper --

Quoted 2 lines

* usermode-emulation binaries (qemu-arm, etc)
* tools (everything else)

apparently provided by 'qemu-utils' on Debian (without virtiofsd).

@Debian: Maybe virtiofsd deserves its own qemu-virtiofsd package?

Quoted 2 lines

(I think the guest-agent may be a fourth group.)

(or part of tools?)

#5 lore raw
Quoted 29 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Peter Maydell
Quoted 17 lines · Helge Deller

Hi Paolo,

Quoted 4 lines · Paolo Bonzini

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

Quoted 8 lines
Quoted 4 lines

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.

Possibly, but it's not the categorization we use upstream, which splits our binaries into three groups:
* system-emulation binaries (qemu-system-arm, etc)

Quoted 5 lines

Subcategory:

* helpers (required to use system-emulation binaries), apparently provided by 'qemu-system-common' on Debian -- except virtiofsd which is not a helper --

No, I don't think this is a subcategory. We provide
--enable-system
--enable-user
--enable-tools
but there is no
--enable-helper
and qemu-bridge-helper, etc are built via --enable-tools.

Quoted 6 lines
Quoted 2 lines

* usermode-emulation binaries (qemu-arm, etc)
* tools (everything else)

apparently provided by 'qemu-utils' on Debian (without virtiofsd).

Quoted 6 lines

@Debian: Maybe virtiofsd deserves its own qemu-virtiofsd package?

Quoted 2 lines

(I think the guest-agent may be a fourth group.)

(or part of tools?)

No, it looks like we build it via --enable-guest-agent, and --enable-tools is not checked.

thanks
-- PMM

#6 lore raw
Quoted 44 lines · Peter Maydell
Quoted 29 lines · Philippe Mathieu-Daudé
Quoted 18 lines · Peter Maydell
Quoted 17 lines · Helge Deller

Hi Paolo,

Quoted 4 lines · Paolo Bonzini

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

Quoted 8 lines
Quoted 4 lines

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.

Possibly, but it's not the categorization we use upstream, which splits our binaries into three groups:
* system-emulation binaries (qemu-system-arm, etc)

Quoted 5 lines

Subcategory:

* helpers (required to use system-emulation binaries), apparently provided by 'qemu-system-common' on Debian -- except virtiofsd which is not a helper --

No, I don't think this is a subcategory. We provide
--enable-system
--enable-user
--enable-tools
but there is no
--enable-helper
and qemu-bridge-helper, etc are built via --enable-tools.

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

(I think the guest-agent may be a fourth group.)
...

No, it looks like we build it via --enable-guest-agent, and --enable-tools is not checked.

Given that info, would it then make sense to keep as is:
--enable-system
--enable-user
--enable-tools -> qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd --enable-guest-agent -> guest agents (option is there already, but not used)
and additonally add:
--enable-utils -> qemu-img, qemu-io, qemu-nbd

This would leave the helpers in --enable-tools.

Helge

#7 lore raw

20.12.2022 23:56, Helge Deller wrote:
..

Quoted 7 lines

Given that info, would it then make sense to keep as is:
   --enable-system
   --enable-user
   --enable-tools -> qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd --enable-guest-agent -> guest agents  (option is there already, but not used)
and additonally add:
   --enable-utils  -> qemu-img, qemu-io, qemu-nbd

tools and utils is confusing, it smells like it's the same thing.

I understand that in theory, eg virtiofsd can be used without qemu, with some other software, but in practice there's no such software, and I don't think it will emerge in a (near) future. All the "tools" listed above are system-mode helpers really, this is why I package them in qemu-SYSTEM-common in Debian. They're not very useful for now without some of qemu-system-xx. To my view, these should be part of --enable-system, at least for now. Maybe with a separate --enable-system-helpers to turn them off if needed but I don't think it's necessary.

The new "utils" category above actually *is* useful by its own, especially qemu-nbd, for example to access .qcow2 images from the host OS. This is what eg Redhat is shipping in qemu-io package. And these are good to have without --enable-system.

/mjt

#8 lore raw
Quoted 21 lines · Michael Tokarev

20.12.2022 23:56, Helge Deller wrote:
..

Quoted 7 lines

Given that info, would it then make sense to keep as is:
    --enable-system
    --enable-user
    --enable-tools -> qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd --enable-guest-agent -> guest agents  (option is there already, but not used) and additonally add:
    --enable-utils  -> qemu-img, qemu-io, qemu-nbd

tools and utils is confusing, it smells like it's the same thing.

I understand that in theory, eg virtiofsd can be used without qemu, with some other software, but in practice there's no such software, and I don't think it will emerge in a (near) future. All the "tools" listed above are system-mode helpers really, this is why I package them in qemu-SYSTEM-common in Debian. They're not very useful for now without some of qemu-system-xx.  To my view, these should be part of --enable-system, at least for now. Maybe with a separate --enable-system-helpers to turn them off if needed but I don't think it's necessary.

Quoted 4 lines

The new "utils" category above actually *is* useful by its own, especially qemu-nbd, for example to access .qcow2 images from the host OS. This is what eg Redhat is shipping in qemu-io package.  And these are good to have without --enable-system.

Michael, please note, that the patch I initially sent in this thread is *exactly* doing what you summarize here.

Helge

#9 lore raw
In reply to Helge Deller
Quoted 53 lines · Helge Deller

Hi Paolo,

Quoted 42 lines · Paolo Bonzini
Quoted 41 lines · Helge Deller
meson.build +4 -2
@@ -3581,7 +3581,7 @@ subdir('qga')
 
   # Don't build qemu-keymap if xkbcommon is not explicitly enabled
   # when we don't build tools or system
-if xkbcommon.found()
+if xkbcommon.found() and have_system
     # used for the update-keymaps target, so include rules even if !have_tools
     qemu_keymap = executable('qemu-keymap', files('qemu-keymap.c', 'ui/input-keymap.c') + genh,
                              dependencies: [qemuutil, xkbcommon], install: have_tools)
@@ -3596,7 +3596,9 @@ if have_tools
                  dependencies: [blockdev, qemuutil, gnutls, selinux],
                  install: true)
 
-  subdir('storage-daemon')
+  if have_system
+    subdir('storage-daemon')
+  endif
     subdir('contrib/rdmacm-mux')
     subdir('contrib/elf2dmp')
 
@@ -3611,7 +3613,7 @@ if have_tools
       subdir('contrib/vhost-user-scsi')

    endif

-  if targetos == 'linux'
+  if targetos == 'linux' and have_system executable('qemu-bridge-helper', files('qemu-bridge-helper.c'),
                 dependencies: [qemuutil, libcap_ng],
                 install: true,

tools/meson.build +1 -1
index 10eb3a043f..740d572a94 100644
@@ -5,7 +5,7 @@ have_virtiofsd = get_option('virtiofsd') \
                error_message: 'virtiofsd requires libcap-ng-devel and seccomp-devel') \
       .require(have_vhost_user,
                error_message: 'virtiofsd needs vhost-user-support') \
-    .disable_auto_if(not have_tools and not have_system) \
+    .disable_auto_if(not have_system) \
       .allowed()
 
Quoted 7 lines
Quoted 2 lines

  if have_virtiofsd

These are wrong. qemu-bridge-helper, virtiofsd, qemu-storage-daemon and qemu-keymap *are* tools; if they fail to build due to any dependencies, or due to other compilation issues, you need to add tests to meson.build and check for the cause of the issues.

Quoted 7 lines

No doubt, those are *tools*.
But aren't those only required when you run system- and/or user-emulation?

Looking at the packaging of qemu in debian:

qemu-system-common debian package consists of thse *tools*:
qemu-bridge-helper, vhost-user-gpu, virtfs-proxy-helper, virtiofsd

Well technically vhost-user-gpu and virtiofsd are usable (or should be) without QEMU as they are vhost-user backends and could be used with other VMMs that support vhost-user (kvm-tool, crosvm, xen-vhost-frontend).

Not withstanding that virtiofsd is now a legacy binary as the production backend for vhost-user virtio-fs is the rust rewrite:

https://gitlab.com/virtio-fs/virtiofsd

Quoted 13 lines

qemu-utils debian package consists of the *utilities*:
qemu-img, qemu-io, qemu-nbd

IMHO this categorization makes sense.
My patch was targetting at making "qemu-utils" possible to build and that's why I think the tools you mention should depend on "have_system".

Quoted 1 line

The rest is okay.

Ok, thanks!

Helge

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#10 lore raw
In reply to Helge Deller
Quoted 22 lines · Helge Deller

Reduce amount of compiled code and installed binaries if just the qemu utilities such as qemu-img, qemu-io and qemu-nbd have to be built. This helps to make those tools easier available on hosts where qemu-system or qemu-user isn't supported.

Reason for this patch is that qemu fail to build on debian for some seconday non-release architectures (e.g. hppa, sh4, ia64), as can be seen here:
https://buildd.debian.org/status/package.php?p=qemu&suite=sid
This patch helps to clean up the build so that these tools can be provided on those platforms in future.

index fbdb442fdf..4a499db441 100644

qapi/meson.build +0 -0
@@ -51,12 +51,12 @@ qapi_all_modules = [
    'transaction',
    'virtio',
    'yank',

qapi/meson.build qapi_all_modules = [ · line 62

Quoted 10 lines
qapi/meson.build +1 -1
+  'qdev',
  ]
  if have_system
    qapi_all_modules += [
      'acpi',
      'audio',
-    'qdev',
      'pci',
      'rdma',
      'rocker',

Why do you want qdev out of sysemu?

#11 lore raw
Quoted 27 lines · Philippe Mathieu-Daudé
Quoted 26 lines · Helge Deller

Reduce amount of compiled code and installed binaries if just the qemu utilities such as qemu-img, qemu-io and qemu-nbd have to be built.  This helps to make those tools easier available on hosts where qemu-system or qemu-user isn't supported.

Reason for this patch is that qemu fail to build on debian for some seconday non-release architectures (e.g. hppa, sh4, ia64), as can be seen here:
https://buildd.debian.org/status/package.php?p=qemu&suite=sid
This patch helps to clean up the build so that these tools can be provided on those platforms in future.

index fbdb442fdf..4a499db441 100644

qapi/meson.build +1 -0
@@ -51,12 +51,12 @@ qapi_all_modules = [
     'transaction',
     'virtio',
     'yank',
+  'qdev',

  ]
  if have_system
    qapi_all_modules += [

Quoted 8 lines
Quoted 6 lines

      'acpi',
      'audio',
-    'qdev',
      'pci',
      'rdma',
      'rocker',

Why do you want qdev out of sysemu?

I just moved it some lines below sysemu.

Helge