DTB build housekeeping
cover DTB build housekeeping
This series removes the bundled device tree blobs from the repository which are redundant to the bundled device tree sources. To make this work, the device tree compiler 'dtc' is now required for the boards which rely on these device trees. Building these board can be enforced with a 'dtc' build option which should help packagers. For simplicity, 'dtc' is searched for in the system only rather than also considering the 'dtc' submodule.
I also tried unsing Meson's source_set::add(when: ...) for building only the needed DTBs but eventually found it to be too over-engineered.
Last but not least two inconsistencies regarding QEMU_FILE_TYPE_DTB are fixed which seem to be latent bugs (one of which materializes in my e500-fdt branch).
Bernhard Beschow (5):
hw: Fix type constant for DTB files pc-bios/dtb/meson: Prefer target name to be outfile, not infile scripts/meson-buildoptions: Sort coroutine_backend choices
lexicographically
configure: Add dtc option
configure: Require dtc for boards with bundled device trees
MAINTAINERS | 2 +- meson.build | 12 ++++++++++++ hw/arm/boot.c | 2 +- hw/ppc/e500.c | 2 +- Kconfig.host | 3 +++ hw/microblaze/Kconfig | 4 ++-- hw/ppc/Kconfig | 4 ++-- meson_options.txt | 3 +++ pc-bios/dtb/bamboo.dtb | Bin 3211 -> 0 bytes pc-bios/dtb/canyonlands.dtb | Bin 9779 -> 0 bytes pc-bios/dtb/meson.build | 19 ++++++++----------- pc-bios/dtb/petalogix-ml605.dtb | Bin 9882 -> 0 bytes pc-bios/dtb/petalogix-s3adsp1800.dtb | Bin 8161 -> 0 bytes scripts/meson-buildoptions.sh | 5 ++++- 14 files changed, 37 insertions(+), 19 deletions(-) delete mode 100644 pc-bios/dtb/bamboo.dtb delete mode 100644 pc-bios/dtb/canyonlands.dtb delete mode 100644 pc-bios/dtb/petalogix-ml605.dtb delete mode 100644 pc-bios/dtb/petalogix-s3adsp1800.dtb
Signature · 1 line
2.49.0
PATCH 1/5 hw: Fix type constant for DTB files
Commit fcb1ad456c58 ("system/datadir: Add new type constant for DTB files") introduced a new type constant for DTB files and converted the boards with bundled device trees to use it. Convert the other boards for consistency.
--- hw/arm/boot.c | 2 +- hw/ppc/e500.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
hw/arm/boot.c +1 -1
@@ -527,7 +527,7 @@ int arm_load_dtb(hwaddr addr, const struct arm_boot_info *binfo,if (binfo->dtb_filename) {char *filename; - filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , binfo->dtb_filename); + filename =qemu_find_file (QEMU_FILE_TYPE_DTB , binfo->dtb_filename);if (!filename) {fprintf (stderr,"Couldn't open dtb file %s\n" , binfo->dtb_filename); goto fail;
hw/ppc/e500.c +1 -1
@@ -411,7 +411,7 @@ static int ppce500_load_device_tree(PPCE500MachineState *pms,if (dtb_file) {char *filename; - filename =qemu_find_file (QEMU_FILE_TYPE_BIOS , dtb_file); + filename =qemu_find_file (QEMU_FILE_TYPE_DTB , dtb_file);if (!filename) { goto out; }
Signature · 1 line
2.49.0
Re: [PATCH 1/5] hw: Fix type constant for DTB files
Quoted 10 lines · Bernhard Beschow
Commit
fcb1ad456c58("system/datadir: Add new type constant for DTB files") introduced a new type constant for DTB files and converted the boards with bundled device trees to use it. Convert the other boards for consistency.Fixes fcb1ad456c58 ("system/datadir: Add new type constant for DTB files")Signed-off-by Bernhard Beschow <[email protected]>--- hw/arm/boot.c | 2 +- hw/ppc/e500.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
PATCH 2/5 pc-bios/dtb/meson: Prefer target name to be outfile, not infile
In reply to Bernhard BeschowMakes this custom_target() usage consistent with other ones in QEMU.
--- pc-bios/dtb/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
pc-bios/dtb/meson.build +1 -1
@@ -9,7 +9,7 @@ dtc = find_program('dtc', required: false) if dtc.found() foreach out : dtbs f = fs.replace_suffix(out, '.dts') - custom_target(f, + custom_target(out, build_by_default: have_system, input: files(f), output: out,
Signature · 1 line
2.49.0
Re: [PATCH 2/5] pc-bios/dtb/meson: Prefer target name to be outfile, not infile
Quoted 13 lines · Bernhard Beschow
Makes this
custom_target()usage consistent with other ones in QEMU.Fixes 6e0dc9d2a88a ("meson: compile bundled device trees")Signed-off-by Bernhard Beschow <[email protected]>--- pc-bios/dtb/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)pc-bios/dtb/meson.build +0
-0
pc-bios/dtb/meson.build dtc = find_program('dtc', required: false) · line 15
Quoted 8 lines
pc-bios/dtb/meson.build +1
-1@@ -9,7 +9,7 @@ dtc = find_program('dtc', required: false) if dtc.found() foreach out : dtbs f = fs.replace_suffix(out, '.dts') - custom_target(f, + custom_target(out, build_by_default: have_system, input: files(f), output: out,
That looks better, indeed.
PATCH 3/5 scripts/meson-buildoptions: Sort coroutine_backend choices lexicographically
In reply to Bernhard BeschowWhen changing meson_options.txt, this script gets updated automatically by QEMU tooling which sorts the choices lexicographically. Fix this in preparation of the ext patch.
--- scripts/meson-buildoptions.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/meson-buildoptions.sh +1 -1
@@ -80,7 +80,7 @@ meson_options_help() { printf "%s\n" ' --tls-priority=VALUE Default TLS protocol/cipher priority string' printf "%s\n" ' [NORMAL]' printf "%s\n" ' --with-coroutine=CHOICE coroutine backend to use (choices:' - printf "%s\n" ' auto/sigaltstack/ucontext/windows/wasm)' + printf "%s\n" ' auto/sigaltstack/ucontext/wasm/windows)' printf "%s\n" ' --with-pkgversion=VALUE use specified string as sub-version of the' printf "%s\n" ' package' printf "%s\n" ' --with-suffix=VALUE Suffix for QEMU data/modules/config directories'
Signature · 1 line
2.49.0
Re: [PATCH 3/5] scripts/meson-buildoptions: Sort coroutine_backend choices lexicographically
Quoted 3 lines · Bernhard Beschow
When changing meson_options.txt, this script gets updated automatically by QEMU tooling which sorts the choices lexicographically. Fix this in preparation of the ext patch.
Typo: next patch
Quoted 20 lines
Fixes ccc403ed5844 ("meson: Add wasm build in build scripts")Signed-off-by Bernhard Beschow <[email protected]>---
scripts/meson-buildoptions.sh| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)scripts/meson-buildoptions.sh +1
-1@@ -80,7 +80,7 @@ meson_options_help() { printf "%s\n" ' --tls-priority=VALUE Default TLS protocol/cipher priority string' printf "%s\n" ' [NORMAL]' printf "%s\n" ' --with-coroutine=CHOICE coroutine backend to use (choices:' - printf "%s\n" ' auto/sigaltstack/ucontext/windows/wasm)' + printf "%s\n" ' auto/sigaltstack/ucontext/wasm/windows)' printf "%s\n" ' --with-pkgversion=VALUE use specified string as sub-version of the' printf "%s\n" ' package' printf "%s\n" ' --with-suffix=VALUE Suffix for QEMU data/modules/config directories'
Re: [PATCH 3/5] scripts/meson-buildoptions: Sort coroutine_backend choices lexicographically
In reply to Bernhard BeschowQuoted 3 lines · Bernhard Beschow
When changing meson_options.txt, this script gets updated automatically by QEMU tooling which sorts the choices lexicographically. Fix this in preparation of the ext patch.
s/ext/next/
Quoted 11 lines
Fixes ccc403ed5844 ("meson: Add wasm build in build scripts")Signed-off-by Bernhard Beschow <[email protected]>--- scripts/meson-buildoptions.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/meson-buildoptions.sh +0
-0
scripts/meson-buildoptions.sh meson_options_help() { · line 86
Quoted 8 lines
scripts/meson-buildoptions.sh +1
-1@@ -80,7 +80,7 @@ meson_options_help() { printf "%s\n" ' --tls-priority=VALUE Default TLS protocol/cipher priority string' printf "%s\n" ' [NORMAL]' printf "%s\n" ' --with-coroutine=CHOICE coroutine backend to use (choices:' - printf "%s\n" ' auto/sigaltstack/ucontext/windows/wasm)' + printf "%s\n" ' auto/sigaltstack/ucontext/wasm/windows)' printf "%s\n" ' --with-pkgversion=VALUE use specified string as sub-version of the' printf "%s\n" ' package' printf "%s\n" ' --with-suffix=VALUE Suffix for QEMU data/modules/config directories'
Kohei Tokunaga
Reviewed-byRe: [PATCH 3/5] scripts/meson-buildoptions: Sort coroutine_backend choices lexicographically
I should've made sure to run the tool, sorry. Thanks for spotting and fixing this issue.
PATCH 4/5 configure: Add dtc option
In reply to Bernhard BeschowThe next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
--- meson.build | 12 ++++++++++++ Kconfig.host | 3 +++ meson_options.txt | 3 +++ pc-bios/dtb/meson.build | 1 - scripts/meson-buildoptions.sh | 3 +++ 5 files changed, 21 insertions(+), 1 deletion(-)
meson.build +12 -0
@@ -2119,6 +2119,15 @@ if fdt_opt == 'internal' fdt = libfdt_proj.get_variable('libfdt_dep') endif +dtc = not_found +dtc_opt = get_option('dtc') +if dtc_opt == 'enabled' or (dtc_opt == 'auto' and have_system) + dtc = find_program('dtc', required: dtc_opt == 'enabled') + if not dtc.found() and dtc_opt == 'enabled' + error('dtc not found') + endif +endif + rdma = not_found if not get_option('rdma').auto() or have_system rdma_libs = [cc.find_library('rdmacm', has_headers: ['rdma/rdma_cma.h'],
@@ -2477,6 +2486,7 @@ config_host_data.set('CONFIG_FREEBSD', host_os == 'freebsd') config_host_data.set('CONFIG_CAPSTONE', capstone.found()) config_host_data.set('CONFIG_COCOA', cocoa.found()) config_host_data.set('CONFIG_DARWIN', host_os == 'darwin') +config_host_data.set('CONFIG_DTC', dtc.found()) config_host_data.set('CONFIG_FDT', fdt.found()) config_host_data.set('CONFIG_FUZZ', get_option('fuzzing')) config_host_data.set('CONFIG_GCOV', get_option('b_coverage'))
@@ -3314,6 +3324,7 @@ host_kconfig = \ (libcbor.found() ? ['CONFIG_LIBCBOR=y'] : []) + \ (gnutls.found() ? ['CONFIG_GNUTLS=y'] : []) + \ (x11.found() ? ['CONFIG_X11=y'] : []) + \ + (dtc.found() ? ['CONFIG_DTC=y'] : []) + \ (fdt.found() ? ['CONFIG_FDT=y'] : []) + \ (have_vhost_user ? ['CONFIG_VHOST_USER=y'] : []) + \ (have_vhost_vdpa ? ['CONFIG_VHOST_VDPA=y'] : []) + \
@@ -4703,6 +4714,7 @@ summary_info += {'sphinx-build': sphinx_build} if config_host.has_key('GDB') summary_info += {'gdb': config_host['GDB']} endif +summary_info += {'dtc': dtc} summary_info += {'iasl': iasl} summary_info += {'genisoimage': config_host['GENISOIMAGE']} if host_os == 'windows' and have_ga
Kconfig.host +3 -0
@@ -29,6 +29,9 @@ config IVSHMEM config TPM bool +config DTC + bool + config FDT bool
meson_options.txt +3 -0
@@ -316,6 +316,9 @@ option('vduse_blk_export', type: 'feature', value: 'auto', option('capstone', type: 'feature', value: 'auto', description: 'Whether and how to find the capstone library') +option('dtc', type: 'combo', value: 'auto', + choices: ['disabled', 'enabled', 'auto'], + description: 'Whether to find the device tree compiler') option('fdt', type: 'combo', value: 'auto', choices: ['disabled', 'enabled', 'auto', 'system', 'internal'], description: 'Whether and how to find the libfdt library')
pc-bios/dtb/meson.build +0 -1
@@ -5,7 +5,6 @@ dtbs = [ 'petalogix-s3adsp1800.dtb', ] -dtc = find_program('dtc', required: false) if dtc.found() foreach out : dtbs f = fs.replace_suffix(out, '.dts')
scripts/meson-buildoptions.sh +3 -0
@@ -116,6 +116,7 @@ meson_options_help() { printf "%s\n" ' dmg dmg image format support' printf "%s\n" ' docs Documentations build support' printf "%s\n" ' dsound DirectSound sound support' + printf "%s\n" ' dtc Whether to find the device tree compiler' printf "%s\n" ' fuse FUSE block device export' printf "%s\n" ' fuse-lseek SEEK_HOLE/SEEK_DATA support for FUSE exports' printf "%s\n" ' gcrypt libgcrypt cryptography support'
@@ -310,6 +311,8 @@ _meson_option_parse() { --disable-docs) printf "%s" -Ddocs=disabled ;; --enable-dsound) printf "%s" -Ddsound=enabled ;; --disable-dsound) printf "%s" -Ddsound=disabled ;; + --enable-dtc) printf "%s" -Ddtc=enabled ;; + --disable-dtc) printf "%s" -Ddtc=disabled ;; --enable-fdt) printf "%s" -Dfdt=enabled ;; --disable-fdt) printf "%s" -Dfdt=disabled ;; --enable-fdt=*) quote_sh "-Dfdt=$2" ;;
Signature · 1 line
2.49.0
Re: [PATCH 4/5] configure: Add dtc option
Quoted 3 lines · Bernhard Beschow
The next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
I don't like this. This might result in some machines missing from some distros if they don't notice this and forget to update their packages and these are machines I care about. Also what guarantees that we don't get bugs by compiling the device tree with different dtc versions which might generate different binaries? The binaries we have now are known to work and don't take much space so keeping them for convenience might be simpler. If it worked so far why change it now?
If this was using dtc from the submodule when not using system libfdt then at least it would not result in dropped machines and we had a known working dtc to compile with. Why isn't that used when we already have it for libfdt?
Regards,
BALATON Zoltan
Quoted 115 lines
Signed-off-by Bernhard Beschow <[email protected]>---
meson.build | 12 ++++++++++++ Kconfig.host | 3 +++ meson_options.txt | 3 +++ pc-bios/dtb/meson.build | 1 - scripts/meson-buildoptions.sh | 3 +++ 5 files changed, 21 insertions(+), 1 deletion(-)meson.build +0
-0@@ -2119,6 +2119,15 @@ if fdt_opt == 'internal' fdt = libfdt_proj.get_variable('libfdt_dep')endif
meson.build +9
-0+dtc = not_found +dtc_opt = get_option('dtc') +if dtc_opt == 'enabled' or (dtc_opt == 'auto' and have_system) + dtc = find_program('dtc', required: dtc_opt == 'enabled') + if not dtc.found() and dtc_opt == 'enabled' + error('dtc not found') + endif +endif +rdma =
not_found
if not get_option('rdma').auto() orhave_system
rdma_libs= [cc.find_library('rdmacm',has_headers: ['rdma/rdma_cma.h'],meson.build +2
-0@@ -2477,6 +2486,7 @@ config_host_data.set('CONFIG_FREEBSD', host_os == 'freebsd') config_host_data.set('CONFIG_CAPSTONE', capstone.found()) config_host_data.set('CONFIG_COCOA', cocoa.found()) config_host_data.set('CONFIG_DARWIN', host_os == 'darwin') +config_host_data.set('CONFIG_DTC', dtc.found()) config_host_data.set('CONFIG_FDT', fdt.found()) config_host_data.set('CONFIG_FUZZ', get_option('fuzzing')) config_host_data.set('CONFIG_GCOV', get_option('b_coverage'))@@ -3314,6 +3324,7 @@ host_kconfig = \ (libcbor.found() ? ['CONFIG_LIBCBOR=y'] : []) + \ (gnutls.found() ? ['CONFIG_GNUTLS=y'] : []) + \ (x11.found() ? ['CONFIG_X11=y'] : []) + \ + (dtc.found() ? ['CONFIG_DTC=y'] : []) + \ (fdt.found() ? ['CONFIG_FDT=y'] : []) + \ (have_vhost_user ? ['CONFIG_VHOST_USER=y'] : []) + \ (have_vhost_vdpa ? ['CONFIG_VHOST_VDPA=y'] : []) + \@@ -4703,6 +4714,7 @@ summary_info += {'sphinx-build': sphinx_build} if config_host.has_key('GDB') summary_info += {'gdb': config_host['GDB']}endif
+summary_info += {'dtc': dtc}summary_info+= {'iasl': iasl}summary_info+= {'genisoimage': config_host['GENISOIMAGE']}
ifhost_os== 'windows' andhave_gaKconfig.host +0
-0@@ -29,6 +29,9 @@ config IVSHMEMconfig TPM
boolKconfig.host +3
-0+config DTC + bool +config FDT
boolmeson_options.txt +3
-0@@ -316,6 +316,9 @@ option('vduse_blk_export', type: 'feature', value: 'auto', option('capstone', type: 'feature', value: 'auto', description: 'Whether and how to find the capstone library') +option('dtc', type: 'combo', value: 'auto', + choices: ['disabled', 'enabled', 'auto'], + description: 'Whether to find the device tree compiler') option('fdt', type: 'combo', value: 'auto', choices: ['disabled', 'enabled', 'auto', 'system', 'internal'], description: 'Whether and how to find the libfdt library')pc-bios/dtb/meson.build +0
-0@@ -5,7 +5,6 @@ dtbs = [ 'petalogix-s3adsp1800.dtb',]
-dtc =
find_program('dtc', required: false)
ifdtc.found()
foreach out : dtbsf = fs.replace_suffix(out, '.dts')scripts/meson-buildoptions.sh +3
-0@@ -116,6 +116,7 @@ meson_options_help() { printf "%s\n" ' dmg dmg image format support' printf "%s\n" ' docs Documentations build support' printf "%s\n" ' dsound DirectSound sound support' + printf "%s\n" ' dtc Whether to find the device tree compiler' printf "%s\n" ' fuse FUSE block device export' printf "%s\n" ' fuse-lseek SEEK_HOLE/SEEK_DATA support for FUSE exports' printf "%s\n" ' gcrypt libgcrypt cryptography support'@@ -310,6 +311,8 @@ _meson_option_parse() { --disable-docs) printf "%s" -Ddocs=disabled ;; --enable-dsound) printf "%s" -Ddsound=enabled ;; --disable-dsound) printf "%s" -Ddsound=disabled ;; + --enable-dtc) printf "%s" -Ddtc=enabled ;; + --disable-dtc) printf "%s" -Ddtc=disabled ;; --enable-fdt) printf "%s" -Dfdt=enabled ;; --disable-fdt) printf "%s" -Dfdt=disabled ;; --enable-fdt=*) quote_sh "-Dfdt=$2" ;;
Re: [PATCH 4/5] configure: Add dtc option
Quoted 8 lines · BALATON Zoltan
Quoted 3 lines · Bernhard Beschow
The next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
I don't like this. This might result in some machines missing from some distros if they don't notice this and forget to update their packages and these are machines I care about.
"Someone might not notice & screw up" is pretty much an argument for not doing anything at all. We publish release notes to alert distros to these kind of changes which is a reasonable effort on our part. Distros need to play their part by reading them and adapting to the changes. They can also compare the available options with new and old QEMU to identify any new build features they need to handle when updating.
Quoted 5 lines
Also what guarantees that we don't get bugs
by compiling the device tree with different dtc versions which might generate different binaries? The binaries we have now are known to work and don't take much space so keeping them for convenience might be simpler. If it worked so far why change it now?
Ultimately the DTC project has responsibility to not cause regressions in new releases they issue, as is the case for any 3rd party dep that we rely on.
We have CI jobs that can run on different distros. If we have coverage of the boards in functional test suite that provides a way to validate the operation with differing DTC versions as a sanity check.
Distributing pre-built binaries in source release tarballs is a long standing issue with QEMU's releases. If we can eliminate the pre-bbuilt DTB and only ship the source that is a good thing IMHO.
Quoted 3 lines
If this was using dtc from the submodule when not using system libfdt then at least it would not result in dropped machines and we had a known working dtc to compile with. Why isn't that used when we already have it for libfdt?
Are there any platforms that we target which still lack dtc ? The submodule was there to plug gaps in distro support long ago, but if every distro has dtc available the submodule is redundant.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [PATCH 4/5] configure: Add dtc option
Quoted 6 lines · Daniel P. Berrangé
Quoted 5 lines · BALATON Zoltan
Quoted 3 lines · Bernhard Beschow
The next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
Quoted 6 lines
Quoted 3 lines
I don't like this. This might result in some machines missing from some distros if they don't notice this and forget to update their packages and these are machines I care about.
"Someone might not notice & screw up" is pretty much an argument for not doing anything at all. We publish release notes to alert distros
More like "if it ain't broke don't fix it".
Quoted 24 lines
to these kind of changes which is a reasonable effort on our part. Distros need to play their part by reading them and adapting to the changes. They can also compare the available options with new and old QEMU to identify any new build features they need to handle when updating.
Quoted 5 lines
Also what guarantees that we don't get bugs
by compiling the device tree with different dtc versions which might generate different binaries? The binaries we have now are known to work and don't take much space so keeping them for convenience might be simpler. If it worked so far why change it now?Ultimately the DTC project has responsibility to not cause regressions in new releases they issue, as is the case for any 3rd party dep that we rely on.
We have CI jobs that can run on different distros. If we have coverage of the boards in functional test suite that provides a way to validate the operation with differing DTC versions as a sanity check.
Distributing pre-built binaries in source release tarballs is a long standing issue with QEMU's releases. If we can eliminate the pre-bbuilt DTB and only ship the source that is a good thing IMHO.
Quoted 7 lines
Quoted 3 lines
If this was using dtc from the submodule when not using system libfdt then at least it would not result in dropped machines and we had a known working dtc to compile with. Why isn't that used when we already have it for libfdt?
Are there any platforms that we target which still lack dtc ? The submodule was there to plug gaps in distro support long ago, but if every distro has dtc available the submodule is redundant.
OK, can we then make libfdt and dtc mandatory and not build at all instead of disabling boards without it? I only care that my boards will be in distro packages so users don't have to compile and I don't have to provide binaries which would also be ensured if we require dtc, then the dtbs can be dropped.
There's still a potential issue that some of the dts files are old and may not compile with newer dtc and changing them may upset the guest but hopefully we won't get that.
Regards,
BALATON Zoltan
Re: [PATCH 4/5] configure: Add dtc option
Quoted 44 lines · BALATON Zoltan
Quoted 6 lines · Daniel P. Berrangé
Quoted 5 lines · BALATON Zoltan
Quoted 3 lines · Bernhard Beschow
The next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
Quoted 6 lines
Quoted 3 lines
I don't like this. This might result in some machines missing from some distros if they don't notice this and forget to update their packages and these are machines I care about.
"Someone might not notice & screw up" is pretty much an argument for not doing anything at all. We publish release notes to alert distros
More like "if it ain't broke don't fix it".
Quoted 28 lines
to these kind of changes which is a reasonable effort on our part. Distros need to play their part by reading them and adapting to the changes. They can also compare the available options with new and old QEMU to identify any new build features they need to handle when updating.
Quoted 5 lines
Also what guarantees that we don't get bugs by compiling the device tree with different dtc versions which might generate different binaries? The binaries we have now are known to work and don't take much space so keeping them for convenience might be simpler. If it worked so far why change it now?
Ultimately the DTC project has responsibility to not cause regressions in new releases they issue, as is the case for any 3rd party dep that we rely on.
We have CI jobs that can run on different distros. If we have coverage of the boards in functional test suite that provides a way to validate the operation with differing DTC versions as a sanity check.
Distributing pre-built binaries in source release tarballs is a long standing issue with QEMU's releases. If we can eliminate the pre-bbuilt DTB and only ship the source that is a good thing IMHO.
Quoted 3 lines
If this was using dtc from the submodule when not using system libfdt then at least it would not result in dropped machines and we had a known working dtc to compile with. Why isn't that used when we already have it for libfdt?
Quoted 5 lines
Quoted 3 lines
Are there any platforms that we target which still lack dtc ? The submodule was there to plug gaps in distro support long ago, but if every distro has dtc available the submodule is redundant.
OK, can we then make libfdt and dtc mandatory and not build at all instead of disabling boards without it?
Disabling libfdt was done because of the x86 microvm board, which was adding a libfdt dependency to all of qemu-system-x86_64.
However, several targets still have a hard dependency on libfdt, plus libfdt is supported as a subproject and will be built together with the emulators. So the impact of the change was minimal, and limited only to people using --disable-libfdt or --disable-download on the configure command line. This is not true of Bernard's proposal to use dtc.
For now, one possibility is just to make dtc mandatory whenever one of this boards is used; the best would be to be able to use the dtc that comes from the libfdt subproject. This however requires patching dtc's meson.build so that it calls meson.override_find_program (and it could also use meson.override_dependency while you're at it).
I'll send a patch to the upstream dtc project and you can add it to QEMU via diff_files (see commit 64644bc4eab2f for an example).
Thanks,
Paolo
Re: [PATCH 4/5] configure: Add dtc option
Quoted 1 line · Paolo Bonzini
I'll send a patch to the upstream dtc project and you can add it to QEMU via
diff_files(see commit64644bc4eab2ffor an example).
This is the patch, backported to QEMU's version of the dtc subproject:
libfdt/meson.build +1 -0
@@ -30,6 +30,7 @@ libfdt_dep = declare_dependency( include_directories: libfdt_inc, link_with: libfdt, ) +meson.override_dependency('libfdt', libfdt_dep) install_headers( files(
meson.build +8 -3
@@ -54,6 +54,7 @@ version_gen_h = vcs_tag( subdir('libfdt') +dtc_tools = [] if get_option('tools') flex = find_program('flex', required: true) bison = find_program('bison', required: true)
@@ -77,7 +78,7 @@ if get_option('tools') ) if cc.check_header('fnmatch.h') - executable( + dtc_tools += executable( 'convert-dtsv0', [ lgen.process('convert-dtsv0-lexer.l'),
@@ -88,7 +89,7 @@ if get_option('tools') ) endif - executable( + dtc_tools += executable( 'dtc', [ lgen.process('dtc-lexer.l'),
@@ -108,7 +109,7 @@ if get_option('tools') ) foreach e: ['fdtdump', 'fdtget', 'fdtput', 'fdtoverlay'] - executable(e, files(e + '.c'), dependencies: util_dep, install: true) + dtc_tools += executable(e, files(e + '.c'), dependencies: util_dep, install: true) endforeach install_data(
@@ -118,6 +119,10 @@ if get_option('tools') ) endif +foreach e: dtc_tools + meson.override_find_program(e.name(), e) +endforeach + if not meson.is_cross_build() if py.found() and swig.found() subdir('pylibfdt')
and this is how you can then find dtc in QEMU's meson.build:
meson.build +10 -3
@@ -2121,13 +2121,15 @@ if numa.found() and not cc.links(''' endif fdt = not_found +dtc = not_found fdt_opt = get_option('fdt') if fdt_opt == 'enabled' and get_option('wrap_mode') == 'nodownload' fdt_opt = 'system' endif if fdt_opt in ['enabled', 'system'] or (fdt_opt == 'auto' and have_system) fdt = cc.find_library('fdt', required: fdt_opt == 'system') - if fdt.found() and cc.links(''' + dtc = find_program('dtc', required: fdt_opt == 'system') + if dtc.found() and fdt.found() and cc.links(''' #include <libfdt.h> #include <libfdt_env.h> int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',
@@ -2136,8 +2138,12 @@ if fdt_opt in ['enabled', 'system'] or (fdt_opt == 'auto' and have_system) elif fdt_opt != 'system' fdt_opt = get_option('wrap_mode') == 'nodownload' ? 'disabled' : 'internal' fdt = not_found + dtc = not_found else - error('system libfdt is too old (1.5.1 or newer required)') + if dtc.found() + error('system libfdt is too old (1.5.1 or newer required)') + else + error('device tree compiler not found') endif endif if fdt_opt == 'internal'
@@ -2145,7 +2148,8 @@ if fdt_opt == 'internal' libfdt_proj = subproject('dtc', required: true, default_options: ['tools=false', 'yaml=disabled', 'python=disabled', 'default_library=static']) - fdt = libfdt_proj.get_variable('libfdt_dep') + fdt = dependency('libfdt', required: true) + dtc = find_program('dtc', required: true) endif rdma = not_found
Re: [PATCH 4/5] configure: Add dtc option
Am 12. Juni 2025 05:46:03 UTC schrieb Paolo Bonzini <[email protected]>:
Quoted 103 lines · Paolo Bonzini
Quoted 1 line
I'll send a patch to the upstream dtc project and you can add it to QEMU via
diff_files(see commit64644bc4eab2ffor an example).This is the patch, backported to QEMU's version of the dtc subproject:
libfdt/meson.build +1
-0@@ -30,6 +30,7 @@ libfdt_dep = declare_dependency( include_directories: libfdt_inc, link_with: libfdt, ) +meson.override_dependency('libfdt', libfdt_dep) install_headers( files(meson.build +4
-3@@ -54,6 +54,7 @@ version_gen_h = vcs_tag( subdir('libfdt') +dtc_tools = [] if get_option('tools') flex = find_program('flex', required: true) bison = find_program('bison', required: true)@@ -77,7 +78,7 @@ if get_option('tools') ) if cc.check_header('fnmatch.h') - executable( + dtc_tools += executable( 'convert-dtsv0', [ lgen.process('convert-dtsv0-lexer.l'),@@ -88,7 +89,7 @@ if get_option('tools') ) endif - executable( + dtc_tools += executable( 'dtc', [ lgen.process('dtc-lexer.l'),@@ -108,7 +109,7 @@ if get_option('tools') ) foreach e: ['fdtdump', 'fdtget', 'fdtput', 'fdtoverlay'] - executable(e, files(e + '.c'), dependencies: util_dep, install: true) + dtc_tools += executable(e, files(e + '.c'), dependencies: util_dep, install: true) endforeach install_data(@@ -118,6 +119,10 @@ if get_option('tools') )endif
meson.build +4
-0+foreach e: dtc_tools + meson.override_find_program(e.name(), e) +endforeach +if not
meson.is_cross_build()
ifpy.found()andswig.found()
subdir('pylibfdt')and this is how you can then find dtc in QEMU's meson.build:
meson.build +0
-0@@ -2121,13 +2121,15 @@ if numa.found() and not cc.links('''endif
fdt =not_found
+dtc =not_foundfdt_opt=get_option('fdt')
iffdt_opt== 'enabled' andget_option('wrap_mode')== 'nodownload'
fdt_opt= 'system'
endif
iffdt_optin ['enabled', 'system'] or (fdt_opt== 'auto' andhave_system)
fdt =cc.find_library('fdt', required: fdt_opt == 'system')meson.build +7
-2- if fdt.found() and cc.links(''' + dtc = find_program('dtc', required: fdt_opt == 'system') + if dtc.found() and fdt.found() and cc.links(''' #include <libfdt.h> #include <libfdt_env.h> int main(void) { fdt_find_max_phandle(NULL, NULL); return 0; }''',@@ -2136,8 +2138,12 @@ if fdt_opt in ['enabled', 'system'] or (fdt_opt == 'auto' and have_system) elif fdt_opt != 'system' fdt_opt = get_option('wrap_mode') == 'nodownload' ? 'disabled' : 'internal' fdt = not_found + dtc = not_found else - error('system libfdt is too old (1.5.1 or newer required)') + if dtc.found() + error('system libfdt is too old (1.5.1 or newer required)') + else + error('device tree compiler not found') endifendif
iffdt_opt== 'internal'meson.build +2
-1@@ -2145,7 +2148,8 @@ if fdt_opt == 'internal' libfdt_proj = subproject('dtc', required: true, default_options: ['tools=false', 'yaml=disabled', 'python=disabled', 'default_library=static']) - fdt = libfdt_proj.get_variable('libfdt_dep') + fdt = dependency('libfdt', required: true) + dtc = find_program('dtc', required: true)
Quoted 3 lines
endif
rdma =not_found
Thanks Paolo, I'll send a v2.
Best regards,
Bernhard
Re: [PATCH 4/5] configure: Add dtc option
In reply to Paolo BonziniQuoted 53 lines · Paolo Bonzini
Quoted 44 lines · BALATON Zoltan
Quoted 6 lines · Daniel P. Berrangé
Quoted 5 lines · BALATON Zoltan
Quoted 3 lines · Bernhard Beschow
The next patch will make dtc mandatory for boards with a bundled DTB, causing these boards to be omitted when dtc is missing. Allow packagers to force the build of those boards by providing an option.
Quoted 6 lines
Quoted 3 lines
I don't like this. This might result in some machines missing from some distros if they don't notice this and forget to update their packages and these are machines I care about.
"Someone might not notice & screw up" is pretty much an argument for not doing anything at all. We publish release notes to alert distros
More like "if it ain't broke don't fix it".
Quoted 28 lines
to these kind of changes which is a reasonable effort on our part. Distros need to play their part by reading them and adapting to the changes. They can also compare the available options with new and old QEMU to identify any new build features they need to handle when updating.
Quoted 5 lines
Also what guarantees that we don't get bugs by compiling the device tree with different dtc versions which might generate different binaries? The binaries we have now are known to work and don't take much space so keeping them for convenience might be simpler. If it worked so far why change it now?
Ultimately the DTC project has responsibility to not cause regressions in new releases they issue, as is the case for any 3rd party dep that we rely on.
We have CI jobs that can run on different distros. If we have coverage of the boards in functional test suite that provides a way to validate the operation with differing DTC versions as a sanity check.
Distributing pre-built binaries in source release tarballs is a long standing issue with QEMU's releases. If we can eliminate the pre-bbuilt DTB and only ship the source that is a good thing IMHO.
Quoted 3 lines
If this was using dtc from the submodule when not using system libfdt then at least it would not result in dropped machines and we had a known working dtc to compile with. Why isn't that used when we already have it for libfdt?
Quoted 5 lines
Quoted 3 lines
Are there any platforms that we target which still lack dtc ? The submodule was there to plug gaps in distro support long ago, but if every distro has dtc available the submodule is redundant.
OK, can we then make libfdt and dtc mandatory and not build at all instead of disabling boards without it?
Disabling libfdt was done because of the x86 microvm board, which was adding a libfdt dependency to all of qemu-system-x86_64.
Quoted 7 lines
However, several targets still have a hard dependency on libfdt, plus libfdt is supported as a subproject and will be built together with the emulators. So the impact of the change was minimal, and limited only to people using --disable-libfdt or --disable-download on the configure command line. This
is not true of Bernard's proposal to use dtc.For now, one possibility is just to make dtc mandatory whenever one of this boards is used; the best would be to be able to use the dtc that comes from the libfdt subproject. This however requires patching dtc's meson.build so that it calls meson.override_find_program (and it could also use meson.override_dependency while you're at it).
I'll send a patch to the upstream dtc project and you can add it to QEMU via
diff_files(see commit64644bc4eab2ffor an example).
This seems getting too complicated just to get rid of some small binaries so I'd just leave it as it is now but if Bernhard wants to make this change I'm OK with it as long as my boards won't get disabled in configs where now they are built I.e. if no system dtc is found now the dtbs are installed; if this used dtc from submodule instead that would be OK but dropping the board is not. I'm also OK with replacing the submodule with always requiring system libfdt and dtc but that may be more work for distros to update their package dependencies.
Regards,
BALATON Zoltan
PATCH 5/5 configure: Require dtc for boards with bundled device trees
In reply to Bernhard BeschowAvoid having generated DTBs checked in which can become inconsistent with their sources.
Note that when dtc is not found, the boards with bundled DTBs will be omitted from the build. Finding dtc, and thus building these boards, can be enforced with the dtc build option.
--- MAINTAINERS | 2 +- hw/microblaze/Kconfig | 4 ++-- hw/ppc/Kconfig | 4 ++-- pc-bios/dtb/bamboo.dtb | Bin 3211 -> 0 bytes pc-bios/dtb/canyonlands.dtb | Bin 9779 -> 0 bytes pc-bios/dtb/meson.build | 16 +++++++--------- pc-bios/dtb/petalogix-ml605.dtb | Bin 9882 -> 0 bytes pc-bios/dtb/petalogix-s3adsp1800.dtb | Bin 8161 -> 0 bytes 8 files changed, 12 insertions(+), 14 deletions(-) delete mode 100644 pc-bios/dtb/bamboo.dtb delete mode 100644 pc-bios/dtb/canyonlands.dtb delete mode 100644 pc-bios/dtb/petalogix-ml605.dtb delete mode 100644 pc-bios/dtb/petalogix-s3adsp1800.dtb
MAINTAINERS +1 -1
@@ -1605,7 +1605,7 @@ F: hw/pci-host/ppc440_pcix.c F: hw/display/sm501* F: hw/ide/sii3112.c F: hw/rtc/m41t80.c -F: pc-bios/dtb/canyonlands.dt[sb] +F: pc-bios/dtb/canyonlands.dts F: pc-bios/u-boot-sam460ex-20100605.bin F: roms/u-boot-sam460ex F: docs/system/ppc/amigang.rst
hw/microblaze/Kconfig +2 -2
@@ -1,7 +1,7 @@ config PETALOGIX_S3ADSP1800 bool default y - depends on MICROBLAZE + depends on MICROBLAZE && DTC select PFLASH_CFI01 select XILINX select XILINX_AXI
@@ -11,7 +11,7 @@ config PETALOGIX_S3ADSP1800 config PETALOGIX_ML605 bool default y - depends on MICROBLAZE + depends on MICROBLAZE && DTC select PFLASH_CFI01 select SERIAL_MM select SSI_M25P80
hw/ppc/Kconfig +2 -2
@@ -47,7 +47,7 @@ config POWERNV config PPC440 bool default y - depends on PPC && FDT + depends on PPC && FDT && DTC imply PCI_DEVICES imply TEST_DEVICES imply E1000_PCI
@@ -66,7 +66,7 @@ config PPC4XX config SAM460EX bool default y - depends on PPC && FDT + depends on PPC && FDT && DTC select PFLASH_CFI01 select IDE_SII3112 select M41T80
pc-bios/dtb/bamboo.dtb deleted binary +0 -0
pc-bios/dtb/canyonlands.dtb deleted binary +0 -0
pc-bios/dtb/meson.build +7 -9
@@ -1,13 +1,13 @@ -dtbs = [ - 'bamboo.dtb', - 'canyonlands.dtb', - 'petalogix-ml605.dtb', - 'petalogix-s3adsp1800.dtb', +dts_sources = [ + 'bamboo.dts', + 'canyonlands.dts', + 'petalogix-s3adsp1800.dts', + 'petalogix-ml605.dts', ] if dtc.found() - foreach out : dtbs - f = fs.replace_suffix(out, '.dts') + foreach f : dts_sources + out = fs.replace_suffix(f, '.dtb') custom_target(out, build_by_default: have_system, input: files(f),
@@ -17,6 +17,4 @@ if dtc.found() command: [ dtc, '-q', '-I', 'dts', '-O', 'dtb', '-o', '@OUTPUT@', '@INPUT0@' ]) endforeach -else - install_data(dtbs, install_dir: qemu_datadir / 'dtb') endif
pc-bios/dtb/petalogix-ml605.dtb deleted binary +0 -0
pc-bios/dtb/petalogix-s3adsp1800.dtb deleted binary +0 -0
Signature · 1 line
2.49.0
Paolo Bonzini
AppliedQuoted 7 lines · Bernhard Beschow
This series removes the bundled device tree blobs from the repository which are redundant to the bundled device tree sources. To make this work, the device tree compiler 'dtc' is now required for the boards which rely on these device trees. Building these board can be enforced with a 'dtc' build option which should help packagers. For simplicity, 'dtc' is searched for in the system only rather than also considering the 'dtc' submodule.
Quoted 5 lines
I also tried unsing Meson's
source_set::add(when: ...)for building only the needed DTBs but eventually found it to be too over-engineered.Last but not least two inconsistencies regarding
QEMU_FILE_TYPE_DTBare fixed which seem to be latent bugs (one of which materializes in my e500-fdt branch).
Applied patches 1-3, thanks.
Paolo
Quoted 28 lines
Bernhard Beschow (5):
hw: Fix type constant for DTB files pc-bios/dtb/meson: Prefer target name to be outfile, not infilescripts/meson-buildoptions: Sortcoroutine_backendchoices
lexicographically
configure: Add dtc option
configure: Require dtc for boards with bundled device treesMAINTAINERS | 2 +- meson.build | 12 ++++++++++++ hw/arm/boot.c | 2 +- hw/ppc/e500.c | 2 +- Kconfig.host | 3 +++ hw/microblaze/Kconfig | 4 ++-- hw/ppc/Kconfig | 4 ++-- meson_options.txt | 3 +++ pc-bios/dtb/bamboo.dtb | Bin 3211 -> 0 bytes pc-bios/dtb/canyonlands.dtb | Bin 9779 -> 0 bytes pc-bios/dtb/meson.build | 19 ++++++++----------- pc-bios/dtb/petalogix-ml605.dtb | Bin 9882 -> 0 bytes pc-bios/dtb/petalogix-s3adsp1800.dtb | Bin 8161 -> 0 bytes scripts/meson-buildoptions.sh | 5 ++++- 14 files changed, 37 insertions(+), 19 deletions(-)delete mode 100644 pc-bios/dtb/bamboo.dtb delete mode 100644 pc-bios/dtb/canyonlands.dtb delete mode 100644 pc-bios/dtb/petalogix-ml605.dtb delete mode 100644 pc-bios/dtb/petalogix-s3adsp1800.dtb