sunxi: produce working binaries by default
Subsystems (2)
cover sunxi: produce working binaries by default
Hi,
currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit 4c78028737c3 ("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.
So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.
While at it, replace .its generator script with more generic Dust template based approach, which should allow for more code sharing between boards and thus reduce code duplication.
Cheers,
Petr
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403105.html
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403650.html
Ref: https://lists.denx.de/pipermail/u-boot/2019-December/393556.html
Cc: Harald Seiler <[email protected]>
Cc: Tom Rini <[email protected]>
Cc: Michal Simek <[email protected]>
Petr ?tetiar (3):
Kconfig: add config options for automatic builds
tools: add Dust based .its file templating
sunxi: replace .its file generator with Dust template
Kconfig | 17 +- Makefile | 8 + board/sunxi/mksunxi_fit_atf.sh | 87 - board/sunxi/u-boot-its.mk | 19 + board/sunxi/u-boot.its.dust | 45 + include/u-boot-its.mk | 37 + tools/ashes/ashes.py | 2723 ++++++++++++++++++++++++++++++++ 7 files changed, 2848 insertions(+), 88 deletions(-) delete mode 100755 board/sunxi/mksunxi_fit_atf.sh create mode 100644 board/sunxi/u-boot-its.mk create mode 100644 board/sunxi/u-boot.its.dust create mode 100644 include/u-boot-its.mk create mode 100755 tools/ashes/ashes.py
PATCH 1/3 Kconfig: add config options for automatic builds
Currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit 4c78028737c3 ("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.
So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.
--- Kconfig | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Kconfig +12 -0
@@ -20,6 +20,18 @@ config BROKEN This option cannot be enabled. It is used as dependency for broken and incomplete features. +config BUILDBOT + bool "Set build defaults for automatic builds" + help + This option allows setting of usable defaults for automatic builds. + +config BUILDBOT_BROKEN_BINARIES + bool "Allow building of broken binaries" + depends on BUILDBOT + help + Resulting images wont be used for runtime testing, thus completion + of build is preferred. + config DEPRECATED bool help
PATCH 2/3 tools: add Dust based .its file templating
In reply to Petr ŠtetiarCurrently boards with more complex image requirements can provide .its generator script, which lead to a bunch of similar shell scripts, thus duplication of a lot of similar code.
This patch adds a posibility to use a template based approach, where one just writes .its Dust template which is then populated with JSON data using Ashes utility.
Ashes is single file Python2/3 templating utility implementing Dust[1] JavaScript templating engine syntax.
Templates can be files or passed at the command line. Models, the input data to the template, are passed in as JSON, either as a command line option, or through stdin.
Ashes is currently used in production settings at PayPal and Rackspace.
Imported verbatim from Git repo[2] at commit 9b4317a72aa0 ("bump version for 19.2.1dev"), just replaced license info with SPDX license tag.
1. http://akdubya.github.io/dustjs
2. https://github.com/mahmoud/ashes
--- include/u-boot-its.mk | 37 + tools/ashes/ashes.py | 2723 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 2760 insertions(+) create mode 100644 include/u-boot-its.mk create mode 100755 tools/ashes/ashes.py
include/u-boot-its.mk new file +37 -0
@@ -0,0 +1,37 @@ +comma=, + +U_BOOT_ITS_DUST = $(srctree)/board/$(BOARDDIR)/u-boot.its.dust +U_BOOT_ITS_JSON = $(srctree)/u-boot.its.json + +$(U_BOOT_ITS_JSON): FORCE + $(call cmd,gen_its_json) + +define check_its_dep + @if ! test -f "$(1)"; then \ + if test "$(CONFIG_BUILDBOT_BROKEN_BINARIES)" = "y"; then \ + touch "$(1)"; \ + else \ + echo "ERROR: $(2) file $(1) NOT found. If you want to build without " >&2; \ + echo "a $(2) file (creating a NON-FUNCTIONAL binary), then enable" >&2; \ + echo "config option CONFIG_BUILDBOT_BROKEN_BINARIES." >&2; \ + false; \ + fi \ + fi +endef + +dtb_of_list=$(subst ",,$(CONFIG_OF_LIST)) +define gen_its_json_dtbs + $(foreach dtb,$(dtb_of_list), { \ + "path": "$(patsubst %,arch/$(ARCH)/dts/%.dtb,$(dtb))"$(comma) \ + "name": "$(dtb)" } \ + $(shell [ "$(dtb)" != "$(lastword $(dtb_of_list))" ] && echo $(comma))) +endef + +quiet_cmd_gen_its = GENITS $@ +cmd_gen_its = \ + $(srctree)/tools/ashes/ashes.py \ + $(if $(ASHES_VERBOSE),--verbose) \ + --no-filter \ + --template $(U_BOOT_ITS_DUST) \ + --model $(U_BOOT_ITS_JSON) \ + --output $@
tools/ashes/ashes.py new file +2723 -0
2,723 lines not on this page · raw message
PATCH 3/3 sunxi: replace .its file generator with Dust template
In reply to Petr ŠtetiarReplace current .its file based generator with generic Dust template based approach, including the proper .its file dependency tracking, so working images are produced by default.
bl31.bin file is mandatory for functional, usable and bootable binaries, and currently missing bl31.bin is not hard error, just a warning.
--- Kconfig | 5 +- Makefile | 8 ++++ board/sunxi/mksunxi_fit_atf.sh | 87 ---------------------------------- board/sunxi/u-boot-its.mk | 19 ++++++++ board/sunxi/u-boot.its.dust | 45 ++++++++++++++++++ 5 files changed, 76 insertions(+), 88 deletions(-) delete mode 100755 board/sunxi/mksunxi_fit_atf.sh create mode 100644 board/sunxi/u-boot-its.mk create mode 100644 board/sunxi/u-boot.its.dust
Kconfig +4 -1
@@ -517,10 +517,13 @@ config SPL_FIT_SOURCE U-Boot FIT image. This could specify further image to load and/or execute. +config SPL_FIT_TEMPLATE + bool "Use Dust template based .its file generation for U-Boot FIT image" + default y if SPL_LOAD_FIT && ARCH_SUNXI + config SPL_FIT_GENERATOR string ".its file generator script for U-Boot FIT image" depends on SPL_FIT - default "board/sunxi/mksunxi_fit_atf.sh" if SPL_LOAD_FIT && ARCH_SUNXI default "arch/arm/mach-rockchip/make_fit_atf.py" if SPL_LOAD_FIT && ARCH_ROCKCHIP default "arch/arm/mach-zynqmp/mkimage_fit_atf.sh" if SPL_LOAD_FIT && ARCH_ZYNQMP default "arch/riscv/lib/mkimage_fit_opensbi.sh" if SPL_LOAD_FIT && RISCV
Makefile +8 -0
@@ -1290,6 +1290,14 @@ $(U_BOOT_ITS): $(U_BOOT_ITS_DEPS) FORCE endif endif +ifeq ($(CONFIG_SPL_FIT_TEMPLATE),y) +U_BOOT_ITS := u-boot.its +include $(srctree)/include/u-boot-its.mk +sinclude $(srctree)/board/$(BOARDDIR)/u-boot-its.mk +$(U_BOOT_ITS): $(U_BOOT_ITS_DEPS) $(U_BOOT_ITS_JSON) FORCE + $(call cmd,gen_its) +endif + ifdef CONFIG_SPL_LOAD_FIT MKIMAGEFLAGS_u-boot.img = -f auto -A $(ARCH) -T firmware -C none -O u-boot \ -a $(CONFIG_SYS_TEXT_BASE) -e $(CONFIG_SYS_UBOOT_START) \
board/sunxi/mksunxi_fit_atf.sh deleted +0 -87
87 lines not on this page · raw message
board/sunxi/u-boot-its.mk new file +19 -0
@@ -0,0 +1,19 @@ +BL31 := $(CURDIR)/bl31.bin +BL31_ADDR := $(if $(CONFIG_MACH_SUN50I_H6),0x104000,0x104000) + +$(BL31): + $(call check_its_dep,$@,BL31) + +U_BOOT_ITS_DEPS += $(BL31) + +quiet_cmd_gen_its_json = GENITS_JSON $@ +define cmd_gen_its_json + echo '{ \ + "bl31": "$(BL31)", \ + "load": "$(BL31_ADDR)", \ + "dtbs": [ \ + $(call gen_its_json_dtbs) \ + ] \ + }' > $@ +endef +
board/sunxi/u-boot.its.dust new file +45 -0
45 lines not on this page · raw message
cover sunxi: produce working binaries by default
In reply to Petr ŠtetiarHi Petr,
Quoted 6 lines
currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit
4c78028737c3("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.
Quoted 7 lines
So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.
While at it, replace .its generator script with more generic Dust template based approach, which should allow for more code sharing between boards and thus reduce code duplication.
I'm not sure what Dust is but a the original script for making the .itb files was based on python with pyelftools dep and some we're happy with the added python deps it so it was rewritten into the latest shell script. I'm not sure from looking what extra deps the dust templating adds vs other options but it's worth the consideration.
Peter
Quoted 30 lines
Cheers,
Petr
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403105.html
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403650.html
Ref: https://lists.denx.de/pipermail/u-boot/2019-December/393556.html
Cc: Harald Seiler <[email protected]>
Cc: Tom Rini <[email protected]>
Cc: Michal Simek <[email protected]>Petr ?tetiar (3):
Kconfig: add config options for automatic builds
tools: add Dust based .its file templating
sunxi: replace .its file generator with Dust templateKconfig | 17 +- Makefile | 8 + board/sunxi/mksunxi_fit_atf.sh | 87 - board/sunxi/u-boot-its.mk | 19 + board/sunxi/u-boot.its.dust | 45 + include/u-boot-its.mk | 37 + tools/ashes/ashes.py | 2723 ++++++++++++++++++++++++++++++++ 7 files changed, 2848 insertions(+), 88 deletions(-) delete mode 100755 board/sunxi/mksunxi_fit_atf.sh create mode 100644 board/sunxi/u-boot-its.mk create mode 100644 board/sunxi/u-boot.its.dust create mode 100644 include/u-boot-its.mk create mode 100755 tools/ashes/ashes.py