Skip to contentSkip to conversation

sunxi: produce working binaries by default

PATCH New Petr Štetiar 2 participants 5 messages Mar 23, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover sunxi: produce working binaries by default

Hi,

currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit 4c78028737c3 ("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.

So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.

While at it, replace .its generator script with more generic Dust template based approach, which should allow for more code sharing between boards and thus reduce code duplication.

Cheers,

Petr

Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403105.html
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403650.html
Ref: https://lists.denx.de/pipermail/u-boot/2019-December/393556.html
Cc: Harald Seiler <[email protected]>
Cc: Tom Rini <[email protected]>
Cc: Michal Simek <[email protected]>

Petr ?tetiar (3):
Kconfig: add config options for automatic builds
tools: add Dust based .its file templating
sunxi: replace .its file generator with Dust template

 Kconfig                        |   17 +-
 Makefile                       |    8 +
 board/sunxi/mksunxi_fit_atf.sh |   87 -
 board/sunxi/u-boot-its.mk      |   19 +
 board/sunxi/u-boot.its.dust    |   45 +
 include/u-boot-its.mk          |   37 +
 tools/ashes/ashes.py           | 2723 ++++++++++++++++++++++++++++++++
 7 files changed, 2848 insertions(+), 88 deletions(-)
 delete mode 100755 board/sunxi/mksunxi_fit_atf.sh
 create mode 100644 board/sunxi/u-boot-its.mk
 create mode 100644 board/sunxi/u-boot.its.dust
 create mode 100644 include/u-boot-its.mk
 create mode 100755 tools/ashes/ashes.py
#1 lore raw

PATCH 1/3 Kconfig: add config options for automatic builds

Currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit 4c78028737c3 ("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.

So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.

---
 Kconfig | 12 ++++++++++++
 1 file changed, 12 insertions(+)
Kconfig +12 -0
index 66148ce47790..7799a3b19629 100644
@@ -20,6 +20,18 @@ config BROKEN
 	  This option cannot be enabled. It is used as dependency
 	  for broken and incomplete features.
 
+config BUILDBOT
+	bool "Set build defaults for automatic builds"
+	help
+	  This option allows setting of usable defaults for automatic builds.
+
+config BUILDBOT_BROKEN_BINARIES
+	bool "Allow building of broken binaries"
+	depends on BUILDBOT
+	help
+	  Resulting images wont be used for runtime testing, thus completion
+	  of build is preferred.
+
 config DEPRECATED
 	bool
 	help
#2 lore raw

PATCH 2/3 tools: add Dust based .its file templating

In reply to Petr Štetiar

Currently boards with more complex image requirements can provide .its generator script, which lead to a bunch of similar shell scripts, thus duplication of a lot of similar code.

This patch adds a posibility to use a template based approach, where one just writes .its Dust template which is then populated with JSON data using Ashes utility.

Ashes is single file Python2/3 templating utility implementing Dust[1] JavaScript templating engine syntax.

Templates can be files or passed at the command line. Models, the input data to the template, are passed in as JSON, either as a command line option, or through stdin.

Ashes is currently used in production settings at PayPal and Rackspace.

Imported verbatim from Git repo[2] at commit 9b4317a72aa0 ("bump version for 19.2.1dev"), just replaced license info with SPDX license tag.

1. http://akdubya.github.io/dustjs
2. https://github.com/mahmoud/ashes

---
 include/u-boot-its.mk |   37 +
 tools/ashes/ashes.py  | 2723 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 2760 insertions(+)
 create mode 100644 include/u-boot-its.mk
 create mode 100755 tools/ashes/ashes.py
include/u-boot-its.mk new file +37 -0
new file mode 100644
index 000000000000..86f2573a2ebf
@@ -0,0 +1,37 @@
+comma=,
+
+U_BOOT_ITS_DUST = $(srctree)/board/$(BOARDDIR)/u-boot.its.dust
+U_BOOT_ITS_JSON = $(srctree)/u-boot.its.json
+
+$(U_BOOT_ITS_JSON): FORCE
+	$(call cmd,gen_its_json)
+
+define check_its_dep
+	@if ! test -f "$(1)"; then \
+		if test "$(CONFIG_BUILDBOT_BROKEN_BINARIES)" = "y"; then \
+			touch "$(1)"; \
+		else \
+			echo "ERROR: $(2) file $(1) NOT found. If you want to build without " >&2; \
+			echo "a $(2) file (creating a NON-FUNCTIONAL binary), then enable" >&2; \
+			echo "config option CONFIG_BUILDBOT_BROKEN_BINARIES." >&2; \
+			false; \
+		fi \
+	fi
+endef
+
+dtb_of_list=$(subst ",,$(CONFIG_OF_LIST))
+define gen_its_json_dtbs
+	$(foreach dtb,$(dtb_of_list), { \
+		"path": "$(patsubst %,arch/$(ARCH)/dts/%.dtb,$(dtb))"$(comma) \
+		"name": "$(dtb)" } \
+		$(shell [ "$(dtb)" != "$(lastword $(dtb_of_list))" ] && echo $(comma)))
+endef
+
+quiet_cmd_gen_its = GENITS  $@
+cmd_gen_its = \
+	$(srctree)/tools/ashes/ashes.py \
+	$(if $(ASHES_VERBOSE),--verbose) \
+	--no-filter \
+	--template $(U_BOOT_ITS_DUST) \
+	--model $(U_BOOT_ITS_JSON) \
+	--output $@
tools/ashes/ashes.py new file +2723 -0

2,723 lines not on this page · raw message

#3 lore raw

PATCH 3/3 sunxi: replace .its file generator with Dust template

In reply to Petr Štetiar

Replace current .its file based generator with generic Dust template based approach, including the proper .its file dependency tracking, so working images are produced by default.

bl31.bin file is mandatory for functional, usable and bootable binaries, and currently missing bl31.bin is not hard error, just a warning.

---
 Kconfig                        |  5 +-
 Makefile                       |  8 ++++
 board/sunxi/mksunxi_fit_atf.sh | 87 ----------------------------------
 board/sunxi/u-boot-its.mk      | 19 ++++++++
 board/sunxi/u-boot.its.dust    | 45 ++++++++++++++++++
 5 files changed, 76 insertions(+), 88 deletions(-)
 delete mode 100755 board/sunxi/mksunxi_fit_atf.sh
 create mode 100644 board/sunxi/u-boot-its.mk
 create mode 100644 board/sunxi/u-boot.its.dust
Kconfig +4 -1
index 7799a3b19629..ebdc3beddb56 100644
@@ -517,10 +517,13 @@ config SPL_FIT_SOURCE
 	  U-Boot FIT image. This could specify further image to load and/or
 	  execute.
 
+config SPL_FIT_TEMPLATE
+	bool "Use Dust template based .its file generation for U-Boot FIT image"
+	default y if SPL_LOAD_FIT && ARCH_SUNXI
+
 config SPL_FIT_GENERATOR
 	string ".its file generator script for U-Boot FIT image"
 	depends on SPL_FIT
-	default "board/sunxi/mksunxi_fit_atf.sh" if SPL_LOAD_FIT && ARCH_SUNXI
 	default "arch/arm/mach-rockchip/make_fit_atf.py" if SPL_LOAD_FIT && ARCH_ROCKCHIP
 	default "arch/arm/mach-zynqmp/mkimage_fit_atf.sh" if SPL_LOAD_FIT && ARCH_ZYNQMP
 	default "arch/riscv/lib/mkimage_fit_opensbi.sh" if SPL_LOAD_FIT && RISCV
Makefile +8 -0
index fa687f13a588..131e59a2fe01 100644
@@ -1290,6 +1290,14 @@ $(U_BOOT_ITS): $(U_BOOT_ITS_DEPS) FORCE
 endif
 endif
 
+ifeq ($(CONFIG_SPL_FIT_TEMPLATE),y)
+U_BOOT_ITS := u-boot.its
+include $(srctree)/include/u-boot-its.mk
+sinclude $(srctree)/board/$(BOARDDIR)/u-boot-its.mk
+$(U_BOOT_ITS): $(U_BOOT_ITS_DEPS) $(U_BOOT_ITS_JSON) FORCE
+	$(call cmd,gen_its)
+endif
+
 ifdef CONFIG_SPL_LOAD_FIT
 MKIMAGEFLAGS_u-boot.img = -f auto -A $(ARCH) -T firmware -C none -O u-boot \
 	-a $(CONFIG_SYS_TEXT_BASE) -e $(CONFIG_SYS_UBOOT_START) \
board/sunxi/mksunxi_fit_atf.sh deleted +0 -87

87 lines not on this page · raw message

board/sunxi/u-boot-its.mk new file +19 -0
new file mode 100644
index 000000000000..c6294f18889f
@@ -0,0 +1,19 @@
+BL31 := $(CURDIR)/bl31.bin
+BL31_ADDR := $(if $(CONFIG_MACH_SUN50I_H6),0x104000,0x104000)
+
+$(BL31):
+	$(call check_its_dep,$@,BL31)
+
+U_BOOT_ITS_DEPS += $(BL31)
+
+quiet_cmd_gen_its_json = GENITS_JSON  $@
+define cmd_gen_its_json
+	echo '{ \
+		"bl31": "$(BL31)", \
+		"load": "$(BL31_ADDR)", \
+		"dtbs": [ \
+			$(call gen_its_json_dtbs) \
+		] \
+	}' > $@
+endef
+
board/sunxi/u-boot.its.dust new file +45 -0

45 lines not on this page · raw message

#4 lore raw

cover sunxi: produce working binaries by default

In reply to Petr Štetiar

Hi Petr,

Quoted 6 lines

currently its not possible to distinguish between normal builds and builds performed by the build bots/CI, thus leading to a workarounds like for example in commit 4c78028737c3 ("mksunxi_fit_atf.sh: Allow for this to complete when bl31.bin is missing"), where producing unusable binaries is preferred in favor of a green automatic builds.

Quoted 7 lines

So lets try to fix this properly, add BUILDBOT config options which could be set on the build bots/CI and the codebase can use this new config option to workaround the issues in more clear manner.

While at it, replace .its generator script with more generic Dust template based approach, which should allow for more code sharing between boards and thus reduce code duplication.

I'm not sure what Dust is but a the original script for making the .itb files was based on python with pyelftools dep and some we're happy with the added python deps it so it was rewritten into the latest shell script. I'm not sure from looking what extra deps the dust templating adds vs other options but it's worth the consideration.

Peter

Quoted 30 lines

Cheers,

Petr

Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403105.html
Ref: https://lists.denx.de/pipermail/u-boot/2020-March/403650.html
Ref: https://lists.denx.de/pipermail/u-boot/2019-December/393556.html
Cc: Harald Seiler <[email protected]>
Cc: Tom Rini <[email protected]>
Cc: Michal Simek <[email protected]>

Petr ?tetiar (3):
Kconfig: add config options for automatic builds
tools: add Dust based .its file templating
sunxi: replace .its file generator with Dust template

 Kconfig                        |   17 +-
 Makefile                       |    8 +
 board/sunxi/mksunxi_fit_atf.sh |   87 -
 board/sunxi/u-boot-its.mk      |   19 +
 board/sunxi/u-boot.its.dust    |   45 +
 include/u-boot-its.mk          |   37 +
 tools/ashes/ashes.py           | 2723 ++++++++++++++++++++++++++++++++
 7 files changed, 2848 insertions(+), 88 deletions(-)
 delete mode 100755 board/sunxi/mksunxi_fit_atf.sh
 create mode 100644 board/sunxi/u-boot-its.mk
 create mode 100644 board/sunxi/u-boot.its.dust
 create mode 100644 include/u-boot-its.mk
 create mode 100755 tools/ashes/ashes.py