binman: add support for creating dummy files for external blobs
RFC 1/1 binman: add support for creating dummy files for external blobs
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.
--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)
Makefile +1 -0
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ -a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \ -a spl-dtb=$(CONFIG_SPL_OF_REAL) \ -a tpl-dtb=$(CONFIG_SPL_OF_REAL) \ + $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \ $(BINMAN_$(@F)) OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
tools/binman/cmdline.py +2 -0
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.''' help='Configuration file (.dtb) to use') build_parser.add_argument('--fake-dtb', action='store_true', help='Use fake device tree contents (for testing only)') + build_parser.add_argument('--fake-ext-blobs', action='store_true', + help='Create fake ext blobs with dummy content (for testing only)') build_parser.add_argument('-i', '--image', type=str, action='append', help='Image filename to build (if not specified, build all)') build_parser.add_argument('-I', '--indir', action='append',
tools/binman/control.py +16 -0
@@ -8,6 +8,7 @@ from collections import OrderedDict import glob import os +import pathlib import pkg_resources import re
@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths, return image +def PrepareFakeExtBlobs(images): + for (_, image) in images.items(): + for entry in image._entries.values(): + if entry.allow_dummy: + if not pathlib.Path(entry._filename).is_file(): + tout.Warning("Missing external blob '%s', fake it" % + entry._filename) + with open(entry._filename, "wb") as out: + out.truncate(1024) + + def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): """Prepare the images to be processed and select the device tree
@@ -629,6 +641,10 @@ def Binman(args): images = PrepareImagesAndDtbs(dtb_fname, args.image, args.update_fdt, use_expanded) + + if args.fake_ext_blobs: + PrepareFakeExtBlobs(images) + if args.test_section_timeout: # Set the first image to timeout, used in testThreadTimeout() images[list(images.keys())[0]].test_section_timeout = True
tools/binman/entry.py +1 -0
@@ -100,6 +100,7 @@ class Entry(object): self.missing = False self.external = False self.allow_missing = False + self.allow_dummy = False @staticmethod def Lookup(node_path, etype, expanded):
tools/binman/etype/blob_ext.py +1 -0
@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob): def __init__(self, section, etype, node): Entry_blob.__init__(self, section, etype, node) self.external = True + self.allow_dummy = True
Signature · 1 line
2.30.2
Quoted 9 lines · Heiko Thiery
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile optionBINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.
Quoted 8 lines
Signed-off-by Heiko Thiery <[email protected]>--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)
Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.
Signature · 1 line
Tom
Quoted 20 lines · Tom Rini
Quoted 9 lines · Heiko Thiery
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile optionBINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.Quoted 8 lines
Signed-off-by Heiko Thiery <[email protected]>--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)
Quoted 4 lines
Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.
It looks reasonable for me and it is not different as we did before binman reporting warnings that the binary does not work. Fine if thsi can be merged, I will unblock some i.MX boards (Heiko's for example) that I cannot currently merge due to CI failures.
Regards,
Stefano
Signature · 5 lines
===================================================================== DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: [email protected] =====================================================================
Hi Stefano,
Am Di., 23. Nov. 2021 um 16:44 Uhr schrieb Stefano Babic <[email protected]>:
Quoted 27 lines · Tom Rini
Quoted 20 lines · Heiko Thiery
Quoted 10 lines
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile optionBINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.Quoted 8 lines
Signed-off-by Heiko Thiery <[email protected]>--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)Quoted 4 lines
Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.
Quoted 4 lines
It looks reasonable for me and it is not different as we did before binman reporting warnings that the binary does not work. Fine if thsi can be merged, I will unblock some i.MX boards (Heiko's for example) that I cannot currently merge due to CI failures.
Before you add my patch for the imx8mq board we need to make a change in buildman to make the binman flag option "--fake-ext-blobs" usable. I haven't done that yet, because I wanted to wait if this patch will be accepted.
Signature · 1 line
Heiko
Hi Heiko,
Quoted 10 lines · Heiko Thiery
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile optionBINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.
Quoted 8 lines
Signed-off-by Heiko Thiery <[email protected]>--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)
This seems like a reasonable solution to me.
Quoted 39 lines
Makefile +1
-0@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ -a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \ -a spl-dtb=$(CONFIG_SPL_OF_REAL) \ -a tpl-dtb=$(CONFIG_SPL_OF_REAL) \ + $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \ $(BINMAN_$(@F)) OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihextools/binman/cmdline.py +2
-0@@ -52,6 +52,8 @@ controlled by a description in the board device tree.''' help='Configuration file (.dtb) to use') build_parser.add_argument('--fake-dtb', action='store_true', help='Use fake device tree contents (for testing only)') + build_parser.add_argument('--fake-ext-blobs', action='store_true', + help='Create fake ext blobs with dummy content (for testing only)') build_parser.add_argument('-i', '--image', type=str, action='append', help='Image filename to build (if not specified, build all)') build_parser.add_argument('-I', '--indir', action='append',tools/binman/control.py +1
-0@@ -8,6 +8,7 @@ from collections import OrderedDict import glob import os +import pathlib import pkg_resources import re
tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 405
Quoted 4 lines
tools/binman/control.py +1
-0@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths, return image +def PrepareFakeExtBlobs(images):
Please add a function comment
tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 413
Quoted 8 lines
tools/binman/control.py +8
-0+ for (_, image) in images.items(): + for entry in image._entries.values(): + if entry.allow_dummy: + if not pathlib.Path(entry._filename).is_file(): + tout.Warning("Missing external blob '%s', fake it" % + entry._filename) + with open(entry._filename, "wb") as out: + out.truncate(1024)
Won't this create files for all external blobs that are missing? Is that the intent? If so, we can do this in Entry_blob.ObtainContents().
Should we use a size of 0? Or will that make some tools complain?
tools/binman/control.py def Binman(args): · line 646
Quoted 12 lines
tools/binman/control.py +5
-0+ + def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): """Prepare the images to be processed and select the device tree@@ -629,6 +641,10 @@ def Binman(args): images = PrepareImagesAndDtbs(dtb_fname, args.image, args.update_fdt, use_expanded) + + if args.fake_ext_blobs: + PrepareFakeExtBlobs(images)
This should go in ProcessImage(). See the SetAllowMissing() and see if you can follow the same pattern. Basically we tell each section that it should allow missing blobs, then check that in Entry_blot.ObtainContents()
tools/binman/entry.py class Entry(object): · line 103
Quoted 13 lines
tools/binman/control.py +1
-0+ if args.test_section_timeout: # Set the first image to timeout, used in testThreadTimeout() images[list(images.keys())[0]].test_section_timeout = Truetools/binman/entry.py +1
-0@@ -100,6 +100,7 @@ class Entry(object): self.missing = False self.external = False self.allow_missing = False + self.allow_dummy = False
Please add a comment about this above.
Quoted 13 lines
tools/binman/entry.py +0
-0@staticmethod def Lookup(node_path, etype, expanded):tools/binman/etype/blob_ext.py +1
-1@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob): def __init__(self, section, etype, node): Entry_blob.__init__(self, section, etype, node) self.external = True + self.allow_dummy = True --
Quoted 2 lines
2.30.2
Also please run 'binman test -T' as you need a test for this.
Regards,
Simon
Hi Simon,
Am Di., 23. Nov. 2021 um 17:15 Uhr schrieb Simon Glass <[email protected]>:
Quoted 26 lines
Hi Heiko,
Quoted 10 lines · Heiko Thiery
While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile optionBINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.Quoted 8 lines
Signed-off-by Heiko Thiery <[email protected]>--- Makefile | 1 + tools/binman/cmdline.py | 2 ++ tools/binman/control.py | 16 ++++++++++++++++ tools/binman/entry.py | 1 + tools/binman/etype/blob_ext.py | 1 + 5 files changed, 21 insertions(+)This seems like a reasonable solution to me.
Quoted 1 line
Quoted 10 lines
Makefile cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ · line 1311
Quoted 10 lines
Makefile +1
-0@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ -a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \ -a spl-dtb=$(CONFIG_SPL_OF_REAL) \ -a tpl-dtb=$(CONFIG_SPL_OF_REAL) \ + $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \ $(BINMAN_$(@F))
This will only work if we add the BINMAN_FAKE_EXT_BLOBS=y to the make call as argument/option. This means we also have to change the buildman for the CI. Or do you know if it is also possible to use an environment variable to achieve this? Then we could simply set the env var for the CI builds.
currently:
make CROSS_COMPILE=aarch64-linux-gnu- -j4 BINMAN_FAKE_EXT_BLOBS=y
nice to have
export BINMAN_FAKE_EXT_BLOBS=y
make CROSS_COMPILE=aarch64-linux-gnu- -j4
What do you think?
Quoted 44 lines
Quoted 28 lines
Makefile +0
-0OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihextools/binman/cmdline.py +2
-0@@ -52,6 +52,8 @@ controlled by a description in the board device tree.''' help='Configuration file (.dtb) to use') build_parser.add_argument('--fake-dtb', action='store_true', help='Use fake device tree contents (for testing only)') + build_parser.add_argument('--fake-ext-blobs', action='store_true', + help='Create fake ext blobs with dummy content (for testing only)') build_parser.add_argument('-i', '--image', type=str, action='append', help='Image filename to build (if not specified, build all)') build_parser.add_argument('-I', '--indir', action='append',tools/binman/control.py +1
-0@@ -8,6 +8,7 @@ from collections import OrderedDict import glob import os +import pathlib import pkg_resources import retools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 405
Quoted 4 lines
tools/binman/control.py +1
-0@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths, return image +def PrepareFakeExtBlobs(images):Please add a function comment
tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 413
Quoted 8 lines
tools/binman/control.py +8
-0+ for (_, image) in images.items(): + for entry in image._entries.values(): + if entry.allow_dummy: + if not pathlib.Path(entry._filename).is_file(): + tout.Warning("Missing external blob '%s', fake it" % + entry._filename) + with open(entry._filename, "wb") as out: + out.truncate(1024)
Quoted 2 lines
Won't this create files for all external blobs that are missing? Is that the intent? If so, we can do this in
Entry_blob.ObtainContents().
Yes this will create the dummy files for all external blobs.
Quoted 2 lines
Should we use a size of 0? Or will that make some tools complain?
mkimage will complain about files that are too small.
Quoted 7 lines
Quoted 7 lines
tools/binman/control.py +2
-0+ + def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): """Prepare the images to be processed and select the device tree@@ -629,6 +641,10 @@ def Binman(args):
Quoted 10 lines
tools/binman/control.py def Binman(args): · line 646
Quoted 5 lines
tools/binman/control.py +3
-0images = PrepareImagesAndDtbs(dtb_fname, args.image, args.update_fdt, use_expanded) + + if args.fake_ext_blobs: + PrepareFakeExtBlobs(images)This should go in
ProcessImage(). See theSetAllowMissing()and see if you can follow the same pattern. Basically we tell each section that it should allow missing blobs, then check that inEntry_blot.ObtainContents()
I changed this to the proposed. I had to add this also to the 'Entry_mkimage' when I add the external blob inside this node.
spl {
filename = "spl.bin";
mkimage {
args = "-n spl/u-boot-spl.cfgout -T imx8mimage -e 0x7e1000"; blob {
filename = "u-boot-spl-ddr.bin";
};
hdmi_fw: blob-ext@5 {
filename = "signed_hdmi_imx8m.bin";
}; };
};
Quoted 7 lines
Quoted 7 lines
tools/binman/control.py +1
-0+ if args.test_section_timeout: # Set the first image to timeout, used in testThreadTimeout() images[list(images.keys())[0]].test_section_timeout = Truetools/binman/entry.py +0
-1--- a/tools/binman/entry.py
Quoted 8 lines
tools/binman/entry.py class Entry(object): · line 103
Quoted 6 lines
+++
b/tools/binman/entry.pytools/binman/entry.py +1
-0@@ -100,6 +100,7 @@ class Entry(object): self.missing = False self.external = False self.allow_missing = False + self.allow_dummy = FalsePlease add a comment about this above.
Ok.
Quoted 18 lines
Quoted 14 lines
tools/binman/entry.py +0
-0@staticmethod def Lookup(node_path, etype, expanded):tools/binman/etype/blob_ext.py +1
-1@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob): def __init__(self, section, etype, node): Entry_blob.__init__(self, section, etype, node) self.external = True + self.allow_dummy = True --Quoted 2 lines
2.30.2
Also please run 'binman test -T' as you need a test for this.
Currently I'm struggling doing that.
Signature · 1 line
Heiko