Skip to contentSkip to conversation

binman: add support for creating dummy files for external blobs

RFC Superseded Heiko Thiery 4 participants 6 messages Nov 16, 2021 – Nov 25, 2021 Kbuild

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

RFC 1/1 binman: add support for creating dummy files for external blobs

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        | 16 ++++++++++++++++
 tools/binman/entry.py          |  1 +
 tools/binman/etype/blob_ext.py |  1 +
 5 files changed, 21 insertions(+)
Makefile +1 -0
index f911f70344..1a833a1637 100644
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \
 		-a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
 		-a spl-dtb=$(CONFIG_SPL_OF_REAL) \
 		-a tpl-dtb=$(CONFIG_SPL_OF_REAL) \
+		$(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
 		$(BINMAN_$(@F))
 
 OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
tools/binman/cmdline.py +2 -0
index d6156df408..2b29981cb4 100644
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
             help='Configuration file (.dtb) to use')
     build_parser.add_argument('--fake-dtb', action='store_true',
             help='Use fake device tree contents (for testing only)')
+    build_parser.add_argument('--fake-ext-blobs', action='store_true',
+            help='Create fake ext blobs with dummy content (for testing only)')
     build_parser.add_argument('-i', '--image', type=str, action='append',
             help='Image filename to build (if not specified, build all)')
     build_parser.add_argument('-I', '--indir', action='append',
tools/binman/control.py +16 -0
index 0dbcbc28e9..f95aadd1f3 100644
@@ -8,6 +8,7 @@
 from collections import OrderedDict
 import glob
 import os
+import pathlib
 import pkg_resources
 import re
 
@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths,
     return image
 
 
+def PrepareFakeExtBlobs(images):
+    for (_, image) in images.items():
+        for entry in image._entries.values():
+            if entry.allow_dummy:
+                if not pathlib.Path(entry._filename).is_file():
+                    tout.Warning("Missing external blob '%s', fake it" %
+                                 entry._filename)
+                    with open(entry._filename, "wb") as out:
+                        out.truncate(1024)
+
+
 def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded):
     """Prepare the images to be processed and select the device tree
 
@@ -629,6 +641,10 @@ def Binman(args):
 
             images = PrepareImagesAndDtbs(dtb_fname, args.image,
                                           args.update_fdt, use_expanded)
+
+            if args.fake_ext_blobs:
+                PrepareFakeExtBlobs(images)
+
             if args.test_section_timeout:
                 # Set the first image to timeout, used in testThreadTimeout()
                 images[list(images.keys())[0]].test_section_timeout = True
tools/binman/entry.py +1 -0
index 70222718ea..2f5c38d15a 100644
@@ -100,6 +100,7 @@ class Entry(object):
         self.missing = False
         self.external = False
         self.allow_missing = False
+        self.allow_dummy = False
 
     @staticmethod
     def Lookup(node_path, etype, expanded):
tools/binman/etype/blob_ext.py +1 -0
index d6b0ca17c3..85bc9fe6b8 100644
@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob):
     def __init__(self, section, etype, node):
         Entry_blob.__init__(self, section, etype, node)
         self.external = True
+        self.allow_dummy = True
Signature · 1 line
2.30.2
#1 lore raw
Quoted 9 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

Quoted 8 lines
---
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        | 16 ++++++++++++++++
 tools/binman/entry.py          |  1 +
 tools/binman/etype/blob_ext.py |  1 +
 5 files changed, 21 insertions(+)

Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.

Signature · 1 line
Tom
#2 lore raw
Quoted 20 lines · Tom Rini
Quoted 9 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

Quoted 8 lines
---
  Makefile                       |  1 +
  tools/binman/cmdline.py        |  2 ++
  tools/binman/control.py        | 16 ++++++++++++++++
  tools/binman/entry.py          |  1 +
  tools/binman/etype/blob_ext.py |  1 +
  5 files changed, 21 insertions(+)
Quoted 4 lines

Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.

It looks reasonable for me and it is not different as we did before binman reporting warnings that the binary does not work. Fine if thsi can be merged, I will unblock some i.MX boards (Heiko's for example) that I cannot currently merge due to CI failures.

Regards,
Stefano

Signature · 5 lines
=====================================================================
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: [email protected]
=====================================================================
#3 lore raw

Hi Stefano,

Am Di., 23. Nov. 2021 um 16:44 Uhr schrieb Stefano Babic <[email protected]>:

Quoted 27 lines · Tom Rini
Quoted 20 lines · Heiko Thiery
Quoted 10 lines

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

Quoted 8 lines
---
  Makefile                       |  1 +
  tools/binman/cmdline.py        |  2 ++
  tools/binman/control.py        | 16 ++++++++++++++++
  tools/binman/entry.py          |  1 +
  tools/binman/etype/blob_ext.py |  1 +
  5 files changed, 21 insertions(+)
Quoted 4 lines

Since this ends up being a problem for a number of boards, does anyone have comments on this? At first glance, this seems reasonable to me, and a follow-up making CI make use of this would be handy to finish evaluating.

Quoted 4 lines

It looks reasonable for me and it is not different as we did before binman reporting warnings that the binary does not work. Fine if thsi can be merged, I will unblock some i.MX boards (Heiko's for example) that I cannot currently merge due to CI failures.

Before you add my patch for the imx8mq board we need to make a change in buildman to make the binman flag option "--fake-ext-blobs" usable. I haven't done that yet, because I wanted to wait if this patch will be accepted.

Signature · 1 line
Heiko
#4 lore raw
In reply to Heiko Thiery

Hi Heiko,

Quoted 10 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

Quoted 8 lines
---
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        | 16 ++++++++++++++++
 tools/binman/entry.py          |  1 +
 tools/binman/etype/blob_ext.py |  1 +
 5 files changed, 21 insertions(+)

This seems like a reasonable solution to me.

Quoted 39 lines
Makefile +1 -0
index f911f70344..1a833a1637 100644
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \
                -a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
                -a spl-dtb=$(CONFIG_SPL_OF_REAL) \
                -a tpl-dtb=$(CONFIG_SPL_OF_REAL) \
+               $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
                $(BINMAN_$(@F))
 
 OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
tools/binman/cmdline.py +2 -0
index d6156df408..2b29981cb4 100644
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
             help='Configuration file (.dtb) to use')
     build_parser.add_argument('--fake-dtb', action='store_true',
             help='Use fake device tree contents (for testing only)')
+    build_parser.add_argument('--fake-ext-blobs', action='store_true',
+            help='Create fake ext blobs with dummy content (for testing only)')
     build_parser.add_argument('-i', '--image', type=str, action='append',
             help='Image filename to build (if not specified, build all)')
     build_parser.add_argument('-I', '--indir', action='append',
tools/binman/control.py +1 -0
index 0dbcbc28e9..f95aadd1f3 100644
@@ -8,6 +8,7 @@
 from collections import OrderedDict
 import glob
 import os
+import pathlib
 import pkg_resources
 import re
 

tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 405

Quoted 4 lines
tools/binman/control.py +1 -0
@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths,
     return image
 
 
+def PrepareFakeExtBlobs(images):

Please add a function comment

tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 413

Quoted 8 lines
tools/binman/control.py +8 -0
+    for (_, image) in images.items():
+        for entry in image._entries.values():
+            if entry.allow_dummy:
+                if not pathlib.Path(entry._filename).is_file():
+                    tout.Warning("Missing external blob '%s', fake it" %
+                                 entry._filename)
+                    with open(entry._filename, "wb") as out:
+                        out.truncate(1024)

Won't this create files for all external blobs that are missing? Is that the intent? If so, we can do this in Entry_blob.ObtainContents().

Should we use a size of 0? Or will that make some tools complain?

tools/binman/control.py def Binman(args): · line 646

Quoted 12 lines
tools/binman/control.py +5 -0
+
+
 def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded):
     """Prepare the images to be processed and select the device tree
 
@@ -629,6 +641,10 @@ def Binman(args):
 
             images = PrepareImagesAndDtbs(dtb_fname, args.image,
                                           args.update_fdt, use_expanded)
+
+            if args.fake_ext_blobs:
+                PrepareFakeExtBlobs(images)

This should go in ProcessImage(). See the SetAllowMissing() and see if you can follow the same pattern. Basically we tell each section that it should allow missing blobs, then check that in Entry_blot.ObtainContents()

tools/binman/entry.py class Entry(object): · line 103

Quoted 13 lines
tools/binman/control.py +1 -0
+
             if args.test_section_timeout:
                 # Set the first image to timeout, used in testThreadTimeout()
                 images[list(images.keys())[0]].test_section_timeout = True
tools/binman/entry.py +1 -0
index 70222718ea..2f5c38d15a 100644
@@ -100,6 +100,7 @@ class Entry(object):
         self.missing = False
         self.external = False
         self.allow_missing = False
+        self.allow_dummy = False

Please add a comment about this above.

Quoted 13 lines
tools/binman/entry.py +0 -0
     @staticmethod
     def Lookup(node_path, etype, expanded):
tools/binman/etype/blob_ext.py +1 -1
index d6b0ca17c3..85bc9fe6b8 100644
@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob):
     def __init__(self, section, etype, node):
         Entry_blob.__init__(self, section, etype, node)
         self.external = True
+        self.allow_dummy = True
--
Quoted 2 lines

2.30.2

Also please run 'binman test -T' as you need a test for this.

Regards,
Simon

#5 lore raw

Hi Simon,

Am Di., 23. Nov. 2021 um 17:15 Uhr schrieb Simon Glass <[email protected]>:

Quoted 26 lines

Hi Heiko,

Quoted 10 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

Quoted 8 lines
---
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        | 16 ++++++++++++++++
 tools/binman/entry.py          |  1 +
 tools/binman/etype/blob_ext.py |  1 +
 5 files changed, 21 insertions(+)

This seems like a reasonable solution to me.

Quoted 1 line
Quoted 10 lines

Makefile cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ · line 1311

Quoted 10 lines
Makefile +1 -0
index f911f70344..1a833a1637 100644
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \
                -a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
                -a spl-dtb=$(CONFIG_SPL_OF_REAL) \
                -a tpl-dtb=$(CONFIG_SPL_OF_REAL) \
+               $(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
                $(BINMAN_$(@F))

This will only work if we add the BINMAN_FAKE_EXT_BLOBS=y to the make call as argument/option. This means we also have to change the buildman for the CI. Or do you know if it is also possible to use an environment variable to achieve this? Then we could simply set the env var for the CI builds.

currently:
make CROSS_COMPILE=aarch64-linux-gnu- -j4 BINMAN_FAKE_EXT_BLOBS=y

nice to have
export BINMAN_FAKE_EXT_BLOBS=y
make CROSS_COMPILE=aarch64-linux-gnu- -j4

What do you think?

Quoted 44 lines
Quoted 28 lines
Makefile +0 -0
 OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
tools/binman/cmdline.py +2 -0
index d6156df408..2b29981cb4 100644
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
             help='Configuration file (.dtb) to use')
     build_parser.add_argument('--fake-dtb', action='store_true',
             help='Use fake device tree contents (for testing only)')
+    build_parser.add_argument('--fake-ext-blobs', action='store_true',
+            help='Create fake ext blobs with dummy content (for testing only)')
     build_parser.add_argument('-i', '--image', type=str, action='append',
             help='Image filename to build (if not specified, build all)')
     build_parser.add_argument('-I', '--indir', action='append',
tools/binman/control.py +1 -0
index 0dbcbc28e9..f95aadd1f3 100644
@@ -8,6 +8,7 @@
 from collections import OrderedDict
 import glob
 import os
+import pathlib
 import pkg_resources
 import re
 

tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 405

Quoted 4 lines
tools/binman/control.py +1 -0
@@ -401,6 +402,17 @@ def ReplaceEntries(image_fname, input_fname, indir, entry_paths,
     return image
 
 
+def PrepareFakeExtBlobs(images):

Please add a function comment

tools/binman/control.py def ReplaceEntries(image_fname, input_fname, indir, entry_paths, · line 413

Quoted 8 lines
tools/binman/control.py +8 -0
+    for (_, image) in images.items():
+        for entry in image._entries.values():
+            if entry.allow_dummy:
+                if not pathlib.Path(entry._filename).is_file():
+                    tout.Warning("Missing external blob '%s', fake it" %
+                                 entry._filename)
+                    with open(entry._filename, "wb") as out:
+                        out.truncate(1024)
Quoted 2 lines

Won't this create files for all external blobs that are missing? Is that the intent? If so, we can do this in Entry_blob.ObtainContents().

Yes this will create the dummy files for all external blobs.

Quoted 2 lines

Should we use a size of 0? Or will that make some tools complain?

mkimage will complain about files that are too small.

Quoted 7 lines
Quoted 7 lines
tools/binman/control.py +2 -0
+
+
 def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded):
     """Prepare the images to be processed and select the device tree
 
@@ -629,6 +641,10 @@ def Binman(args):
 
Quoted 10 lines

tools/binman/control.py def Binman(args): · line 646

Quoted 5 lines
tools/binman/control.py +3 -0
             images = PrepareImagesAndDtbs(dtb_fname, args.image,
                                           args.update_fdt, use_expanded)
+
+            if args.fake_ext_blobs:
+                PrepareFakeExtBlobs(images)

This should go in ProcessImage(). See the SetAllowMissing() and see if you can follow the same pattern. Basically we tell each section that it should allow missing blobs, then check that in Entry_blot.ObtainContents()

I changed this to the proposed. I had to add this also to the 'Entry_mkimage' when I add the external blob inside this node.

spl {
filename = "spl.bin";

  mkimage {
    args = "-n spl/u-boot-spl.cfgout -T imx8mimage -e 0x7e1000";
    blob {
      filename = "u-boot-spl-ddr.bin";
    };
    hdmi_fw: blob-ext@5 {
      filename = "signed_hdmi_imx8m.bin";
    };

};
};

Quoted 7 lines
Quoted 7 lines
tools/binman/control.py +1 -0
+
             if args.test_section_timeout:
                 # Set the first image to timeout, used in testThreadTimeout()
                 images[list(images.keys())[0]].test_section_timeout = True
tools/binman/entry.py +0 -1
index 70222718ea..2f5c38d15a 100644
--- a/tools/binman/entry.py
Quoted 8 lines

tools/binman/entry.py class Entry(object): · line 103

Quoted 6 lines

+++ b/tools/binman/entry.py

tools/binman/entry.py +1 -0
@@ -100,6 +100,7 @@ class Entry(object):
         self.missing = False
         self.external = False
         self.allow_missing = False
+        self.allow_dummy = False

Please add a comment about this above.

Ok.

Quoted 18 lines
Quoted 14 lines
tools/binman/entry.py +0 -0
     @staticmethod
     def Lookup(node_path, etype, expanded):
tools/binman/etype/blob_ext.py +1 -1
index d6b0ca17c3..85bc9fe6b8 100644
@@ -26,3 +26,4 @@ class Entry_blob_ext(Entry_blob):
     def __init__(self, section, etype, node):
         Entry_blob.__init__(self, section, etype, node)
         self.external = True
+        self.allow_dummy = True
--
Quoted 2 lines

2.30.2

Also please run 'binman test -T' as you need a test for this.

Currently I'm struggling doing that.

Signature · 1 line
Heiko