Skip to contentSkip to conversation

binman: add support for creating dummy files for external blobs

RFC PATCH Superseded Heiko Thiery 2 participants 6 messages Nov 29, 2021 – Jan 5, 2022 Kbuild

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH v2 1/1 binman: add support for creating dummy files for external blobs

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)
Makefile +1 -0
index f911f70344..1a833a1637 100644
@@ -1307,6 +1307,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \
 		-a tpl-bss-pad=$(if $(CONFIG_TPL_SEPARATE_BSS),,1) \
 		-a spl-dtb=$(CONFIG_SPL_OF_REAL) \
 		-a tpl-dtb=$(CONFIG_SPL_OF_REAL) \
+		$(if $(BINMAN_FAKE_EXT_BLOBS),--fake-ext-blobs) \
 		$(BINMAN_$(@F))
 
 OBJCOPYFLAGS_u-boot.ldr.hex := -I binary -O ihex
tools/binman/cmdline.py +2 -0
index d6156df408..2b29981cb4 100644
@@ -52,6 +52,8 @@ controlled by a description in the board device tree.'''
             help='Configuration file (.dtb) to use')
     build_parser.add_argument('--fake-dtb', action='store_true',
             help='Use fake device tree contents (for testing only)')
+    build_parser.add_argument('--fake-ext-blobs', action='store_true',
+            help='Create fake ext blobs with dummy content (for testing only)')
     build_parser.add_argument('-i', '--image', type=str, action='append',
             help='Image filename to build (if not specified, build all)')
     build_parser.add_argument('-I', '--indir', action='append',
tools/binman/control.py +7 -2
index 0dbcbc28e9..fa034bddef 100644
@@ -479,7 +479,8 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded):
 
 
 def ProcessImage(image, update_fdt, write_map, get_contents=True,
-                 allow_resize=True, allow_missing=False):
+                 allow_resize=True, allow_missing=False,
+                 allow_fake_blobs=False):
     """Perform all steps for this image, including checking and # writing it.
 
     This means that errors found with a later image will be reported after
@@ -495,12 +496,14 @@ def ProcessImage(image, update_fdt, write_map, get_contents=True,
         allow_resize: True to allow entries to change size (this does a re-pack
             of the entries), False to raise an exception
         allow_missing: Allow blob_ext objects to be missing
+        allow_fake_blobs: Allow blob_ext objects to be faked with dummy files
 
     Returns:
         True if one or more external blobs are missing, False if all are present
     """
     if get_contents:
         image.SetAllowMissing(allow_missing)
+        image.SetAllowFakeBlob(allow_fake_blobs)
         image.GetEntryContents()
     image.GetEntryOffsets()
 
@@ -629,13 +632,15 @@ def Binman(args):
 
             images = PrepareImagesAndDtbs(dtb_fname, args.image,
                                           args.update_fdt, use_expanded)
+
             if args.test_section_timeout:
                 # Set the first image to timeout, used in testThreadTimeout()
                 images[list(images.keys())[0]].test_section_timeout = True
             missing = False
             for image in images.values():
                 missing |= ProcessImage(image, args.update_fdt, args.map,
-                                        allow_missing=args.allow_missing)
+                                        allow_missing=args.allow_missing,
+                                        allow_fake_blobs=args.fake_ext_blobs)
 
             # Write the updated FDTs to our output files
             for dtb_item in state.GetAllFdts():
tools/binman/entry.py +11 -0
index 70222718ea..9aa4359be1 100644
@@ -70,6 +70,8 @@ class Entry(object):
         missing: True if this entry is missing its contents
         allow_missing: Allow children of this entry to be missing (used by
             subclasses such as Entry_section)
+        allow_fake: Allow creating a dummy fake file if the blob file is not
+            available. This is mainly used for testing.
         external: True if this entry contains an external binary blob
     """
     def __init__(self, section, etype, node, name_prefix=''):
@@ -100,6 +102,7 @@ class Entry(object):
         self.missing = False
         self.external = False
         self.allow_missing = False
+        self.allow_fake = False
 
     @staticmethod
     def Lookup(node_path, etype, expanded):
@@ -898,6 +901,14 @@ features to produce new behaviours.
         # This is meaningless for anything other than sections
         pass
 
+    def SetAllowFakeBlob(self, allow_fake):
+        """Set whether a section allows to create a fake blob
+
+        Args:
+            allow_fake: True if allowed, False if not allowed
+        """
+        pass
+
     def CheckMissing(self, missing_list):
         """Check if any entries in this section have missing external blobs
 
tools/binman/etype/blob.py +7 -0
index fae86ca3ec..0c7469409f 100644
@@ -5,6 +5,8 @@
 # Entry-type module for blobs, which are binary objects read from files
 #
 
+import pathlib
+
 from binman.entry import Entry
 from binman import state
 from dtoc import fdt_util
@@ -36,6 +38,11 @@ class Entry_blob(Entry):
         self._filename = fdt_util.GetString(self._node, 'filename', self.etype)
 
     def ObtainContents(self):
+        if self.allow_fake and not pathlib.Path(self._filename).is_file():
+            tout.Warning("Missing blob '%s', fake it" % self._filename)
+            with open(self._filename, "wb") as out:
+                out.truncate(1024)
+
         self._filename = self.GetDefaultFilename()
         self._pathname = tools.GetInputFilename(self._filename,
             self.external and self.section.GetAllowMissing())
tools/binman/etype/blob_ext.py +8 -0
index d6b0ca17c3..fba6271de2 100644
@@ -26,3 +26,11 @@ class Entry_blob_ext(Entry_blob):
     def __init__(self, section, etype, node):
         Entry_blob.__init__(self, section, etype, node)
         self.external = True
+
+    def SetAllowFakeBlob(self, allow_fake):
+        """Set whether the entry allows to create a fake blob
+
+        Args:
+            allow_fake_blob: True if allowed, False if not allowed
+        """
+        self.allow_fake = allow_fake
tools/binman/etype/mkimage.py +9 -0
index e49977522e..a797d9314e 100644
@@ -61,3 +61,12 @@ class Entry_mkimage(Entry):
             entry = Entry.Create(self, node)
             entry.ReadNode()
             self._mkimage_entries[entry.name] = entry
+
+    def SetAllowFakeBlob(self, allow_fake):
+        """Set whether the sub nodes allows to create a fake blob
+
+        Args:
+            allow_fake: True if allowed, False if not allowed
+        """
+        for entry in self._mkimage_entries.values():
+            entry.SetAllowFakeBlob(allow_fake)
tools/binman/etype/section.py +9 -0
index e2949fc916..52cbbe0f73 100644
@@ -689,6 +689,15 @@ class Entry_section(Entry):
         for entry in self._entries.values():
             entry.SetAllowMissing(allow_missing)
 
+    def SetAllowFakeBlob(self, allow_fake):
+        """Set whether a section allows to create a fake blob
+
+        Args:
+            allow_fake_blob: True if allowed, False if not allowed
+        """
+        for entry in self._entries.values():
+            entry.SetAllowFakeBlob(allow_fake)
+
     def CheckMissing(self, missing_list):
         """Check if any entries in this section have missing external blobs
 
Signature · 1 line
2.20.1
#1 lore raw

Hi Heiko,

Quoted 24 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

Quoted 9 lines
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)

This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked.

Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template.

Regards,
Simon

#2 lore raw

Hi Heiko,

Quoted 45 lines · Simon Glass

Hi Heiko,

Quoted 24 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

Quoted 9 lines
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)

This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked.

Quoted 3 lines

Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template.

Any word on this? I'd like to get this feature in and take a look at missing vendor tools, too.

Also I think your feature should be on by default.

Regards,
Simon

#3 lore raw

Hi Simon,

Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass <[email protected]>:

Quoted 49 lines

Hi Heiko,

Quoted 45 lines · Simon Glass

Hi Heiko,

Quoted 24 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

Quoted 9 lines
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)

This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked.

Quoted 6 lines
Quoted 3 lines

Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template.

Any word on this? I'd like to get this feature in and take a look at missing vendor tools, too.

I have a new version available with your comments included. But the tests are still missing. In the past days I had no motivation to work on that. I will try to do this in the next days.

Quoted 1 line

Also I think your feature should be on by default.

Signature · 1 line
Heiko
#4 lore raw

Hi Heiko,

Quoted 58 lines · Heiko Thiery

Hi Simon,

Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass <[email protected]>:

Quoted 53 lines

Hi Heiko,

Quoted 48 lines · Simon Glass

Hi Heiko,

Quoted 24 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

Quoted 9 lines
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)

This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked.

Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template.

Quoted 6 lines
Quoted 2 lines

Any word on this? I'd like to get this feature in and take a look at missing vendor tools, too.

I have a new version available with your comments included. But the tests are still missing. In the past days I had no motivation to work on that. I will try to do this in the next days.

OK thanks. If you need help with the test-coverage part, let me know.

Quoted 2 lines
Quoted 2 lines

Also I think your feature should be on by default.

Regards,
Simon

#5 lore raw

Hi Simon,

Am Mi., 5. Jan. 2022 um 15:05 Uhr schrieb Simon Glass <[email protected]>:

Quoted 62 lines

Hi Heiko,

Quoted 58 lines · Heiko Thiery

Hi Simon,

Am So., 2. Jan. 2022 um 18:15 Uhr schrieb Simon Glass <[email protected]>:

Quoted 53 lines

Hi Heiko,

Quoted 48 lines · Simon Glass

Hi Heiko,

Quoted 24 lines · Heiko Thiery

While converting to binman for an imx8mq board, it has been found that building in the u-boot CI fails. This is because an imx8mq requires an external binary (signed_hdmi_imx8m.bin). If this file cannot be found mkimage fails.
To be able to build this board in the u-boot CI a binman option (--fake-ext-blobs) is introduced that can be switched on via the u-boot makefile option BINMAN_FAKE_EXT_BLOBS. With that the needed dummy files are created.

---
v2:
- pass allow_fake_blobs to ProcessImage()
- set AllowAllowFakeBlob() to images/entries
- create fake blob in Entry_blot.ObtainContents() when file is missing and creation is allowed

still missing:
- unittest
- option to set BINMAN_FAKE_EXT_BLOBS in Makefile via environment variable. With that we could simply set this env variable in the CI (gitlab-ci.yml) with adding support to buildman.

Quoted 9 lines
 Makefile                       |  1 +
 tools/binman/cmdline.py        |  2 ++
 tools/binman/control.py        |  9 +++++++--
 tools/binman/entry.py          | 11 +++++++++++
 tools/binman/etype/blob.py     |  7 +++++++
 tools/binman/etype/blob_ext.py |  8 ++++++++
 tools/binman/etype/mkimage.py  |  9 +++++++++
 tools/binman/etype/section.py  |  9 +++++++++
 8 files changed, 54 insertions(+), 2 deletions(-)

This looks good to me! The only thing is that instead of the warning you should just print a single line at the end saying which blobs were faked. See missing_list in ProcessImage() for how that could work. You can set self.fake_blob in your blob.ObtainContents() and then have a similar thing to CheckMissing() to actually collect the list of entries which were faked.

Also, for the real version can you please add a test (so 'binman test -T' stays at 100% test coverage) and some docs in binman.rst ? You can use testMissingBlob() as a template.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Any word on this? I'd like to get this feature in and take a look at missing vendor tools, too.

I have a new version available with your comments included. But the tests are still missing. In the past days I had no motivation to work on that. I will try to do this in the next days.

OK thanks. If you need help with the test-coverage part, let me know.

I sent a new version about an hour ago with unittest included.

Signature · 1 line
Heiko