Skip to contentSkip to conversation

binman: rockchip: Migrate from rockchip SPL_FIT_GENERATOR script

PATCH Superseded Simon Glass 3 participants 74 messages Feb 23, 2022 – Mar 14, 2022

Subsystems (4)

v2 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v2 cover binman: rockchip: Migrate from rockchip SPL_FIT_GENERATOR script

At present rockchip 64-bit boards make use of a FIT-generator script written in Python. The script supports splitting an ELF file into several 'loadable' nodes in the FIT. Binman does not current support this feature.

This series adds binman support for ELF splitting. This works by adding a new 'fit,operation' property to the FIT subnodes, allowing this new way of generating nodes.

Some other fixes and improvements are needed along the way.

A new, common binman description is added for 64-bit boards which includes the required u-boot.itb file.

The existing script is removed, so that only a few zynq boards are now using a SPL_FIT_GENERATOR script:

avnet_ultrazedev_cc_v1_0_ultrazedev_som_v1_0
xilinx_zynqmp_virt

Migration of those is hopefully in progress.

Note however that tools/k3_fit_atf.sh remains, used by a few boards that enable CONFIG_TI_SECURE_DEVICE so this series is copied there too:

    am335x_hs_evm
    am335x_hs_evm_uart
    am43xx_hs_evm
    am57xx_hs_evm
    am57xx_hs_evm_usb
    am65x_hs_evm_a53
    am65x_hs_evm_r5
    dra7xx_hs_evm
    dra7xx_hs_evm_usb
    j721e_hs_evm_a72
    j721e_hs_evm_r5
    k2e_hs_evm
    k2g_hs_evm
    k2hk_hs_evm
    k2l_hs_evm

Ivan Mikhaylov has sent a patch to help with these, but I need to take a look at the testing side. In any case they should really be using binman for the image generation.

Changes in v2:
- Add new patch to make GetArgs() more flexible
- Add new patch to remove remove_defconfig()
- Add new patch to use re.fullmatch() to avoid extra check
- Add new patch to correct Kconfig help for TPL_BINMAN_SYMBOLS
- Add new patch to tidy up implementaiton of AddStringList()
- Add new patch to rename load_segments() and module failure
- Add new patch to tweak collect_contents_to_file() and docs
- Add patch to rename ExpandToLimit to extend_to_limit
- Add patch to rename ExpandEntries to gen_entries
- Add new patch to refactor fit to generate output at the end
- Add a patch to change how faked blobs are created
- Add a patch to make fake blobs zero-sized by default
- Add a patch to allow mkimage to use a non-zero fake-blob size
- Rewrite this to use the new FIT entry-type implementation
- Rename op-tee to tee-os
- Rename op-tee to tee-os
- Drop use of .itb2
- Drop patches previously applied
- Add various suggestions from Alper Nebi Yasak
- Add patches to refactor binman's FIT support

Simon Glass (25):
dtoc: Make GetArgs() more flexible
moveconfig: Remove remove_defconfig()
moveconfig: Use re.fullmatch() to avoid extra check
spl: Correct Kconfig help for TPL_BINMAN_SYMBOLS
dtoc: Tidy up implementaiton of AddStringList()
elf: Rename load_segments() and module failure
binman: Tweak collect_contents_to_file() and docs
binman: Rename ExpandToLimit to extend_to_limit
binman: Rename ExpandEntries to gen_entries
binman: Refactor fit to generate output at the end
binman: Rename tools parameter to btools
binman: Change how faked blobs are created
binman: Make fake blobs zero-sized by default
binman: Allow mkimage to use a non-zero fake-blob size
binman: Read the fit entries only once
binman: Fix some pylint warnings in fit
binman: Add a consistent way to report errors with fit
binman: Update fit to use node instead of subnode
binman: Keep a separate list of entries for fit
binman: Support splitting an ELF file into multiple nodes
rockchip: evb-rk3288: Drop raw-image support
rockchip: Include binman script in 64-bit boards
rockchip: Support building the all output files in binman
rockchip: Convert all boards to use binman
rockchip: Drop the FIT generator script

 Makefile                                     |  42 +-
 arch/arm/dts/px30-u-boot.dtsi                |   2 +
 arch/arm/dts/rk3308-u-boot.dtsi              |   2 +
 arch/arm/dts/rk3326-odroid-go2-u-boot.dtsi   |   2 +
 arch/arm/dts/rk3328-u-boot.dtsi              |   2 +
 arch/arm/dts/rk3368-u-boot.dtsi              |   1 +
 arch/arm/dts/rk3399-u-boot.dtsi              |   5 +-
 arch/arm/dts/rk3568-u-boot.dtsi              |   2 +
 arch/arm/dts/rockchip-u-boot.dtsi            |  84 +++-
 arch/arm/mach-rockchip/Kconfig               |   6 +
 arch/arm/mach-rockchip/make_fit_atf.py       | 240 ---------
 boot/Kconfig                                 |   3 +-
 common/spl/Kconfig                           |   6 +-
 configs/evb-rk3288_defconfig                 |   1 +
 tools/binman/binman.rst                      |  30 +-
 tools/binman/control.py                      |   2 +-
 tools/binman/elf.py                          |   6 +-
 tools/binman/elf_test.py                     |  18 +-
 tools/binman/entries.rst                     | 146 ++++++
 tools/binman/entry.py                        |  49 +-
 tools/binman/etype/_testing.py               |   2 +-
 tools/binman/etype/blob.py                   |  10 +-
 tools/binman/etype/blob_ext_list.py          |   2 +-
 tools/binman/etype/blob_phase.py             |   2 +-
 tools/binman/etype/files.py                  |   2 +-
 tools/binman/etype/fit.py                    | 501 ++++++++++++++-----
 tools/binman/etype/gbb.py                    |   4 +-
 tools/binman/etype/intel_ifwi.py             |   4 +-
 tools/binman/etype/mkimage.py                |  19 +-
 tools/binman/etype/section.py                |  22 +-
 tools/binman/etype/vblock.py                 |   4 +-
 tools/binman/ftest.py                        | 180 ++++++-
 tools/binman/test/088_expand_size.dts        |   8 +-
 tools/binman/test/089_expand_size_bad.dts    |   2 +-
 tools/binman/test/224_fit_bad_oper.dts       |   2 -
 tools/binman/test/225_expand_size_bad.dts    |  10 +
 tools/binman/test/226_fit_split_elf.dts      |  67 +++
 tools/binman/test/227_fit_bad_dir.dts        |   9 +
 tools/binman/test/228_fit_bad_dir_config.dts |   9 +
 tools/binman/test/229_mkimage_missing.dts    |  18 +
 tools/dtoc/fdt.py                            |   4 +-
 tools/dtoc/fdt_util.py                       |   8 +-
 tools/dtoc/test/dtoc_test_simple.dts         |   2 +
 tools/dtoc/test_fdt.py                       |  12 +-
 tools/moveconfig.py                          |  17 +-
 45 files changed, 1060 insertions(+), 509 deletions(-)
 delete mode 100755 arch/arm/mach-rockchip/make_fit_atf.py
 create mode 100644 tools/binman/test/225_expand_size_bad.dts
 create mode 100644 tools/binman/test/226_fit_split_elf.dts
 create mode 100644 tools/binman/test/227_fit_bad_dir.dts
 create mode 100644 tools/binman/test/228_fit_bad_dir_config.dts
 create mode 100644 tools/binman/test/229_mkimage_missing.dts
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#1 lore raw

PATCH v2 1/25 dtoc: Make GetArgs() more flexible

At present it is not possible to have arguments whic include spaces. Update the function to only split the args if the property is a single string. This is a bit inconsistent, but might still be useful.

---

Changes in v2:
- Add new patch to make GetArgs() more flexible

 tools/dtoc/fdt_util.py               | 8 ++++++--
 tools/dtoc/test/dtoc_test_simple.dts | 2 ++
 tools/dtoc/test_fdt.py               | 6 +++++-
 3 files changed, 13 insertions(+), 3 deletions(-)
tools/dtoc/fdt_util.py +6 -2
index c82e7747aa..57550624bb 100644
@@ -192,8 +192,12 @@ def GetArgs(node, propname):
         value = GetStringList(node, propname)
     else:
         value = []
-    lists = [v.split() for v in value]
-    args = [x for l in lists for x in l]
+    if not value:
+        args = []
+    elif len(value) == 1:
+        args = value[0].split()
+    else:
+        args = value
     return args
 
 def GetBool(node, propname, default=False):
tools/dtoc/test/dtoc_test_simple.dts +2 -0
index 2d321fb034..aef07efeae 100644
@@ -63,5 +63,7 @@
 	orig-node {
 		orig = <1 23 4>;
 		args = "-n first", "second", "-p", "123,456", "-x";
+		args2 = "a space", "there";
+		args3 = "-n first second -p 123,456 -x";
 	};
 };
tools/dtoc/test_fdt.py +5 -1
index 576d65b97e..c2013b9289 100755
@@ -659,8 +659,12 @@ class TestFdtUtil(unittest.TestCase):
             ['multi-word', 'message'],
             fdt_util.GetArgs(self.node, 'stringarray'))
         self.assertEqual([], fdt_util.GetArgs(self.node, 'boolval'))
-        self.assertEqual(['-n', 'first', 'second', '-p', '123,456', '-x'],
+        self.assertEqual(['-n first', 'second', '-p', '123,456', '-x'],
                          fdt_util.GetArgs(node, 'args'))
+        self.assertEqual(['a space', 'there'],
+                         fdt_util.GetArgs(node, 'args2'))
+        self.assertEqual(['-n', 'first', 'second', '-p', '123,456', '-x'],
+                         fdt_util.GetArgs(node, 'args3'))
         with self.assertRaises(ValueError) as exc:
             fdt_util.GetArgs(self.node, 'missing')
         self.assertIn(
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 1/25
#2 lore raw

Re: [PATCH v2 01/25] dtoc: Make GetArgs() more flexible

Quoted 1 line · Simon Glass

At present it is not possible to have arguments whic include spaces.

whic -> which

Quoted 7 lines

Update the function to only split the args if the property is a single string. This is a bit inconsistent, but might still be useful.

---

Quoted 7 lines

Changes in v2:
- Add new patch to make GetArgs() more flexible

 tools/dtoc/fdt_util.py               | 8 ++++++--
 tools/dtoc/test/dtoc_test_simple.dts | 2 ++
 tools/dtoc/test_fdt.py               | 6 +++++-
 3 files changed, 13 insertions(+), 3 deletions(-)

The use cases and design needs more thinking still, but later.

#3 lore raw

PATCH v2 2/25 moveconfig: Remove remove_defconfig()

In reply to Simon Glass

This is not necessary if simpler code is used. Use the split function and drop the unnecessary []

---

Changes in v2:
- Add new patch to remove remove_defconfig()

 tools/moveconfig.py | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)
tools/moveconfig.py +1 -12
index cff1e30658..dea60f4661 100755
@@ -94,17 +94,6 @@ SIZES = {
 RE_REMOVE_DEFCONFIG = re.compile(r'(.*)_defconfig')
 
 ### helper functions ###
-def remove_defconfig(defc):
-    """Drop the _defconfig suffix on a string
-
-    Args:
-        defc (str): String to convert
-
-    Returns:
-        str: string with the '_defconfig' suffix removed
-    """
-    return RE_REMOVE_DEFCONFIG.match(defc)[1]
-
 def check_top_directory():
     """Exit if we are not at the top of source directory."""
     for fname in 'README', 'Licenses':
@@ -1671,7 +1660,7 @@ def do_find_config(config_list):
         print(f"Error: Not in Kconfig: %s" % ' '.join(adhoc))
     else:
         print(f'{len(out)} matches')
-        print(' '.join([remove_defconfig(item) for item in out]))
+        print(' '.join(item.split('_defconfig')[0] for item in out))
 
 
 def prefix_config(cfg):
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 2/25
#4 lore raw

Re: [PATCH v2 02/25] moveconfig: Remove remove_defconfig()

Quoted 5 lines · Simon Glass

This is not necessary if simpler code is used. Use the split function and drop the unnecessary []

Quoted 7 lines

---

Changes in v2:
- Add new patch to remove remove_defconfig()

 tools/moveconfig.py | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)
#5 lore raw

PATCH v2 3/25 moveconfig: Use re.fullmatch() to avoid extra check

In reply to Simon Glass

Simplify the code by using the available function.

---

Changes in v2:
- Add new patch to use re.fullmatch() to avoid extra check

 tools/moveconfig.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
tools/moveconfig.py +2 -2
index dea60f4661..4b0e2e250b 100755
@@ -1610,8 +1610,8 @@ def defconfig_matches(configs, re_match):
         bool: True if any CONFIG matches the regex
     """
     for cfg in configs:
-        m_cfg = re_match.match(cfg)
-        if m_cfg and m_cfg.span()[1] == len(cfg):
+        m_cfg = re_match.fullmatch(cfg)
+        if m_cfg:
             return True
     return False
 
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 3/25
#6 lore raw

Re: [PATCH v2 03/25] moveconfig: Use re.fullmatch() to avoid extra check

Quoted 10 lines · Simon Glass

Simplify the code by using the available function.

---

Changes in v2:
- Add new patch to use re.fullmatch() to avoid extra check

 tools/moveconfig.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

tools/moveconfig.py def defconfig_matches(configs, re_match): · line 1614

Quoted 12 lines
tools/moveconfig.py +2 -2
index dea60f4661..4b0e2e250b 100755
@@ -1610,8 +1610,8 @@ def defconfig_matches(configs, re_match):
         bool: True if any CONFIG matches the regex
     """
     for cfg in configs:
-        m_cfg = re_match.match(cfg)
-        if m_cfg and m_cfg.span()[1] == len(cfg):
+        m_cfg = re_match.fullmatch(cfg)
+        if m_cfg:

Could skip the variable entirely.

Quoted 3 lines
tools/moveconfig.py +0 -0
             return True
     return False
 
#7 lore raw

PATCH v2 4/25 spl: Correct Kconfig help for TPL_BINMAN_SYMBOLS

In reply to Simon Glass

Fix the help which should refer to TPL, not SPL.

---

Changes in v2:
- Add new patch to correct Kconfig help for TPL_BINMAN_SYMBOLS

 common/spl/Kconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
common/spl/Kconfig +3 -3
index 9418d37b2e..dc319adeac 100644
@@ -1334,16 +1334,16 @@ config TPL_SIZE_LIMIT
 	  If this value is zero, it is ignored.
 
 config TPL_BINMAN_SYMBOLS
-	bool "Declare binman symbols in SPL"
+	bool "Declare binman symbols in TPL"
 	depends on SPL_FRAMEWORK && BINMAN
 	default y
 	help
-	  This enables use of symbols in TPL which refer to U-Boot, enabling SPL
+	  This enables use of symbols in TPL which refer to U-Boot, enabling TPL
 	  to obtain the location of U-Boot simply by calling spl_get_image_pos()
 	  and spl_get_image_size().
 
 	  For this to work, you must have a U-Boot image in the binman image, so
-	  binman can update SPL with the location of it.
+	  binman can update TPL with the location of it.
 
 config TPL_FRAMEWORK
 	bool "Support TPL based upon the common SPL framework"
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 4/25
#8 lore raw

Re: [PATCH v2 04/25] spl: Correct Kconfig help for TPL_BINMAN_SYMBOLS

Quoted 11 lines · Simon Glass

Fix the help which should refer to TPL, not SPL.

---

Changes in v2:
- Add new patch to correct Kconfig help for TPL_BINMAN_SYMBOLS

 common/spl/Kconfig | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
#9 lore raw

PATCH v2 5/25 dtoc: Tidy up implementation of AddStringList()

In reply to Simon Glass

Refactor this to avoid a loop. Also add a test for an empty string.

---

Changes in v2:
- Add new patch to tidy up implementaiton of AddStringList()

 tools/dtoc/fdt.py      | 4 +---
 tools/dtoc/test_fdt.py | 6 ++++++
 2 files changed, 7 insertions(+), 3 deletions(-)
tools/dtoc/fdt.py +1 -3
index c16909a876..d933972918 100644
@@ -516,9 +516,7 @@ class Node:
         Returns:
             Prop added
         """
-        out = b''
-        for string in val:
-            out += bytes(string, 'utf-8') + b'\0'
+        out = b'\0'.join(bytes(s, 'utf-8') for s in val) + b'\0' if val else b''
         return self.AddData(prop_name, out)
 
     def AddInt(self, prop_name, val):
tools/dtoc/test_fdt.py +6 -0
index c2013b9289..d33328c392 100755
@@ -550,6 +550,12 @@ class TestProp(unittest.TestCase):
         data = self.fdt.getprop(self.node.Offset(), 'stringlist')
         self.assertEqual(b'123\x00456\0', data)
 
+        val = []
+        self.node.AddStringList('stringlist', val)
+        self.dtb.Sync(auto_resize=True)
+        data = self.fdt.getprop(self.node.Offset(), 'stringlist')
+        self.assertEqual(b'', data)
+
     def test_delete_node(self):
         """Test deleting a node"""
         old_offset = self.fdt.path_offset('/spl-test')
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 5/25
#10 lore raw

Re: [PATCH v2 05/25] dtoc: Tidy up implementation of AddStringList()

Quoted 4 lines · Simon Glass

Refactor this to avoid a loop. Also add a test for an empty string.

Quoted 8 lines

---

Changes in v2:
- Add new patch to tidy up implementaiton of AddStringList()

 tools/dtoc/fdt.py      | 4 +---
 tools/dtoc/test_fdt.py | 6 ++++++
 2 files changed, 7 insertions(+), 3 deletions(-)

tools/dtoc/fdt.py class Node: · line 519

Quoted 12 lines
tools/dtoc/fdt.py +1 -3
index c16909a876..d933972918 100644
@@ -516,9 +516,7 @@ class Node:
         Returns:
             Prop added
         """
-        out = b''
-        for string in val:
-            out += bytes(string, 'utf-8') + b'\0'
+        out = b'\0'.join(bytes(s, 'utf-8') for s in val) + b'\0' if val else b''

Meh, it turned out uglier than I thought it would be. Either is fine really, I didn't add "Reviewed-by"s for v1 only because I thought you'd need to send a v2 for the later FIT stuff anyway.

Quoted 5 lines
tools/dtoc/fdt.py +0 -0
         return self.AddData(prop_name, out)
 
     def AddInt(self, prop_name, val):
 
[...]
#11 lore raw

PATCH v2 6/25 elf: Rename load_segments() and module failure

In reply to Simon Glass

Rename this function to make it clear that it only reads loadable segments. Also update the error for missing module to better match the message emitted by Python.

---

Changes in v2:
- Add new patch to rename load_segments() and module failure

 tools/binman/elf.py      |  6 +++---
 tools/binman/elf_test.py | 18 +++++++++---------
 2 files changed, 12 insertions(+), 12 deletions(-)
tools/binman/elf.py +3 -3
index 5e7d6ae7b9..016e9b2885 100644
@@ -105,7 +105,7 @@ def GetSymbolFileOffset(fname, patterns):
                     return addr - seg['p_vaddr'] + seg['p_offset']
 
     if not ELF_TOOLS:
-        raise ValueError('Python elftools package is not available')
+        raise ValueError("Python: No module named 'elftools'")
 
     syms = {}
     with open(fname, 'rb') as fd:
@@ -371,7 +371,7 @@ def UpdateFile(infile, outfile, start_sym, end_sym, insert):
     tools.write_file(outfile, newdata)
     tout.info('Written to offset %#x' % syms[start_sym].offset)
 
-def read_segments(data):
+def read_loadable_segments(data):
     """Read segments from an ELF file
 
     Args:
@@ -389,7 +389,7 @@ def read_segments(data):
         ValueError: elftools is not available
     """
     if not ELF_TOOLS:
-        raise ValueError('Python elftools package is not available')
+        raise ValueError("Python: No module named 'elftools'")
     with io.BytesIO(data) as inf:
         try:
             elf = ELFFile(inf)
tools/binman/elf_test.py +9 -9
index a67915bda6..4c881a9755 100644
@@ -244,7 +244,7 @@ class TestElf(unittest.TestCase):
             fname = self.ElfTestFile('embed_data')
             with self.assertRaises(ValueError) as e:
                 elf.GetSymbolFileOffset(fname, ['embed_start', 'embed_end'])
-            self.assertIn('Python elftools package is not available',
+            self.assertIn("Python: No module named 'elftools'",
                       str(e.exception))
         finally:
             elf.ELF_TOOLS = old_val
@@ -258,31 +258,31 @@ class TestElf(unittest.TestCase):
         offset = elf.GetSymbolFileOffset(fname, ['missing_sym'])
         self.assertEqual({}, offset)
 
-    def test_read_segments(self):
-        """Test for read_segments()"""
+    def test_read_loadable_segments(self):
+        """Test for read_loadable_segments()"""
         if not elf.ELF_TOOLS:
             self.skipTest('Python elftools not available')
         fname = self.ElfTestFile('embed_data')
-        segments, entry = elf.read_segments(tools.read_file(fname))
+        segments, entry = elf.read_loadable_segments(tools.read_file(fname))
 
     def test_read_segments_fail(self):
-        """Test for read_segments() without elftools"""
+        """Test for read_loadable_segments() without elftools"""
         try:
             old_val = elf.ELF_TOOLS
             elf.ELF_TOOLS = False
             fname = self.ElfTestFile('embed_data')
             with self.assertRaises(ValueError) as e:
-                elf.read_segments(tools.read_file(fname))
-            self.assertIn('Python elftools package is not available',
+                elf.read_loadable_segments(tools.read_file(fname))
+            self.assertIn("Python: No module named 'elftools'",
                           str(e.exception))
         finally:
             elf.ELF_TOOLS = old_val
 
     def test_read_segments_bad_data(self):
-        """Test for read_segments() with an invalid ELF file"""
+        """Test for read_loadable_segments() with an invalid ELF file"""
         fname = self.ElfTestFile('embed_data')
         with self.assertRaises(ValueError) as e:
-            elf.read_segments(tools.get_bytes(100, 100))
+            elf.read_loadable_segments(tools.get_bytes(100, 100))
         self.assertIn('Magic number does not match', str(e.exception))
 
 
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 6/25
#12 lore raw

Re: [PATCH v2 06/25] elf: Rename load_segments() and module failure

Quoted 6 lines · Simon Glass

Rename this function to make it clear that it only reads loadable segments. Also update the error for missing module to better match the message emitted by Python.

Quoted 8 lines

---

Changes in v2:
- Add new patch to rename load_segments() and module failure

 tools/binman/elf.py      |  6 +++---
 tools/binman/elf_test.py | 18 +++++++++---------
 2 files changed, 12 insertions(+), 12 deletions(-)
#13 lore raw

PATCH v2 7/25 binman: Tweak collect_contents_to_file() and docs

In reply to Simon Glass

Update the return value of this function, fix the 'create' typo and update the documentation for clarity.

---

Changes in v2:
- Add new patch to tweak collect_contents_to_file() and docs

 tools/binman/binman.rst       | 25 +++++++++++++++----------
 tools/binman/entry.py         |  8 ++++----
 tools/binman/etype/mkimage.py |  2 +-
 3 files changed, 20 insertions(+), 15 deletions(-)
tools/binman/binman.rst +15 -10
index 771645380e..0061e43659 100644
@@ -1375,18 +1375,20 @@ Some entry types deal with data obtained from others. For example,
     };
 
 This shows mkimage being passed a file consisting of SPL and U-Boot proper. It
-is create by calling `Entry.collect_contents_to_file()`. Note that in this case,
-the data is passed to mkimage for processing but does not appear separately in
-the image. It may not appear at all, depending on what mkimage does. The
-contents of the `mkimage` entry are entirely dependent on the processing done
-by the entry, with the provided subnodes (`u-boot-spl` and `u-boot`) simply
-providing the input data for that processing.
+is created by calling `Entry.collect_contents_to_file()`. Note that in this
+case, the data is passed to mkimage for processing but does not appear
+separately in the image. It may not appear at all, depending on what mkimage
+does. The contents of the `mkimage` entry are entirely dependent on the
+processing done by the entry, with the provided subnodes (`u-boot-spl` and
+`u-boot`) simply providing the input data for that processing.
 
 Note that `Entry.collect_contents_to_file()` simply concatenates the data from
 the different entries together, with no control over alignment, etc. Another
 approach is to subclass `Entry_section` so that those features become available,
 such as `size` and `pad-byte`. Then the contents of the entry can be obtained by
-calling `BuildSectionData()`.
+calling `super().BuildSectionData()` in the entry's BuildSectionData()
+implementation to get the input data, then write it to a file and process it
+however is desired.
 
 There are other ways to obtain data also, depending on the situation. If the
 entry type is simply signing data which exists elsewhere in the image, then
@@ -1396,6 +1398,7 @@ is used by `Entry_vblock`, for example::
 
     u_boot: u-boot {
     };
+
     vblock {
         content = <&u_boot &dtb>;
         keyblock = "firmware.keyblock";
@@ -1440,9 +1443,11 @@ The `soc-fw` node is a `blob-ext` (i.e. it reads in a named binary file) whereas
 a known blob type.
 
 When adding new entry types you are encouraged to use subnodes to provide the
-data for processing, unless the `content` approach is more suitable. Ad-hoc
-properties and other methods of obtaining data are discouraged, since it adds to
-confusion for users.
+data for processing, unless the `content` approach is more suitable. Consider
+whether the input entries are contained within (or consumed by) the entry, vs
+just being 'referenced' by the entry. In the latter case, the `content` approach
+makes more sense. Ad-hoc properties and other methods of obtaining data are
+discouraged, since it adds to confusion for users.
 
 History / Credits
 -----------------
tools/binman/entry.py +4 -4
index bf68a85b24..de8526618d 100644
@@ -1136,16 +1136,16 @@ features to produce new behaviours.
 
         Returns:
             Tuple:
-                bytes: Concatenated data from all the entries (or False)
-                str: Filename of file written (or False if no data)
-                str: Unique portion of filename (or False if no data)
+                bytes: Concatenated data from all the entries (or None)
+                str: Filename of file written (or None if no data)
+                str: Unique portion of filename (or None if no data)
         """
         data = b''
         for entry in entries:
             # First get the input data and put it in a file. If not available,
             # try later.
             if not entry.ObtainContents():
-                return False, False, False
+                return None, None, None
             data += entry.GetData()
         uniq = self.GetUniqueName()
         fname = tools.get_output_filename(f'{prefix}.{uniq}')
tools/binman/etype/mkimage.py +1 -1
index 8e74ebb9c0..c28141ff69 100644
@@ -53,7 +53,7 @@ class Entry_mkimage(Entry):
     def ObtainContents(self):
         data, input_fname, uniq = self.collect_contents_to_file(
             self._mkimage_entries.values(), 'mkimage')
-        if data is False:
+        if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 7/25
#14 lore raw

Re: [PATCH v2 07/25] binman: Tweak collect_contents_to_file() and docs

Quoted 7 lines · Simon Glass

Update the return value of this function, fix the 'create' typo and update the documentation for clarity.

---

Quoted 7 lines

Changes in v2:
- Add new patch to tweak collect_contents_to_file() and docs

 tools/binman/binman.rst       | 25 +++++++++++++++----------
 tools/binman/entry.py         |  8 ++++----
 tools/binman/etype/mkimage.py |  2 +-
 3 files changed, 20 insertions(+), 15 deletions(-)
#15 lore raw

PATCH v2 8/25 binman: Rename ExpandToLimit to extend_to_limit

In reply to Simon Glass

The word 'expand' is used for entries which generate subentries. It is also used for entries that can have an '_expanded' version which is used to break out its contents.

Rather than talking about expanding an entry's size, use the term 'extending'. It is slightly more precise and avoids the above conflicts.

This change renders the old 'expand-size' property invalid, so add an error check for that.

---

Changes in v2:
- Add patch to rename ExpandToLimit to extend_to_limit

 tools/binman/binman.rst                   |  2 +-
 tools/binman/entry.py                     |  8 +++++---
 tools/binman/etype/section.py             | 10 +++++-----
 tools/binman/ftest.py                     |  8 ++++++++
 tools/binman/test/088_expand_size.dts     |  8 ++++----
 tools/binman/test/089_expand_size_bad.dts |  2 +-
 tools/binman/test/225_expand_size_bad.dts | 10 ++++++++++
 7 files changed, 34 insertions(+), 14 deletions(-)
 create mode 100644 tools/binman/test/225_expand_size_bad.dts
tools/binman/binman.rst +1 -1
index 0061e43659..85f8337a31 100644
@@ -480,7 +480,7 @@ image-pos:
     for each entry. This makes it easy to find out exactly where the entry
     ended up in the image, regardless of parent sections, etc.
 
-expand-size:
+extend-size:
     Expand the size of this entry to fit available space. This space is only
     limited by the size of the image/section and the position of the next
     entry.
tools/binman/entry.py +5 -3
index de8526618d..a59eb56f14 100644
@@ -233,6 +233,8 @@ class Entry(object):
         """
         if 'pos' in self._node.props:
             self.Raise("Please use 'offset' instead of 'pos'")
+        if 'expand-size' in self._node.props:
+            self.Raise("Please use 'extend-size' instead of 'expand-size'")
         self.offset = fdt_util.GetInt(self._node, 'offset')
         self.size = fdt_util.GetInt(self._node, 'size')
         self.orig_offset = fdt_util.GetInt(self._node, 'orig-offset')
@@ -260,7 +262,7 @@ class Entry(object):
                        self.align_size)
         self.align_end = fdt_util.GetInt(self._node, 'align-end')
         self.offset_unset = fdt_util.GetBool(self._node, 'offset-unset')
-        self.expand_size = fdt_util.GetBool(self._node, 'expand-size')
+        self.expand_size = fdt_util.GetBool(self._node, 'extend-size')
         self.missing_msg = fdt_util.GetString(self._node, 'missing-msg')
 
         # This is only supported by blobs and sections at present
@@ -772,8 +774,8 @@ features to produce new behaviours.
             name = '%s.%s' % (node.name, name)
         return name
 
-    def ExpandToLimit(self, limit):
-        """Expand an entry so that it ends at the given offset limit"""
+    def extend_to_limit(self, limit):
+        """Extent an entry so that it ends at the given offset limit"""
         if self.offset + self.size < limit:
             self.size = limit - self.offset
             # Request the contents again, since changing the size requires that
tools/binman/etype/section.py +5 -5
index 25159074ba..0cfee5f3e3 100644
@@ -385,7 +385,7 @@ class Entry_section(Entry):
         self._PackEntries()
         if self._sort:
             self._SortEntries()
-        self._ExpandEntries()
+        self._extend_entries()
 
         data = self.BuildSectionData(True)
         self.SetContents(data)
@@ -403,17 +403,17 @@ class Entry_section(Entry):
             offset = entry.Pack(offset)
         return offset
 
-    def _ExpandEntries(self):
-        """Expand any entries that are permitted to"""
+    def _extend_entries(self):
+        """Extend any entries that are permitted to"""
         exp_entry = None
         for entry in self._entries.values():
             if exp_entry:
-                exp_entry.ExpandToLimit(entry.offset)
+                exp_entry.extend_to_limit(entry.offset)
                 exp_entry = None
             if entry.expand_size:
                 exp_entry = entry
         if exp_entry:
-            exp_entry.ExpandToLimit(self.size)
+            exp_entry.extend_to_limit(self.size)
 
     def _SortEntries(self):
         """Sort entries by offset"""
tools/binman/ftest.py +8 -0
index 8f00db6945..62528ccbfa 100644
@@ -5321,6 +5321,14 @@ fdt         fdtmap                Extract the devicetree blob from the fdtmap
         self.assertIn("Node '/binman/fit': Unknown operation 'unknown'",
                       str(exc.exception))
 
+    def test_uses_expand_size(self):
+        """Test that the 'expand-size' property cannot be used anymore"""
+        with self.assertRaises(ValueError) as e:
+           data = self._DoReadFile('225_expand_size_bad.dts')
+        self.assertIn(
+            "Node '/binman/u-boot': Please use 'extend-size' instead of 'expand-size'",
+            str(e.exception))
+
 
 if __name__ == "__main__":
     unittest.main()
tools/binman/test/088_expand_size.dts +4 -4
index c8a01308ec..f352699e37 100644
@@ -5,7 +5,7 @@
 	binman {
 		size = <40>;
 		fill {
-			expand-size;
+			extend-size;
 			fill-byte = [61];
 			size = <0>;
 		};
@@ -13,7 +13,7 @@
 			offset = <8>;
 		};
 		section {
-			expand-size;
+			extend-size;
 			pad-byte = <0x62>;
 			intel-mrc {
 			};
@@ -25,7 +25,7 @@
 		section2 {
 			type = "section";
 			fill {
-				expand-size;
+				extend-size;
 				fill-byte = [63];
 				size = <0>;
 			};
@@ -35,7 +35,7 @@
 		};
 		fill2 {
 			type = "fill";
-			expand-size;
+			extend-size;
 			fill-byte = [64];
 			size = <0>;
 		};
tools/binman/test/089_expand_size_bad.dts +1 -1
index edc0e5cf68..edc60e43fd 100644
@@ -4,7 +4,7 @@
 / {
 	binman {
 		_testing {
-			expand-size;
+			extend-size;
 			return-contents-once;
 		};
 		u-boot {
tools/binman/test/225_expand_size_bad.dts new file +10 -0
new file mode 100644
index 0000000000..d4ad9a6a1a
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0+
+/dts-v1/;
+
+/ {
+	binman {
+		u-boot {
+			expand-size;
+		};
+	};
+};
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 8/25
#16 lore raw

Re: [PATCH v2 08/25] binman: Rename ExpandToLimit to extend_to_limit

Quoted 16 lines · Simon Glass

The word 'expand' is used for entries which generate subentries. It is also used for entries that can have an '_expanded' version which is used to break out its contents.

Rather than talking about expanding an entry's size, use the term 'extending'. It is slightly more precise and avoids the above conflicts.

This change renders the old 'expand-size' property invalid, so add an error check for that.

---

Changes in v2:
- Add patch to rename ExpandToLimit to extend_to_limit

Quoted 9 lines
 tools/binman/binman.rst                   |  2 +-
 tools/binman/entry.py                     |  8 +++++---
 tools/binman/etype/section.py             | 10 +++++-----
 tools/binman/ftest.py                     |  8 ++++++++
 tools/binman/test/088_expand_size.dts     |  8 ++++----
 tools/binman/test/089_expand_size_bad.dts |  2 +-
 tools/binman/test/225_expand_size_bad.dts | 10 ++++++++++
 7 files changed, 34 insertions(+), 14 deletions(-)
 create mode 100644 tools/binman/test/225_expand_size_bad.dts

I haven't ever used expand-size, but:

tools/binman/binman.rst image-pos: · line 484

Quoted 11 lines
tools/binman/binman.rst +1 -1
index 0061e43659..85f8337a31 100644
@@ -480,7 +480,7 @@ image-pos:
     for each entry. This makes it easy to find out exactly where the entry
     ended up in the image, regardless of parent sections, etc.
 
-expand-size:
+extend-size:
     Expand the size of this entry to fit available space. This space is only

Expand -> Extend

Quoted 16 lines
tools/binman/binman.rst +0 -0
     limited by the size of the image/section and the position of the next
     entry.
tools/binman/entry.py +2 -0
index de8526618d..a59eb56f14 100644
@@ -233,6 +233,8 @@ class Entry(object):
         """
         if 'pos' in self._node.props:
             self.Raise("Please use 'offset' instead of 'pos'")
+        if 'expand-size' in self._node.props:
+            self.Raise("Please use 'extend-size' instead of 'expand-size'")
         self.offset = fdt_util.GetInt(self._node, 'offset')
         self.size = fdt_util.GetInt(self._node, 'size')
         self.orig_offset = fdt_util.GetInt(self._node, 'orig-offset')

tools/binman/entry.py class Entry(object): · line 265

Quoted 5 lines
tools/binman/entry.py +1 -1
@@ -260,7 +262,7 @@ class Entry(object):
                        self.align_size)
         self.align_end = fdt_util.GetInt(self._node, 'align-end')
         self.offset_unset = fdt_util.GetBool(self._node, 'offset-unset')
-        self.expand_size = fdt_util.GetBool(self._node, 'expand-size')
+        self.expand_size = fdt_util.GetBool(self._node, 'extend-size')

Consider also changing the attribute name, I don't see many instances of it with git-grep. Maybe even the test dts names.

tools/binman/entry.py features to produce new behaviours. · line 778

Quoted 11 lines
tools/binman/entry.py +2 -2
         self.missing_msg = fdt_util.GetString(self._node, 'missing-msg')
 
         # This is only supported by blobs and sections at present
@@ -772,8 +774,8 @@ features to produce new behaviours.
             name = '%s.%s' % (node.name, name)
         return name
 
-    def ExpandToLimit(self, limit):
-        """Expand an entry so that it ends at the given offset limit"""
+    def extend_to_limit(self, limit):
+        """Extent an entry so that it ends at the given offset limit"""

Extent -> Extend

Quoted 4 lines
tools/binman/entry.py +0 -0
         if self.offset + self.size < limit:
             self.size = limit - self.offset
             # Request the contents again, since changing the size requires that
[...]
#17 lore raw

PATCH v2 9/25 binman: Rename ExpandEntries to gen_entries

In reply to Simon Glass

Leave the 'expand' term for use by entry types which have an expanded version of themselves. Rename this method to indicate that it generates subentries.

---

Changes in v2:
- Add patch to rename ExpandEntries to gen_entries

 tools/binman/control.py          | 2 +-
 tools/binman/entry.py            | 4 ++--
 tools/binman/etype/blob_phase.py | 2 +-
 tools/binman/etype/files.py      | 2 +-
 tools/binman/etype/section.py    | 6 +++---
 5 files changed, 8 insertions(+), 8 deletions(-)
tools/binman/control.py +1 -1
index a179f78129..be2b422e62 100644
@@ -501,7 +501,7 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded):
     # entry offsets remain the same.
     for image in images.values():
         image.CollectBintools()
-        image.ExpandEntries()
+        image.gen_entries()
         if update_fdt:
             image.AddMissingProperties(True)
         image.ProcessFdt(dtb)
tools/binman/entry.py +2 -2
index a59eb56f14..90a40bb274 100644
@@ -284,8 +284,8 @@ class Entry(object):
         """
         return {}
 
-    def ExpandEntries(self):
-        """Expand out entries which produce other entries
+    def gen_entries(self):
+        """Allow entries to generate other entries
 
         Some entries generate subnodes automatically, from which sub-entries
         are then created. This method allows those to be added to the binman
tools/binman/etype/blob_phase.py +1 -1
index ed25e467a1..aba93b759b 100644
@@ -39,7 +39,7 @@ class Entry_blob_phase(Entry_section):
         self.dtb_file = dtb_file
         self.bss_pad = bss_pad
 
-    def ExpandEntries(self):
+    def gen_entries(self):
         """Create the subnodes"""
         names = [self.root_fname + '-nodtb', self.root_fname + '-dtb']
         if self.bss_pad:
tools/binman/etype/files.py +1 -1
index 0650a69c55..210a16d671 100644
@@ -46,7 +46,7 @@ class Entry_files(Entry_section):
         self._require_matches = fdt_util.GetBool(self._node,
                                                 'require-matches')
 
-    def ExpandEntries(self):
+    def gen_entries(self):
         files = tools.get_input_filename_glob(self._pattern)
         if self._require_matches and not files:
             self.Raise("Pattern '%s' matched no files" % self._pattern)
tools/binman/etype/section.py +3 -3
index 0cfee5f3e3..f2d401d95b 100644
@@ -233,10 +233,10 @@ class Entry_section(Entry):
                        todo)
         return True
 
-    def ExpandEntries(self):
-        super().ExpandEntries()
+    def gen_entries(self):
+        super().gen_entries()
         for entry in self._entries.values():
-            entry.ExpandEntries()
+            entry.gen_entries()
 
     def AddMissingProperties(self, have_image_pos):
         """Add new properties to the device tree as needed for this entry"""
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 9/25
#18 lore raw

Re: [PATCH v2 09/25] binman: Rename ExpandEntries to gen_entries

Quoted 10 lines · Simon Glass

Leave the 'expand' term for use by entry types which have an expanded version of themselves. Rename this method to indicate that it generates subentries.

---

Changes in v2:
- Add patch to rename ExpandEntries to gen_entries

Quoted 6 lines
 tools/binman/control.py          | 2 +-
 tools/binman/entry.py            | 4 ++--
 tools/binman/etype/blob_phase.py | 2 +-
 tools/binman/etype/files.py      | 2 +-
 tools/binman/etype/section.py    | 6 +++---
 5 files changed, 8 insertions(+), 8 deletions(-)
#19 lore raw

PATCH v2 10/25 binman: Refactor fit to generate output at the end

In reply to Simon Glass

At present the fit implementation creates the output tree while scanning the FIT description. Then it updates the tree later when the data is known.

This works, but is a bit confusing, since it requires mixing the scanning code with the generation code, with a fix-up step at the end.

It is actually possible to do this in two phases, one to scan everything and the other to generate the FIT. Thus the FIT is generated in one pass, when everything is known.

Update the code accordingly. The only functional change is that the 'data' property for each node are now last instead of first, which is really a more natural position. Update the affected test to deal with this.

One wrinkle is that the calculated properties (image-pos, size and offset) are now added before the FIT is generated. so we must filter these out when copying properties from the binman description to the FIT.

Most of the change here is splitting out some of the code from the ReadEntries() implementation into _BuildInput(). So despite the large diff, most of the code is the same. It is not feasible to split this patch up, so far as I can tell.

---

Changes in v2:
- Add new patch to refactor fit to generate output at the end

 tools/binman/etype/fit.py              | 178 ++++++++++++++-----------
 tools/binman/ftest.py                  |  13 +-
 tools/binman/test/224_fit_bad_oper.dts |   2 -
 3 files changed, 109 insertions(+), 84 deletions(-)
tools/binman/etype/fit.py +102 -76
index 2d4c5f6545..61c72780e9 100644
@@ -209,6 +209,81 @@ class Entry_fit(Entry_section):
         return oper
 
     def ReadEntries(self):
+        def _add_entries(base_node, depth, node):
+            """Add entries for any nodes that need them
+
+            Args:
+                base_node: Base Node of the FIT (with 'description' property)
+                depth: Current node depth (0 is the base 'fit' node)
+                node: Current node to process
+
+            He we only need to provide binman entries which are used to define
+            the 'data' for each image. We create an entry_Section for each.
+            """
+            rel_path = node.path[len(base_node.path):]
+            in_images = rel_path.startswith('/images')
+            has_images = depth == 2 and in_images
+            if has_images:
+                # This node is a FIT subimage node (e.g. "/images/kernel")
+                # containing content nodes. We collect the subimage nodes and
+                # section entries for them here to merge the content subnodes
+                # together and put the merged contents in the subimage node's
+                # 'data' property later.
+                entry = Entry.Create(self.section, node, etype='section')
+                entry.ReadNode()
+                # The hash subnodes here are for mkimage, not binman.
+                entry.SetUpdateHash(False)
+                self._entries[rel_path] = entry
+
+            for subnode in node.subnodes:
+                _add_entries(base_node, depth + 1, subnode)
+
+        _add_entries(self._node, 0, self._node)
+
+    def BuildSectionData(self, required):
+        """Build FIT entry contents
+
+        This adds the 'data' properties to the input ITB (Image-tree Binary)
+        then runs mkimage to process it.
+
+        Args:
+            required: True if the data must be present, False if it is OK to
+                return None
+
+        Returns:
+            Contents of the section (bytes)
+        """
+        data = self._BuildInput()
+        uniq = self.GetUniqueName()
+        input_fname = tools.get_output_filename('%s.itb' % uniq)
+        output_fname = tools.get_output_filename('%s.fit' % uniq)
+        tools.write_file(input_fname, data)
+        tools.write_file(output_fname, data)
+
+        args = {}
+        ext_offset = self._fit_props.get('fit,external-offset')
+        if ext_offset is not None:
+            args = {
+                'external': True,
+                'pad': fdt_util.fdt32_to_cpu(ext_offset.value)
+                }
+        if self.mkimage.run(reset_timestamp=True, output_fname=output_fname,
+                            **args) is None:
+            # Bintool is missing; just use empty data as the output
+            self.record_missing_bintool(self.mkimage)
+            return tools.get_bytes(0, 1024)
+
+        return tools.read_file(output_fname)
+
+    def _BuildInput(self):
+        """Finish the FIT by adding the 'data' properties to it
+
+        Arguments:
+            fdt: FIT to update
+
+        Returns:
+            New fdt contents (bytes)
+        """
         def _process_prop(pname, prop):
             """Process special properties
 
@@ -236,9 +311,15 @@ class Entry_fit(Entry_section):
                     val = val[1:].replace('DEFAULT-SEQ', str(seq + 1))
                     fsw.property_string(pname, val)
                     return
+            elif pname.startswith('fit,'):
+                # Ignore these, which are commands for binman to process
+                return
+            elif pname in ['offset', 'size', 'image-pos']:
+                # Don't add binman's calculated properties
+                return
             fsw.property(pname, prop.bytes)
 
-        def _scan_gen_fdt_nodes(subnode, depth, in_images):
+        def _gen_fdt_nodes(subnode, depth, in_images):
             """Generate FDT nodes
 
             This creates one node for each member of self._fdts using the
@@ -277,7 +358,7 @@ class Entry_fit(Entry_section):
                     else:
                         self.Raise("Generator node requires 'fit,fdt-list' property")
 
-        def _scan_node(subnode, depth, in_images):
+        def _gen_node(subnode, depth, in_images):
             """Generate nodes from a template
 
             This creates one node for each member of self._fdts using the
@@ -294,10 +375,10 @@ class Entry_fit(Entry_section):
             """
             oper = self._get_operation(subnode)
             if oper == OP_GEN_FDT_NODES:
-                _scan_gen_fdt_nodes(subnode, depth, in_images)
+                _gen_fdt_nodes(subnode, depth, in_images)
 
-        def _AddNode(base_node, depth, node):
-            """Add a node to the FIT
+        def _add_node(base_node, depth, node):
+            """Add nodes to the output FIT
 
             Args:
                 base_node: Base Node of the FIT (with 'description' property)
@@ -307,104 +388,49 @@ class Entry_fit(Entry_section):
             There are two cases to deal with:
                 - hash and signature nodes which become part of the FIT
                 - binman entries which are used to define the 'data' for each
-                  image
+                  image, so don't appear in the FIT
             """
+            # Copy over all the relevant properties
             for pname, prop in node.props.items():
-                if not pname.startswith('fit,'):
-                    _process_prop(pname, prop)
+                _process_prop(pname, prop)
 
             rel_path = node.path[len(base_node.path):]
             in_images = rel_path.startswith('/images')
+
             has_images = depth == 2 and in_images
             if has_images:
-                # This node is a FIT subimage node (e.g. "/images/kernel")
-                # containing content nodes. We collect the subimage nodes and
-                # section entries for them here to merge the content subnodes
-                # together and put the merged contents in the subimage node's
-                # 'data' property later.
-                entry = Entry.Create(self.section, node, etype='section')
-                entry.ReadNode()
-                # The hash subnodes here are for mkimage, not binman.
-                entry.SetUpdateHash(False)
-                self._entries[rel_path] = entry
+                entry = self._entries[rel_path]
+                data = entry.GetData()
+                fsw.property('data', bytes(data))
 
             for subnode in node.subnodes:
                 if has_images and not (subnode.name.startswith('hash') or
                                        subnode.name.startswith('signature')):
                     # This subnode is a content node not meant to appear in
                     # the FIT (e.g. "/images/kernel/u-boot"), so don't call
-                    # fsw.add_node() or _AddNode() for it.
+                    # fsw.add_node() or _add_node() for it.
                     pass
                 elif self.GetImage().generate and subnode.name.startswith('@'):
-                    _scan_node(subnode, depth, in_images)
+                    subnode_path = f'{rel_path}/{subnode.name}'
+                    entry = self._entries.get(subnode_path)
+                    _gen_node(subnode, depth, in_images)
+                    if entry:
+                        del self._entries[subnode_path]
                 else:
                     with fsw.add_node(subnode.name):
-                        _AddNode(base_node, depth + 1, subnode)
+                        _add_node(base_node, depth + 1, subnode)
 
         # Build a new tree with all nodes and properties starting from the
         # entry node
         fsw = libfdt.FdtSw()
         fsw.finish_reservemap()
         with fsw.add_node(''):
-            _AddNode(self._node, 0, self._node)
+            _add_node(self._node, 0, self._node)
         fdt = fsw.as_fdt()
 
         # Pack this new FDT and scan it so we can add the data later
         fdt.pack()
-        self._fdt = Fdt.FromData(fdt.as_bytearray())
-        self._fdt.Scan()
-
-    def BuildSectionData(self, required):
-        """Build FIT entry contents
-
-        This adds the 'data' properties to the input ITB (Image-tree Binary)
-        then runs mkimage to process it.
-
-        Args:
-            required: True if the data must be present, False if it is OK to
-                return None
-
-        Returns:
-            Contents of the section (bytes)
-        """
-        data = self._BuildInput(self._fdt)
-        uniq = self.GetUniqueName()
-        input_fname = tools.get_output_filename('%s.itb' % uniq)
-        output_fname = tools.get_output_filename('%s.fit' % uniq)
-        tools.write_file(input_fname, data)
-        tools.write_file(output_fname, data)
-
-        args = {}
-        ext_offset = self._fit_props.get('fit,external-offset')
-        if ext_offset is not None:
-            args = {
-                'external': True,
-                'pad': fdt_util.fdt32_to_cpu(ext_offset.value)
-                }
-        if self.mkimage.run(reset_timestamp=True, output_fname=output_fname,
-                            **args) is None:
-            # Bintool is missing; just use empty data as the output
-            self.record_missing_bintool(self.mkimage)
-            return tools.get_bytes(0, 1024)
-
-        return tools.read_file(output_fname)
-
-    def _BuildInput(self, fdt):
-        """Finish the FIT by adding the 'data' properties to it
-
-        Arguments:
-            fdt: FIT to update
-
-        Returns:
-            New fdt contents (bytes)
-        """
-        for path, section in self._entries.items():
-            node = fdt.GetNode(path)
-            data = section.GetData()
-            node.AddData('data', data)
-
-        fdt.Sync(auto_resize=True)
-        data = fdt.GetContents()
+        data = fdt.as_bytearray()
         return data
 
     def SetImagePos(self, image_pos):
tools/binman/ftest.py +7 -6
index 62528ccbfa..4eac913d06 100644
@@ -3780,6 +3780,7 @@ class TestFunctional(unittest.TestCase):
         dtb.Scan()
         props = self._GetPropTree(dtb, BASE_DTB_PROPS + REPACK_DTB_PROPS)
 
+        self.maxDiff = None
         self.assertEqual({
             'image-pos': 0,
             'offset': 0,
@@ -3793,19 +3794,19 @@ class TestFunctional(unittest.TestCase):
             'fit:offset': 4,
             'fit:size': 1840,
 
-            'fit/images/kernel:image-pos': 160,
-            'fit/images/kernel:offset': 156,
+            'fit/images/kernel:image-pos': 304,
+            'fit/images/kernel:offset': 300,
             'fit/images/kernel:size': 4,
 
-            'fit/images/kernel/u-boot:image-pos': 160,
+            'fit/images/kernel/u-boot:image-pos': 304,
             'fit/images/kernel/u-boot:offset': 0,
             'fit/images/kernel/u-boot:size': 4,
 
-            'fit/images/fdt-1:image-pos': 456,
-            'fit/images/fdt-1:offset': 452,
+            'fit/images/fdt-1:image-pos': 552,
+            'fit/images/fdt-1:offset': 548,
             'fit/images/fdt-1:size': 6,
 
-            'fit/images/fdt-1/u-boot-spl-dtb:image-pos': 456,
+            'fit/images/fdt-1/u-boot-spl-dtb:image-pos': 552,
             'fit/images/fdt-1/u-boot-spl-dtb:offset': 0,
             'fit/images/fdt-1/u-boot-spl-dtb:size': 6,
 
tools/binman/test/224_fit_bad_oper.dts +0 -2
index cee801e2ea..8a8014ea33 100644
@@ -21,7 +21,5 @@
 				};
 			};
 		};
-		fdtmap {
-		};
 	};
 };
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 10/25
#20 lore raw

Re: [PATCH v2 10/25] binman: Refactor fit to generate output at the end

Quoted 10 lines · Simon Glass

At present the fit implementation creates the output tree while scanning the FIT description. Then it updates the tree later when the data is known.

This works, but is a bit confusing, since it requires mixing the scanning code with the generation code, with a fix-up step at the end.

It is actually possible to do this in two phases, one to scan everything and the other to generate the FIT. Thus the FIT is generated in one pass, when everything is known.

Doing it in one go makes sense to me as well. In general I like the way distinct processing actions/steps are being split into their own blocks or so, and I think this helps move things toward that.

Quoted 16 lines

Update the code accordingly. The only functional change is that the 'data' property for each node are now last instead of first, which is really a more natural position. Update the affected test to deal with this.

One wrinkle is that the calculated properties (image-pos, size and offset) are now added before the FIT is generated. so we must filter these out when copying properties from the binman description to the FIT.

Most of the change here is splitting out some of the code from the ReadEntries() implementation into _BuildInput(). So despite the large diff, most of the code is the same. It is not feasible to split this patch up, so far as I can tell.

---

Quoted 7 lines

Changes in v2:
- Add new patch to refactor fit to generate output at the end

 tools/binman/etype/fit.py              | 178 ++++++++++++++-----------
 tools/binman/ftest.py                  |  13 +-
 tools/binman/test/224_fit_bad_oper.dts |   2 -
 3 files changed, 109 insertions(+), 84 deletions(-)

I still wrote some weird ideas below, mostly for the future, since this patch is mostly moving code around which is fine as is.

Quoted 7 lines
tools/binman/etype/fit.py +0 -0
index 2d4c5f6545..61c72780e9 100644
@@ -209,6 +209,81 @@ class Entry_fit(Entry_section):
         return oper
 

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 220

Quoted 10 lines
tools/binman/etype/fit.py +9 -0
     def ReadEntries(self):
+        def _add_entries(base_node, depth, node):
+            """Add entries for any nodes that need them
+
+            Args:
+                base_node: Base Node of the FIT (with 'description' property)
+                depth: Current node depth (0 is the base 'fit' node)
+                node: Current node to process
+
+            He we only need to provide binman entries which are used to define

He -> Here ?

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 233

Quoted 13 lines
tools/binman/etype/fit.py +13 -0
+            the 'data' for each image. We create an entry_Section for each.
+            """
+            rel_path = node.path[len(base_node.path):]
+            in_images = rel_path.startswith('/images')
+            has_images = depth == 2 and in_images
+            if has_images:
+                # This node is a FIT subimage node (e.g. "/images/kernel")
+                # containing content nodes. We collect the subimage nodes and
+                # section entries for them here to merge the content subnodes
+                # together and put the merged contents in the subimage node's
+                # 'data' property later.
+                entry = Entry.Create(self.section, node, etype='section')
+                entry.ReadNode()

I plan to change 'self.section' to 'self' here later, fixes extracting wrong contents for FIT subentries.

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 237

Quoted 4 lines
tools/binman/etype/fit.py +4 -0
+                # The hash subnodes here are for mkimage, not binman.
+                entry.SetUpdateHash(False)
+                self._entries[rel_path] = entry
+

I also plan to change this to a single-level node name instead of the relative path, lets 'binman extract fit/u-boot' etc. run at all.

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 241

Quoted 4 lines
tools/binman/etype/fit.py +4 -0
+            for subnode in node.subnodes:
+                _add_entries(base_node, depth + 1, subnode)
+
+        _add_entries(self._node, 0, self._node)

I think it's especially visible here what I meant by switching away from
recursion: this recurses through every node but only does anything on immediate subnodes of "/images" (for now?).

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 251

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+
+    def BuildSectionData(self, required):
+        """Build FIT entry contents
+
+        This adds the 'data' properties to the input ITB (Image-tree Binary)
+        then runs mkimage to process it.
+
+        Args:
+            required: True if the data must be present, False if it is OK to
+                return None

I forgot to handle 'required' while converting FIT to section...

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+
+        Returns:
+            Contents of the section (bytes)
+        """
+        data = self._BuildInput()
+        uniq = self.GetUniqueName()
+        input_fname = tools.get_output_filename('%s.itb' % uniq)
+        output_fname = tools.get_output_filename('%s.fit' % uniq)
+        tools.write_file(input_fname, data)
+        tools.write_file(output_fname, data)

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 271

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+
+        args = {}
+        ext_offset = self._fit_props.get('fit,external-offset')
+        if ext_offset is not None:
+            args = {
+                'external': True,
+                'pad': fdt_util.fdt32_to_cpu(ext_offset.value)
+                }
+        if self.mkimage.run(reset_timestamp=True, output_fname=output_fname,
+                            **args) is None:

I have an idea for the far future, to let /image/* nodes sometimes be Entry_collection to handle external offsets in binman so we can take mkimage completely out of this, but no clue how feasible/desirable that end goal is.

Quoted 19 lines
tools/binman/etype/fit.py +15 -0
+            # Bintool is missing; just use empty data as the output
+            self.record_missing_bintool(self.mkimage)
+            return tools.get_bytes(0, 1024)
+
+        return tools.read_file(output_fname)
+
+    def _BuildInput(self):
+        """Finish the FIT by adding the 'data' properties to it
+
+        Arguments:
+            fdt: FIT to update
+
+        Returns:
+            New fdt contents (bytes)
+        """
         def _process_prop(pname, prop):
             """Process special properties
 

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 319

Quoted 9 lines
tools/binman/etype/fit.py +6 -0
@@ -236,9 +311,15 @@ class Entry_fit(Entry_section):
                     val = val[1:].replace('DEFAULT-SEQ', str(seq + 1))
                     fsw.property_string(pname, val)
                     return
+            elif pname.startswith('fit,'):
+                # Ignore these, which are commands for binman to process
+                return
+            elif pname in ['offset', 'size', 'image-pos']:
+                # Don't add binman's calculated properties
+                return

This is one of the things I was thinking of doing, thanks. I encountered the same issue when replacing a FIT entry with the same f"{uniq}.fit" that was used to build it, will try adding a test for that later.

Quoted 14 lines
tools/binman/etype/fit.py +1 -1
             fsw.property(pname, prop.bytes)
 
-        def _scan_gen_fdt_nodes(subnode, depth, in_images):
+        def _gen_fdt_nodes(subnode, depth, in_images):
             """Generate FDT nodes
 
             This creates one node for each member of self._fdts using the
[...]
tools/binman/test/224_fit_bad_oper.dts +0 -0
index cee801e2ea..8a8014ea33 100644

tools/binman/test/224_fit_bad_oper.dts line 25

Quoted 5 lines
tools/binman/test/224_fit_bad_oper.dts +0 -2
@@ -21,7 +21,5 @@
 				};
 			};
 		};
-		fdtmap {
-		};

This looked unrelated at first, but as I understand it, changing the order of things made the bad-operation error happen later and exposed a breakage from the fdtmap entry trying to parse the mock dtb data.

Quoted 2 lines
tools/binman/test/224_fit_bad_oper.dts +0 -0
 	};
 };
#21 lore raw

Re: [PATCH v2 10/25] binman: Refactor fit to generate output at the end

Hi Alper,

Quoted 9 lines · Alper Nebi Yasak
Quoted 7 lines · Simon Glass

At present the fit implementation creates the output tree while scanning the FIT description. Then it updates the tree later when the data is known.

This works, but is a bit confusing, since it requires mixing the scanning code with the generation code, with a fix-up step at the end.

Quoted 7 lines
Quoted 3 lines

It is actually possible to do this in two phases, one to scan everything and the other to generate the FIT. Thus the FIT is generated in one pass, when everything is known.

Doing it in one go makes sense to me as well. In general I like the way distinct processing actions/steps are being split into their own blocks or so, and I think this helps move things toward that.

OK good.

Quoted 64 lines
Quoted 17 lines

Update the code accordingly. The only functional change is that the 'data' property for each node are now last instead of first, which is really a more natural position. Update the affected test to deal with this.

One wrinkle is that the calculated properties (image-pos, size and offset) are now added before the FIT is generated. so we must filter these out when copying properties from the binman description to the FIT.

Most of the change here is splitting out some of the code from the ReadEntries() implementation into _BuildInput(). So despite the large diff, most of the code is the same. It is not feasible to split this patch up, so far as I can tell.

---

Quoted 7 lines

Changes in v2:
- Add new patch to refactor fit to generate output at the end

 tools/binman/etype/fit.py              | 178 ++++++++++++++-----------
 tools/binman/ftest.py                  |  13 +-
 tools/binman/test/224_fit_bad_oper.dts |   2 -
 3 files changed, 109 insertions(+), 84 deletions(-)

I still wrote some weird ideas below, mostly for the future, since this patch is mostly moving code around which is fine as is.

Quoted 7 lines
tools/binman/etype/fit.py +0 -0
index 2d4c5f6545..61c72780e9 100644
@@ -209,6 +209,81 @@ class Entry_fit(Entry_section):
         return oper
 

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 220

Quoted 10 lines
tools/binman/etype/fit.py +9 -0
     def ReadEntries(self):
+        def _add_entries(base_node, depth, node):
+            """Add entries for any nodes that need them
+
+            Args:
+                base_node: Base Node of the FIT (with 'description' property)
+                depth: Current node depth (0 is the base 'fit' node)
+                node: Current node to process
+
+            He we only need to provide binman entries which are used to define

He -> Here ?

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 233

Quoted 13 lines
tools/binman/etype/fit.py +13 -0
+            the 'data' for each image. We create an entry_Section for each.
+            """
+            rel_path = node.path[len(base_node.path):]
+            in_images = rel_path.startswith('/images')
+            has_images = depth == 2 and in_images
+            if has_images:
+                # This node is a FIT subimage node (e.g. "/images/kernel")
+                # containing content nodes. We collect the subimage nodes and
+                # section entries for them here to merge the content subnodes
+                # together and put the merged contents in the subimage node's
+                # 'data' property later.
+                entry = Entry.Create(self.section, node, etype='section')
+                entry.ReadNode()
Quoted 2 lines

I plan to change 'self.section' to 'self' here later, fixes extracting wrong contents for FIT subentries.

OK

Quoted 8 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 237

Quoted 5 lines
tools/binman/etype/fit.py +4 -0
+                # The hash subnodes here are for mkimage, not binman.
+                entry.SetUpdateHash(False)
+                self._entries[rel_path] = entry
+

I also plan to change this to a single-level node name instead of the relative path, lets 'binman extract fit/u-boot' etc. run at all.

That would be good.

Quoted 9 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 241

Quoted 5 lines
tools/binman/etype/fit.py +4 -0
+            for subnode in node.subnodes:
+                _add_entries(base_node, depth + 1, subnode)
+
+        _add_entries(self._node, 0, self._node)

I think it's especially visible here what I meant by switching away from
recursion: this recurses through every node but only does anything on immediate subnodes of "/images" (for now?).

Fair enough.

Quoted 31 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 251

Quoted 11 lines
tools/binman/etype/fit.py +10 -0
+
+    def BuildSectionData(self, required):
+        """Build FIT entry contents
+
+        This adds the 'data' properties to the input ITB (Image-tree Binary)
+        then runs mkimage to process it.
+
+        Args:
+            required: True if the data must be present, False if it is OK to
+                return None

I forgot to handle 'required' while converting FIT to section...

Quoted 17 lines
tools/binman/etype/fit.py +17 -0
+
+        Returns:
+            Contents of the section (bytes)
+        """
+        data = self._BuildInput()
+        uniq = self.GetUniqueName()
+        input_fname = tools.get_output_filename('%s.itb' % uniq)
+        output_fname = tools.get_output_filename('%s.fit' % uniq)
+        tools.write_file(input_fname, data)
+        tools.write_file(output_fname, data)
+
+        args = {}
+        ext_offset = self._fit_props.get('fit,external-offset')
+        if ext_offset is not None:
+            args = {
+                'external': True,
+                'pad': fdt_util.fdt32_to_cpu(ext_offset.value)
Quoted 8 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 271

Quoted 3 lines
tools/binman/etype/fit.py +3 -0
+                }
+        if self.mkimage.run(reset_timestamp=True, output_fname=output_fname,
+                            **args) is None:

I have an idea for the far future, to let /image/* nodes sometimes be Entry_collection to handle external offsets in binman so we can take mkimage completely out of this, but no clue how feasible/desirable that end goal is.

Well I would really prefer to avoid duplicating mkimage functionality. I think it is something to look at when binman is more widespread, but as you can see we are still sorting out the FIT issues.

Quoted 19 lines
Quoted 19 lines
tools/binman/etype/fit.py +15 -0
+            # Bintool is missing; just use empty data as the output
+            self.record_missing_bintool(self.mkimage)
+            return tools.get_bytes(0, 1024)
+
+        return tools.read_file(output_fname)
+
+    def _BuildInput(self):
+        """Finish the FIT by adding the 'data' properties to it
+
+        Arguments:
+            fdt: FIT to update
+
+        Returns:
+            New fdt contents (bytes)
+        """
         def _process_prop(pname, prop):
             """Process special properties
 
Quoted 14 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 319

Quoted 10 lines
tools/binman/etype/fit.py +6 -0
@@ -236,9 +311,15 @@ class Entry_fit(Entry_section):
                     val = val[1:].replace('DEFAULT-SEQ', str(seq + 1))
                     fsw.property_string(pname, val)
                     return
+            elif pname.startswith('fit,'):
+                # Ignore these, which are commands for binman to process
+                return
+            elif pname in ['offset', 'size', 'image-pos']:
+                # Don't add binman's calculated properties
+                return

This is one of the things I was thinking of doing, thanks. I encountered the same issue when replacing a FIT entry with the same f"{uniq}.fit" that was used to build it, will try adding a test for that later.

Yes good.

Quoted 10 lines
Quoted 10 lines
tools/binman/etype/fit.py +1 -1
             fsw.property(pname, prop.bytes)
 
-        def _scan_gen_fdt_nodes(subnode, depth, in_images):
+        def _gen_fdt_nodes(subnode, depth, in_images):
             """Generate FDT nodes
 
             This creates one node for each member of self._fdts using the
[...]
Quoted 14 lines

tools/binman/test/224_fit_bad_oper.dts line 25

Quoted 10 lines
tools/binman/test/224_fit_bad_oper.dts +0 -2
index cee801e2ea..8a8014ea33 100644
@@ -21,7 +21,5 @@
                              };
                      };
              };
-             fdtmap {
-             };

This looked unrelated at first, but as I understand it, changing the order of things made the bad-operation error happen later and exposed a breakage from the fdtmap entry trying to parse the mock dtb data.

Yes, that's it.

Regards,
Simon

#22 lore raw

PATCH v2 11/25 binman: Rename tools parameter to btools

In reply to Simon Glass

This shadows the patman.tools library so rename it to avoid a pylint warning.

---

(no changes since v1)

 tools/binman/entry.py            | 4 ++--
 tools/binman/etype/fit.py        | 6 +++---
 tools/binman/etype/gbb.py        | 4 ++--
 tools/binman/etype/intel_ifwi.py | 4 ++--
 tools/binman/etype/mkimage.py    | 4 ++--
 tools/binman/etype/section.py    | 4 ++--
 tools/binman/etype/vblock.py     | 4 ++--
 7 files changed, 15 insertions(+), 15 deletions(-)
tools/binman/entry.py +2 -2
index 90a40bb274..00a13c5839 100644
@@ -1099,11 +1099,11 @@ features to produce new behaviours.
         """
         pass
 
-    def AddBintools(self, tools):
+    def AddBintools(self, btools):
         """Add the bintools used by this entry type
 
         Args:
-            tools (dict of Bintool):
+            btools (dict of Bintool):
         """
         pass
 
tools/binman/etype/fit.py +3 -3
index 61c72780e9..2b82955226 100644
@@ -484,6 +484,6 @@ class Entry_fit(Entry_section):
             section.SetOffsetSize(offset, size)
             section.SetImagePos(self.image_pos)
 
-    def AddBintools(self, tools):
-        super().AddBintools(tools)
-        self.mkimage = self.AddBintool(tools, 'mkimage')
+    def AddBintools(self, btools):
+        super().AddBintools(btools)
+        self.mkimage = self.AddBintool(btools, 'mkimage')
tools/binman/etype/gbb.py +2 -2
index e32fae27ce..7394e4e5d3 100644
@@ -99,5 +99,5 @@ class Entry_gbb(Entry):
 
         return True
 
-    def AddBintools(self, tools):
-        self.futility = self.AddBintool(tools, 'futility')
+    def AddBintools(self, btools):
+        self.futility = self.AddBintool(btools, 'futility')
tools/binman/etype/intel_ifwi.py +2 -2
index 46bdf116e6..4fa7d636fe 100644
@@ -143,5 +143,5 @@ class Entry_intel_ifwi(Entry_blob_ext):
             for entry in self._ifwi_entries.values():
                 entry.WriteSymbols(self)
 
-    def AddBintools(self, tools):
-        self.ifwitool = self.AddBintool(tools, 'ifwitool')
+    def AddBintools(self, btools):
+        self.ifwitool = self.AddBintool(btools, 'ifwitool')
tools/binman/etype/mkimage.py +2 -2
index c28141ff69..9f4717e4ea 100644
@@ -93,5 +93,5 @@ class Entry_mkimage(Entry):
         for entry in self._mkimage_entries.values():
             entry.CheckFakedBlobs(faked_blobs_list)
 
-    def AddBintools(self, tools):
-        self.mkimage = self.AddBintool(tools, 'mkimage')
+    def AddBintools(self, btools):
+        self.mkimage = self.AddBintool(btools, 'mkimage')
tools/binman/etype/section.py +2 -2
index f2d401d95b..16ee5ed723 100644
@@ -894,6 +894,6 @@ class Entry_section(Entry):
         for entry in self._entries.values():
             entry.CheckAltFormats(alt_formats)
 
-    def AddBintools(self, tools):
+    def AddBintools(self, btools):
         for entry in self._entries.values():
-            entry.AddBintools(tools)
+            entry.AddBintools(btools)
tools/binman/etype/vblock.py +2 -2
index a1de98290b..065b6ed2f6 100644
@@ -97,5 +97,5 @@ class Entry_vblock(Entry_collection):
         data = self.GetVblock(True)
         return self.ProcessContentsUpdate(data)
 
-    def AddBintools(self, tools):
-        self.futility = self.AddBintool(tools, 'futility')
+    def AddBintools(self, btools):
+        self.futility = self.AddBintool(btools, 'futility')
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 11/25
#23 lore raw

Re: [PATCH v2 11/25] binman: Rename tools parameter to btools

Quoted 8 lines · Simon Glass

This shadows the patman.tools library so rename it to avoid a pylint warning.

---

(no changes since v1)

Quoted 8 lines
 tools/binman/entry.py            | 4 ++--
 tools/binman/etype/fit.py        | 6 +++---
 tools/binman/etype/gbb.py        | 4 ++--
 tools/binman/etype/intel_ifwi.py | 4 ++--
 tools/binman/etype/mkimage.py    | 4 ++--
 tools/binman/etype/section.py    | 4 ++--
 tools/binman/etype/vblock.py     | 4 ++--
 7 files changed, 15 insertions(+), 15 deletions(-)
#24 lore raw

PATCH v2 12/25 binman: Change how faked blobs are created

In reply to Simon Glass

At present fake blobs are created but internally an empty blob is used. Change it to use the contents of the faked file. Also return whether the blob was faked, in case the caller needs to know that.

Add a TODO to put fake blobs in their own directory.

---

Changes in v2:
- Add a patch to change how faked blobs are created

 tools/binman/binman.rst             | 3 ++-
 tools/binman/entry.py               | 9 ++++++---
 tools/binman/etype/blob.py          | 7 ++++---
 tools/binman/etype/blob_ext_list.py | 2 +-
 4 files changed, 13 insertions(+), 8 deletions(-)
tools/binman/binman.rst +2 -1
index 85f8337a31..a90364f2fb 100644
@@ -1500,7 +1500,8 @@ Some ideas:
 - Figure out how to make Fdt support changing the node order, so that
   Node.AddSubnode() can support adding a node before another, existing node.
   Perhaps it should completely regenerate the flat tree?
-
+- Put faked files into a separate subdir and remove them on start-up, to avoid
+  seeing them as 'real' files on a subsequent run
 
 --
 Simon Glass <[email protected]>
tools/binman/entry.py +6 -3
index 00a13c5839..2fb0050da5 100644
@@ -997,15 +997,18 @@ features to produce new behaviours.
             fname (str): Filename to check
 
         Returns:
-            fname (str): Filename of faked file
+            tuple:
+                fname (str): Filename of faked file
+                bool: True if the blob was faked, False if not
         """
         if self.allow_fake and not pathlib.Path(fname).is_file():
             outfname = tools.get_output_filename(os.path.basename(fname))
             with open(outfname, "wb") as out:
                 out.truncate(1024)
             self.faked = True
-            return outfname
-        return fname
+            tout.info(f"Entry '{self._node.path}': Faked file '{outfname}'")
+            return outfname, True
+        return fname, False
 
     def CheckFakedBlobs(self, faked_blobs_list):
         """Check if any entries in this section have faked external blobs
tools/binman/etype/blob.py +4 -3
index 25ec5d26c9..89f089e740 100644
@@ -41,10 +41,11 @@ class Entry_blob(Entry):
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
         if not self._pathname:
-            self._pathname = self.check_fake_fname(self._filename)
-            self.SetContents(b'')
+            self._pathname, faked = self.check_fake_fname(self._filename)
             self.missing = True
-            return True
+            if not faked:
+                self.SetContents(b'')
+                return True
 
         self.ReadBlobContents()
         return True
tools/binman/etype/blob_ext_list.py +1 -1
index 76ad32a1ee..f00202e9eb 100644
@@ -37,7 +37,7 @@ class Entry_blob_ext_list(Entry_blob):
         missing = False
         pathnames = []
         for fname in self._filenames:
-            fname = self.check_fake_fname(fname)
+            fname, _ = self.check_fake_fname(fname)
             pathname = tools.get_input_filename(
                 fname, self.external and self.section.GetAllowMissing())
             # Allow the file to be missing
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 12/25
#25 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Quoted 9 lines · Simon Glass

At present fake blobs are created but internally an empty blob is used. Change it to use the contents of the faked file. Also return whether the blob was faked, in case the caller needs to know that.

Add a TODO to put fake blobs in their own directory.

---

Quoted 8 lines

Changes in v2:
- Add a patch to change how faked blobs are created

 tools/binman/binman.rst             | 3 ++-
 tools/binman/entry.py               | 9 ++++++---
 tools/binman/etype/blob.py          | 7 ++++---
 tools/binman/etype/blob_ext_list.py | 2 +-
 4 files changed, 13 insertions(+), 8 deletions(-)

I feel a bit uneasy about all this fake file stuff, but can't figure out exactly why. I guess the patch doesn't make it that worse.

tools/binman/binman.rst Some ideas: · line 1504

Quoted 11 lines
tools/binman/binman.rst +2 -1
index 85f8337a31..a90364f2fb 100644
@@ -1500,7 +1500,8 @@ Some ideas:
 - Figure out how to make Fdt support changing the node order, so that
   Node.AddSubnode() can support adding a node before another, existing node.
   Perhaps it should completely regenerate the flat tree?
-
+- Put faked files into a separate subdir and remove them on start-up, to avoid
+  seeing them as 'real' files on a subsequent run

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

Also, maybe it's better to create them somewhere in the /tmp/binman.* directories so they disappear without effort.

Quoted 16 lines
tools/binman/binman.rst +0 -0
 --
 Simon Glass <[email protected]>
tools/binman/entry.py +3 -1
index 00a13c5839..2fb0050da5 100644
@@ -997,15 +997,18 @@ features to produce new behaviours.
             fname (str): Filename to check
 
         Returns:
-            fname (str): Filename of faked file
+            tuple:
+                fname (str): Filename of faked file
+                bool: True if the blob was faked, False if not
         """

tools/binman/entry.py features to produce new behaviours. · line 1011

Quoted 10 lines
tools/binman/entry.py +3 -2
         if self.allow_fake and not pathlib.Path(fname).is_file():
             outfname = tools.get_output_filename(os.path.basename(fname))
             with open(outfname, "wb") as out:
                 out.truncate(1024)
             self.faked = True
-            return outfname
-        return fname
+            tout.info(f"Entry '{self._node.path}': Faked file '{outfname}'")
+            return outfname, True
+        return fname, False

Can't callers use self.faked for this?

I think I see an edge case when calling this multiple times for the same filename, only the first call would recognize it being a fake file and only the first-entry-to-call would consider itself faked.

Quoted 24 lines
tools/binman/entry.py +0 -0
     def CheckFakedBlobs(self, faked_blobs_list):
         """Check if any entries in this section have faked external blobs
tools/binman/etype/blob.py +4 -3
index 25ec5d26c9..89f089e740 100644
@@ -41,10 +41,11 @@ class Entry_blob(Entry):
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
         if not self._pathname:
-            self._pathname = self.check_fake_fname(self._filename)
-            self.SetContents(b'')
+            self._pathname, faked = self.check_fake_fname(self._filename)
             self.missing = True
-            return True
+            if not faked:
+                self.SetContents(b'')
+                return True
 
         self.ReadBlobContents()
         return True
 
[...]
#26 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Hi Alper,

Quoted 14 lines · Alper Nebi Yasak
Quoted 12 lines · Simon Glass

At present fake blobs are created but internally an empty blob is used. Change it to use the contents of the faked file. Also return whether the blob was faked, in case the caller needs to know that.

Add a TODO to put fake blobs in their own directory.

---

Changes in v2:
- Add a patch to change how faked blobs are created

Quoted 8 lines
Quoted 5 lines
 tools/binman/binman.rst             | 3 ++-
 tools/binman/entry.py               | 9 ++++++---
 tools/binman/etype/blob.py          | 7 ++++---
 tools/binman/etype/blob_ext_list.py | 2 +-
 4 files changed, 13 insertions(+), 8 deletions(-)

I feel a bit uneasy about all this fake file stuff, but can't figure out exactly why. I guess the patch doesn't make it that worse.

Well we have at least one major problem with it, which is that fake blobs need to exist only for the life of the binman run.

Quoted 17 lines

tools/binman/binman.rst Some ideas: · line 1504

Quoted 11 lines
tools/binman/binman.rst +2 -1
index 85f8337a31..a90364f2fb 100644
@@ -1500,7 +1500,8 @@ Some ideas:
 - Figure out how to make Fdt support changing the node order, so that
   Node.AddSubnode() can support adding a node before another, existing node.
   Perhaps it should completely regenerate the flat tree?
-
+- Put faked files into a separate subdir and remove them on start-up, to avoid
+  seeing them as 'real' files on a subsequent run

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

We need the files for mkimage.

Quoted 3 lines

Also, maybe it's better to create them somewhere in the /tmp/binman.* directories so they disappear without effort.

Yes that would be better. I decided to stop this series where it is since that can be done in a follow-up. I did add a TODO though.

Quoted 30 lines
Quoted 17 lines
tools/binman/binman.rst +0 -0
 --
 Simon Glass <[email protected]>
tools/binman/entry.py +3 -1
index 00a13c5839..2fb0050da5 100644
@@ -997,15 +997,18 @@ features to produce new behaviours.
             fname (str): Filename to check
 
         Returns:
-            fname (str): Filename of faked file
+            tuple:
+                fname (str): Filename of faked file
+                bool: True if the blob was faked, False if not
         """

tools/binman/entry.py features to produce new behaviours. · line 1011

Quoted 10 lines
tools/binman/entry.py +3 -2
         if self.allow_fake and not pathlib.Path(fname).is_file():
             outfname = tools.get_output_filename(os.path.basename(fname))
             with open(outfname, "wb") as out:
                 out.truncate(1024)
             self.faked = True
-            return outfname
-        return fname
+            tout.info(f"Entry '{self._node.path}': Faked file '{outfname}'")
+            return outfname, True
+        return fname, False

Can't callers use self.faked for this?

Quoted 3 lines

I think I see an edge case when calling this multiple times for the same filename, only the first call would recognize it being a fake file and only the first-entry-to-call would consider itself faked.

Yes we could and yes there is an edge case. This function returns True if it created the fake, False if it had happened before. I decided to leave it as is since checking self.faked after a function that sets it seems like a side effect. No strong opinions on it though.

Quoted 21 lines
Quoted 21 lines
tools/binman/entry.py +0 -0
     def CheckFakedBlobs(self, faked_blobs_list):
         """Check if any entries in this section have faked external blobs
tools/binman/etype/blob.py +4 -3
index 25ec5d26c9..89f089e740 100644
@@ -41,10 +41,11 @@ class Entry_blob(Entry):
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
         if not self._pathname:
-            self._pathname = self.check_fake_fname(self._filename)
-            self.SetContents(b'')
+            self._pathname, faked = self.check_fake_fname(self._filename)
             self.missing = True
-            return True
+            if not faked:
+                self.SetContents(b'')
+                return True
 
Quoted 4 lines

tools/binman/etype/blob.py class Entry_blob(Entry):

Quoted 4 lines
tools/binman/etype/blob.py +0 -0
         self.ReadBlobContents()
         return True
 
[...]

Regards,
Simon

#27 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Quoted 9 lines · Simon Glass
Quoted 6 lines · Alper Nebi Yasak
Quoted 2 lines · Simon Glass
+- Put faked files into a separate subdir and remove them on start-up, to avoid
+  seeing them as 'real' files on a subsequent run

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

We need the files for mkimage.

But the mkimage etype creates its own files to pass to mkimage anyway, using entry.GetData() in collect_contents_to_file() without reaching into subentries' files.

More specifically, what I'm thinking of is removing check_fake_fname() entirely, instead of managing/removing the files it creates. At a cursory glance, only blob and blob-ext-list seem to use that directly.

#28 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Hi Alper,

Quoted 12 lines · Alper Nebi Yasak
Quoted 9 lines · Simon Glass
Quoted 6 lines · Alper Nebi Yasak
Quoted 2 lines · Simon Glass
+- Put faked files into a separate subdir and remove them on start-up, to avoid
+  seeing them as 'real' files on a subsequent run

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

We need the files for mkimage.

Quoted 7 lines

But the mkimage etype creates its own files to pass to mkimage anyway, using entry.GetData() in collect_contents_to_file() without reaching into subentries' files.

More specifically, what I'm thinking of is removing check_fake_fname() entirely, instead of managing/removing the files it creates. At a cursory glance, only blob and blob-ext-list seem to use that directly.

OK, but if the files don't get created, then mkimage will fail, right?

Regards,
Simon

#29 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Quoted 12 lines · Simon Glass
Quoted 11 lines · Alper Nebi Yasak
Quoted 5 lines · Simon Glass
Quoted 2 lines · Alper Nebi Yasak

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

We need the files for mkimage.

But the mkimage etype creates its own files to pass to mkimage anyway, using entry.GetData() in collect_contents_to_file() without reaching into subentries' files.

Quoted 5 lines
Quoted 3 lines

More specifically, what I'm thinking of is removing check_fake_fname() entirely, instead of managing/removing the files it creates. At a cursory glance, only blob and blob-ext-list seem to use that directly.

OK, but if the files don't get created, then mkimage will fail, right?

I don't think it will, but maybe it's better that I try removing it to see if anything breaks.

#30 lore raw

Re: [PATCH v2 12/25] binman: Change how faked blobs are created

Hi Alper,

Quoted 14 lines · Alper Nebi Yasak
Quoted 12 lines · Simon Glass
Quoted 11 lines · Alper Nebi Yasak
Quoted 5 lines · Simon Glass
Quoted 2 lines · Alper Nebi Yasak

Do we need to create actual files, or is it a convenience thing for blob entry types (because they already read their contents from files)?

We need the files for mkimage.

But the mkimage etype creates its own files to pass to mkimage anyway, using entry.GetData() in collect_contents_to_file() without reaching into subentries' files.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

More specifically, what I'm thinking of is removing check_fake_fname() entirely, instead of managing/removing the files it creates. At a cursory glance, only blob and blob-ext-list seem to use that directly.

OK, but if the files don't get created, then mkimage will fail, right?

I don't think it will, but maybe it's better that I try removing it to see if anything breaks.

Yes, the reason for adding faked blobs was needing to run mkimage which expects files (in the .its or elsewhere) to be present.

Regards,
Simon

#31 lore raw

PATCH v2 13/25 binman: Make fake blobs zero-sized by default

In reply to Simon Glass

On x86 devices having even a small amount of data can cause an overlap between regions. For example, bayleybay complains when the intel-vga region overlaps with u-boot-ucode:

ImagePos Offset Size Name
<none> 00000000 00800000 main-section

   <none>     ff800000  00000080  intel-descriptor
   <none>     ff800400  00000080  intel-me
   <none>     fff00000  00098f24  u-boot-with-ucode-ptr
   <none>     fff98f24  00001aa0  u-boot-dtb-with-ucode
   <none>     fff9a9d0  0002a000  u-boot-ucode
   <none>     fffb0000  00000080  intel-vga

...

It is safer to use an empty file in most cases. Add an option to set the size for those uses that need it.

---

Changes in v2:
- Add a patch to make fake blobs zero-sized by default

 tools/binman/entry.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/binman/entry.py +3 -2
index 2fb0050da5..e4c0fbe23d 100644
@@ -988,13 +988,14 @@ features to produce new behaviours.
         if self.missing:
             missing_list.append(self)
 
-    def check_fake_fname(self, fname):
+    def check_fake_fname(self, fname, size=0):
         """If the file is missing and the entry allows fake blobs, fake it
 
         Sets self.faked to True if faked
 
         Args:
             fname (str): Filename to check
+            size (int): Size of fake file to create
 
         Returns:
             tuple:
@@ -1004,7 +1005,7 @@ features to produce new behaviours.
         if self.allow_fake and not pathlib.Path(fname).is_file():
             outfname = tools.get_output_filename(os.path.basename(fname))
             with open(outfname, "wb") as out:
-                out.truncate(1024)
+                out.truncate(size)
             self.faked = True
             tout.info(f"Entry '{self._node.path}': Faked file '{outfname}'")
             return outfname, True
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 13/25
#32 lore raw

Re: [PATCH v2 13/25] binman: Make fake blobs zero-sized by default

tools/binman/entry.py

Quoted 26 lines · Simon Glass

On x86 devices having even a small amount of data can cause an overlap between regions. For example, bayleybay complains when the intel-vga region overlaps with u-boot-ucode:

   ImagePos    Offset      Size  Name
   <none>    00000000  00800000  main-section
   <none>     ff800000  00000080  intel-descriptor
   <none>     ff800400  00000080  intel-me
   <none>     fff00000  00098f24  u-boot-with-ucode-ptr
   <none>     fff98f24  00001aa0  u-boot-dtb-with-ucode
   <none>     fff9a9d0  0002a000  u-boot-ucode
   <none>     fffb0000  00000080  intel-vga
   ...

It is safer to use an empty file in most cases. Add an option to set the size for those uses that need it.

---

Changes in v2:
- Add a patch to make fake blobs zero-sized by default

 tools/binman/entry.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/binman/entry.py +0 -0

Later on, I think adding 'min-size' and 'max-size' properties for entries could be useful for a few things including this.

#33 lore raw

Re: [PATCH v2 13/25] binman: Make fake blobs zero-sized by default

Hi Alper,

Quoted 25 lines · Alper Nebi Yasak
Quoted 23 lines · Simon Glass

On x86 devices having even a small amount of data can cause an overlap between regions. For example, bayleybay complains when the intel-vga region overlaps with u-boot-ucode:

   ImagePos    Offset      Size  Name
   <none>    00000000  00800000  main-section
   <none>     ff800000  00000080  intel-descriptor
   <none>     ff800400  00000080  intel-me
   <none>     fff00000  00098f24  u-boot-with-ucode-ptr
   <none>     fff98f24  00001aa0  u-boot-dtb-with-ucode
   <none>     fff9a9d0  0002a000  u-boot-ucode
   <none>     fffb0000  00000080  intel-vga
   ...

It is safer to use an empty file in most cases. Add an option to set the size for those uses that need it.

---

Changes in v2:
- Add a patch to make fake blobs zero-sized by default

Quoted 8 lines

tools/binman/entry.py

Quoted 3 lines
 tools/binman/entry.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/binman/entry.py +0 -0

Later on, I think adding 'min-size' and 'max-size' properties for entries could be useful for a few things including this.

Yes I think that might become necessary.

Regards,
Simon

#34 lore raw

PATCH v2 14/25 binman: Allow mkimage to use a non-zero fake-blob size

In reply to Simon Glass

Unfortunately mkimage gets upset with zero-sized files. Update the ObtainContents() method to support specifying the size, if a fake blob is created.

---

Changes in v2:
- Add a patch to allow mkimage to use a non-zero fake-blob size

 tools/binman/entry.py                     | 11 ++++++++---
 tools/binman/etype/_testing.py            |  2 +-
 tools/binman/etype/blob.py                |  5 +++--
 tools/binman/etype/mkimage.py             | 13 ++++++++++++-
 tools/binman/etype/section.py             |  2 +-
 tools/binman/ftest.py                     | 10 ++++++++++
 tools/binman/test/229_mkimage_missing.dts | 18 ++++++++++++++++++
 7 files changed, 53 insertions(+), 8 deletions(-)
 create mode 100644 tools/binman/test/229_mkimage_missing.dts
tools/binman/entry.py +8 -3
index e4c0fbe23d..249f117ace 100644
@@ -415,9 +415,13 @@ class Entry(object):
         self.SetContents(data)
         return size_ok
 
-    def ObtainContents(self):
+    def ObtainContents(self, skip_entry=None, fake_size=0):
         """Figure out the contents of an entry.
 
+        Args:
+            skip_entry (Entry): Entry to skip when obtaining section contents
+            fake_size (int): Size of fake file to create if needed
+
         Returns:
             True if the contents were found, False if another call is needed
             after the other entries are processed.
@@ -1130,12 +1134,13 @@ features to produce new behaviours.
         """
         self.update_hash = update_hash
 
-    def collect_contents_to_file(self, entries, prefix):
+    def collect_contents_to_file(self, entries, prefix, fake_size=0):
         """Put the contents of a list of entries into a file
 
         Args:
             entries (list of Entry): Entries to collect
             prefix (str): Filename prefix of file to write to
+            fake_size (int): Size of fake file to create if needed
 
         If any entry does not have contents yet, this function returns False
         for the data.
@@ -1150,7 +1155,7 @@ features to produce new behaviours.
         for entry in entries:
             # First get the input data and put it in a file. If not available,
             # try later.
-            if not entry.ObtainContents():
+            if not entry.ObtainContents(fake_size=fake_size):
                 return None, None, None
             data += entry.GetData()
         uniq = self.GetUniqueName()
tools/binman/etype/_testing.py +1 -1
index 0800c25899..5089de3642 100644
@@ -82,7 +82,7 @@ class Entry__testing(Entry):
         self.return_contents = True
         self.contents = b'aa'
 
-    def ObtainContents(self):
+    def ObtainContents(self, fake_size=0):
         if self.return_unknown_contents or not self.return_contents:
             return False
         if self.return_contents_later:
tools/binman/etype/blob.py +3 -2
index 89f089e740..ceaefb07b7 100644
@@ -35,13 +35,14 @@ class Entry_blob(Entry):
         super().__init__(section, etype, node)
         self._filename = fdt_util.GetString(self._node, 'filename', self.etype)
 
-    def ObtainContents(self):
+    def ObtainContents(self, fake_size=0):
         self._filename = self.GetDefaultFilename()
         self._pathname = tools.get_input_filename(self._filename,
             self.external and self.section.GetAllowMissing())
         # Allow the file to be missing
         if not self._pathname:
-            self._pathname, faked = self.check_fake_fname(self._filename)
+            self._pathname, faked = self.check_fake_fname(self._filename,
+                                                          fake_size)
             self.missing = True
             if not faked:
                 self.SetContents(b'')
tools/binman/etype/mkimage.py +12 -1
index 9f4717e4ea..5f6def2287 100644
@@ -51,8 +51,9 @@ class Entry_mkimage(Entry):
         self.ReadEntries()
 
     def ObtainContents(self):
+        # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage')
+            self._mkimage_entries.values(), 'mkimage', 1024)
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
@@ -73,6 +74,16 @@ class Entry_mkimage(Entry):
             entry.ReadNode()
             self._mkimage_entries[entry.name] = entry
 
+    def SetAllowMissing(self, allow_missing):
+        """Set whether a section allows missing external blobs
+
+        Args:
+            allow_missing: True if allowed, False if not allowed
+        """
+        self.allow_missing = allow_missing
+        for entry in self._mkimage_entries.values():
+            entry.SetAllowMissing(allow_missing)
+
     def SetAllowFakeBlob(self, allow_fake):
         """Set whether the sub nodes allows to create a fake blob
 
tools/binman/etype/section.py +1 -1
index 16ee5ed723..fab42c20f7 100644
@@ -246,7 +246,7 @@ class Entry_section(Entry):
         for entry in self._entries.values():
             entry.AddMissingProperties(have_image_pos)
 
-    def ObtainContents(self, skip_entry=None):
+    def ObtainContents(self, fake_size=0, skip_entry=None):
         return self.GetEntryContents(skip_entry=skip_entry)
 
     def GetPaddedDataForEntry(self, entry, entry_data):
tools/binman/ftest.py +10 -0
index 4eac913d06..106027d245 100644
@@ -5330,6 +5330,16 @@ fdt         fdtmap                Extract the devicetree blob from the fdtmap
             "Node '/binman/u-boot': Please use 'extend-size' instead of 'expand-size'",
             str(e.exception))
 
+    def testMkimageMissingBlob(self):
+        """Test using mkimage to build an image"""
+        with test_util.capture_sys_output() as (stdout, stderr):
+            self._DoTestFile('229_mkimage_missing.dts', allow_missing=True,
+                             allow_fake_blobs=True)
+        err = stderr.getvalue()
+        self.assertRegex(
+            err,
+            "Image '.*' has faked external blobs and is non-functional: .*")
+
 
 if __name__ == "__main__":
     unittest.main()
tools/binman/test/229_mkimage_missing.dts new file +18 -0
new file mode 100644
index 0000000000..54a5a6c571
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0+
+
+/dts-v1/;
+
+/ {
+	#address-cells = <1>;
+	#size-cells = <1>;
+
+	binman {
+		mkimage {
+			args = "-n test -T script";
+
+			blob-ext {
+				filename = "missing.bin";
+			};
+		};
+	};
+};
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 14/25
#35 lore raw

Re: [PATCH v2 14/25] binman: Allow mkimage to use a non-zero fake-blob size

Quoted 10 lines · Simon Glass

Unfortunately mkimage gets upset with zero-sized files. Update the ObtainContents() method to support specifying the size, if a fake blob is created.

---

Changes in v2:
- Add a patch to allow mkimage to use a non-zero fake-blob size

Quoted 9 lines
 tools/binman/entry.py                     | 11 ++++++++---
 tools/binman/etype/_testing.py            |  2 +-
 tools/binman/etype/blob.py                |  5 +++--
 tools/binman/etype/mkimage.py             | 13 ++++++++++++-
 tools/binman/etype/section.py             |  2 +-
 tools/binman/ftest.py                     | 10 ++++++++++
 tools/binman/test/229_mkimage_missing.dts | 18 ++++++++++++++++++
 7 files changed, 53 insertions(+), 8 deletions(-)
 create mode 100644 tools/binman/test/229_mkimage_missing.dts

I don't exactly like this, but can't think of a nice alternative either.

Quoted 7 lines

[...]

tools/binman/etype/mkimage.py +0 -0
index 9f4717e4ea..5f6def2287 100644

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 56

Quoted 7 lines
tools/binman/etype/mkimage.py +2 -1
@@ -51,8 +51,9 @@ class Entry_mkimage(Entry):
         self.ReadEntries()
 
     def ObtainContents(self):
+        # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage')
+            self._mkimage_entries.values(), 'mkimage', 1024)

I kind of want to say that mkimage-the-etype should be able to handle here whatever it gets from subentries (maybe by writing a single-byte file itself), and mkimage-the-executable should be able to handle zero-size files, but I'm not confident in those opinions.

Quoted 4 lines
tools/binman/etype/mkimage.py +0 -0
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
[...]
#36 lore raw

Re: [PATCH v2 14/25] binman: Allow mkimage to use a non-zero fake-blob size

Hi Alper,

Quoted 35 lines · Alper Nebi Yasak
Quoted 10 lines · Simon Glass

Unfortunately mkimage gets upset with zero-sized files. Update the ObtainContents() method to support specifying the size, if a fake blob is created.

---

Changes in v2:
- Add a patch to allow mkimage to use a non-zero fake-blob size

Quoted 9 lines
 tools/binman/entry.py                     | 11 ++++++++---
 tools/binman/etype/_testing.py            |  2 +-
 tools/binman/etype/blob.py                |  5 +++--
 tools/binman/etype/mkimage.py             | 13 ++++++++++++-
 tools/binman/etype/section.py             |  2 +-
 tools/binman/ftest.py                     | 10 ++++++++++
 tools/binman/test/229_mkimage_missing.dts | 18 ++++++++++++++++++
 7 files changed, 53 insertions(+), 8 deletions(-)
 create mode 100644 tools/binman/test/229_mkimage_missing.dts

I don't exactly like this, but can't think of a nice alternative either.

Quoted 9 lines

[...]

tools/binman/etype/mkimage.py +0 -0
index 9f4717e4ea..5f6def2287 100644
@@ -51,8 +51,9 @@ class Entry_mkimage(Entry):
         self.ReadEntries()
 
Quoted 10 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 56

Quoted 5 lines
tools/binman/etype/mkimage.py +2 -1
     def ObtainContents(self):
+        # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage')
+            self._mkimage_entries.values(), 'mkimage', 1024)

I kind of want to say that mkimage-the-etype should be able to handle here whatever it gets from subentries (maybe by writing a single-byte file itself), and mkimage-the-executable should be able to handle zero-size files, but I'm not confident in those opinions.

Well the entry has no problem with missing files, so that should be OK.

For mkimage I agree it is a strange restriction. Perhaps we should just change it? I don't see what problem it could create.

Quoted 5 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 59

Quoted 5 lines
tools/binman/etype/mkimage.py +0 -0
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
[...]

Regards,
Simon

#37 lore raw

Re: [PATCH v2 14/25] binman: Allow mkimage to use a non-zero fake-blob size

Quoted 7 lines · Simon Glass
Quoted 6 lines · Alper Nebi Yasak
Quoted 5 lines

def ObtainContents(self):

tools/binman/etype/mkimage.py +2 -1
+        # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage')
+            self._mkimage_entries.values(), 'mkimage', 1024)
Quoted 6 lines
Quoted 4 lines

I kind of want to say that mkimage-the-etype should be able to handle here whatever it gets from subentries (maybe by writing a single-byte file itself), and mkimage-the-executable should be able to handle zero-size files, but I'm not confident in those opinions.

Well the entry has no problem with missing files, so that should be OK.

What I meant is when the non-faked input data ends up being zero-sized. A bit contrived, but this still triggers the error:

    mkimage {
        args = ...;

        blob-ext {
            filename = "/dev/null"; /* or any other zero-size file */
        };
    };

which might end up happening e.g. via tee-os with TEE=/dev/null, I remember someone doing that for one of the blobs in a mail but can't find it or recall any details.

Quoted 2 lines

For mkimage I agree it is a strange restriction. Perhaps we should just change it? I don't see what problem it could create.

I don't know either.

#38 lore raw

Re: [PATCH v2 14/25] binman: Allow mkimage to use a non-zero fake-blob size

Hi Alper,

Quoted 27 lines · Alper Nebi Yasak
Quoted 7 lines · Simon Glass
Quoted 6 lines · Alper Nebi Yasak
Quoted 5 lines

def ObtainContents(self):

tools/binman/etype/mkimage.py +2 -1
+        # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage')
+            self._mkimage_entries.values(), 'mkimage', 1024)
Quoted 6 lines
Quoted 4 lines

I kind of want to say that mkimage-the-etype should be able to handle here whatever it gets from subentries (maybe by writing a single-byte file itself), and mkimage-the-executable should be able to handle zero-size files, but I'm not confident in those opinions.

Well the entry has no problem with missing files, so that should be OK.

What I meant is when the non-faked input data ends up being zero-sized. A bit contrived, but this still triggers the error:

    mkimage {
        args = ...;

        blob-ext {
            filename = "/dev/null"; /* or any other zero-size file */
        };
    };
Quoted 3 lines

which might end up happening e.g. via tee-os with TEE=/dev/null, I remember someone doing that for one of the blobs in a mail but can't find it or recall any details.

Hmm OK. So long as the error message is useful then we should be OK. I suspect the /dev/null thing was for when TEE was missing, but binman handles that differently, so we should not use /dev/null anywhere with binman.

Quoted 5 lines
Quoted 3 lines

For mkimage I agree it is a strange restriction. Perhaps we should just change it? I don't see what problem it could create.

I don't know either.

I suppose one way to find out is to change it!

Regards,
Simon

#39 lore raw

PATCH v2 15/25 binman: Read the fit entries only once

In reply to Simon Glass

At present the entries are read twice, once by the entry_Section class and once by the FIT implementation. This is harmless but can be confusing when debugging. Fix it.

---

(no changes since v1)

 tools/binman/etype/fit.py | 1 -
 1 file changed, 1 deletion(-)
tools/binman/etype/fit.py +0 -1
index 2b82955226..7b0c94dfee 100644
@@ -185,7 +185,6 @@ class Entry_fit(Entry_section):
         self.mkimage = None
 
     def ReadNode(self):
-        self.ReadEntries()
         super().ReadNode()
 
     def _get_operation(self, subnode):
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 15/25
#40 lore raw

Re: [PATCH v2 15/25] binman: Read the fit entries only once

Quoted 11 lines · Simon Glass

At present the entries are read twice, once by the entry_Section class and once by the FIT implementation. This is harmless but can be confusing when debugging. Fix it.

---

(no changes since v1)

 tools/binman/etype/fit.py | 1 -
 1 file changed, 1 deletion(-)

This is actually necessary so that we can use 'self' as the section in Entry.Create(self, ...) when creating /image/* sections, which requires some section-related attributes set by super().ReadNode(). It was one of the things I was planning to fix as well, thanks!

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 188

Quoted 10 lines
tools/binman/etype/fit.py +0 -1
index 2b82955226..7b0c94dfee 100644
@@ -185,7 +185,6 @@ class Entry_fit(Entry_section):
         self.mkimage = None
 
     def ReadNode(self):
-        self.ReadEntries()
         super().ReadNode()

The entire function can be removed, but I think some stuff from __init__() actually belongs here so I'm fine with keeping it.

Quoted 2 lines
tools/binman/etype/fit.py +0 -0
     def _get_operation(self, subnode):
#41 lore raw

Re: [PATCH v2 15/25] binman: Read the fit entries only once

Hi Alper,

Quoted 11 lines · Alper Nebi Yasak
Quoted 9 lines · Simon Glass

At present the entries are read twice, once by the entry_Section class and once by the FIT implementation. This is harmless but can be confusing when debugging. Fix it.

---

(no changes since v1)

Quoted 7 lines
Quoted 2 lines
 tools/binman/etype/fit.py | 1 -
 1 file changed, 1 deletion(-)

This is actually necessary so that we can use 'self' as the section in Entry.Create(self, ...) when creating /image/* sections, which requires some section-related attributes set by super().ReadNode(). It was one of the things I was planning to fix as well, thanks!

OK good.

Quoted 10 lines
Quoted 7 lines
tools/binman/etype/fit.py +0 -0
index 2b82955226..7b0c94dfee 100644
@@ -185,7 +185,6 @@ class Entry_fit(Entry_section):
         self.mkimage = None
 
Quoted 6 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 188

Quoted 3 lines
tools/binman/etype/fit.py +0 -1
     def ReadNode(self):
-        self.ReadEntries()
         super().ReadNode()

The entire function can be removed, but I think some stuff from __init__() actually belongs here so I'm fine with keeping it.

Ah yes. I added a patch to move things as you say.

Quoted 3 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 190

Quoted 3 lines
tools/binman/etype/fit.py +0 -0
     def _get_operation(self, subnode):

Regards,
Simon

#42 lore raw

PATCH v2 16/25 binman: Fix some pylint warnings in fit

In reply to Simon Glass

Some warnings have crept in, so fix those that are easy to fix.

---

(no changes since v1)

 tools/binman/etype/fit.py | 50 +++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 25 deletions(-)
tools/binman/etype/fit.py +25 -25
index 7b0c94dfee..70966028e8 100644
@@ -2,10 +2,9 @@
 # Copyright (c) 2016 Google, Inc
 # Written by Simon Glass <[email protected]>
 #
-# Entry-type module for producing a FIT
-#
 
-from collections import defaultdict, OrderedDict
+"""Entry-type module for producing a FIT"""
+
 import libfdt
 
 from binman.entry import Entry, EntryArg
@@ -246,16 +245,16 @@ class Entry_fit(Entry_section):
         then runs mkimage to process it.
 
         Args:
-            required: True if the data must be present, False if it is OK to
-                return None
+            required (bool): True if the data must be present, False if it is OK
+                to return None
 
         Returns:
-            Contents of the section (bytes)
+            bytes: Contents of the section
         """
-        data = self._BuildInput()
+        data = self._build_input()
         uniq = self.GetUniqueName()
-        input_fname = tools.get_output_filename('%s.itb' % uniq)
-        output_fname = tools.get_output_filename('%s.fit' % uniq)
+        input_fname = tools.get_output_filename(f'{uniq}.itb')
+        output_fname = tools.get_output_filename(f'{uniq}.fit')
         tools.write_file(input_fname, data)
         tools.write_file(output_fname, data)
 
@@ -274,14 +273,14 @@ class Entry_fit(Entry_section):
 
         return tools.read_file(output_fname)
 
-    def _BuildInput(self):
+    def _build_input(self):
         """Finish the FIT by adding the 'data' properties to it
 
         Arguments:
             fdt: FIT to update
 
         Returns:
-            New fdt contents (bytes)
+            bytes: New fdt contents
         """
         def _process_prop(pname, prop):
             """Process special properties
@@ -303,9 +302,9 @@ class Entry_fit(Entry_section):
                     if not self._fit_default_dt:
                         self.Raise("Generated 'default' node requires default-dt entry argument")
                     if self._fit_default_dt not in self._fdts:
-                        self.Raise("default-dt entry argument '%s' not found in fdt list: %s" %
-                                   (self._fit_default_dt,
-                                    ', '.join(self._fdts)))
+                        self.Raise(
+                            f"default-dt entry argument '{self._fit_default_dt}' "
+                            f"not found in fdt list: {', '.join(self._fdts)}")
                     seq = self._fdts.index(self._fit_default_dt)
                     val = val[1:].replace('DEFAULT-SEQ', str(seq + 1))
                     fsw.property_string(pname, val)
@@ -352,8 +351,8 @@ class Entry_fit(Entry_section):
             else:
                 if self._fdts is None:
                     if self._fit_list_prop:
-                        self.Raise("Generator node requires '%s' entry argument" %
-                                   self._fit_list_prop.value)
+                        self.Raise('Generator node requires '
+                            f"'{self._fit_list_prop.value}' entry argument")
                     else:
                         self.Raise("Generator node requires 'fit,fdt-list' property")
 
@@ -367,10 +366,10 @@ class Entry_fit(Entry_section):
             first.
 
             Args:
-                subnode (None): Generator node to process
-                depth: Current node depth (0 is the base 'fit' node)
-                in_images: True if this is inside the 'images' node, so that
-                    'data' properties should be generated
+                subnode (Node): Generator node to process
+                depth (int): Current node depth (0 is the base 'fit' node)
+                in_images (bool): True if this is inside the 'images' node, so
+                    that 'data' properties should be generated
             """
             oper = self._get_operation(subnode)
             if oper == OP_GEN_FDT_NODES:
@@ -380,9 +379,10 @@ class Entry_fit(Entry_section):
             """Add nodes to the output FIT
 
             Args:
-                base_node: Base Node of the FIT (with 'description' property)
-                depth: Current node depth (0 is the base 'fit' node)
-                node: Current node to process
+                base_node (Node): Base Node of the FIT (with 'description'
+                    property)
+                depth (int): Current node depth (0 is the base 'fit' node)
+                node (Node): Current node to process
 
             There are two cases to deal with:
                 - hash and signature nodes which become part of the FIT
@@ -439,7 +439,7 @@ class Entry_fit(Entry_section):
         according to where they ended up in the packed FIT file.
 
         Args:
-            image_pos: Position of this entry in the image
+            image_pos (int): Position of this entry in the image
         """
         super().SetImagePos(image_pos)
 
@@ -478,7 +478,7 @@ class Entry_fit(Entry_section):
 
             # This should never happen
             else: # pragma: no cover
-                self.Raise("%s: missing data properties" % (path))
+                self.Raise(f'{path}: missing data properties')
 
             section.SetOffsetSize(offset, size)
             section.SetImagePos(self.image_pos)
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#44 lore raw

PATCH v2 17/25 binman: Add a consistent way to report errors with fit

In reply to Simon Glass

Add a new function to handling reporting errors within a particular subnode of the FIT description. This can be used to make the format of these errors consistent.

---

(no changes since v1)

 tools/binman/etype/fit.py | 33 +++++++++++++++++++++++++--------
 tools/binman/ftest.py     |  2 +-
 2 files changed, 26 insertions(+), 9 deletions(-)
tools/binman/etype/fit.py +25 -8
index 70966028e8..50a9179f9f 100644
@@ -186,11 +186,11 @@ class Entry_fit(Entry_section):
     def ReadNode(self):
         super().ReadNode()
 
-    def _get_operation(self, subnode):
+    def _get_operation(self, base_node, node):
         """Get the operation referenced by a subnode
 
         Args:
-            subnode (Node): Subnode (of the FIT) to check
+            node (Node): Subnode (of the FIT) to check
 
         Returns:
             int: Operation to perform
@@ -198,12 +198,13 @@ class Entry_fit(Entry_section):
         Raises:
             ValueError: Invalid operation name
         """
-        oper_name = subnode.props.get('fit,operation')
+        oper_name = node.props.get('fit,operation')
         if not oper_name:
             return OP_GEN_FDT_NODES
         oper = OPERATIONS.get(oper_name.value)
-        if not oper:
-            self.Raise(f"Unknown operation '{oper_name.value}'")
+        if oper is None:
+            self._raise(base_node, node,
+                        f"Unknown operation '{oper_name.value}'")
         return oper
 
     def ReadEntries(self):
@@ -273,6 +274,20 @@ class Entry_fit(Entry_section):
 
         return tools.read_file(output_fname)
 
+    def _raise(self, base_node, node, msg):
+        """Raise an error with a paticular FIT subnode
+
+        Args:
+            base_node (Node): Base Node of the FIT (with 'description' property)
+            node (Node): FIT subnode containing the error
+            msg (str): Message to report
+
+        Raises:
+            ValueError, as requested
+        """
+        rel_path = node.path[len(base_node.path) + 1:]
+        self.Raise(f"subnode '{rel_path}': {msg}")
+
     def _build_input(self):
         """Finish the FIT by adding the 'data' properties to it
 
@@ -356,7 +371,7 @@ class Entry_fit(Entry_section):
                     else:
                         self.Raise("Generator node requires 'fit,fdt-list' property")
 
-        def _gen_node(subnode, depth, in_images):
+        def _gen_node(base_node, subnode, depth, in_images):
             """Generate nodes from a template
 
             This creates one node for each member of self._fdts using the
@@ -366,12 +381,14 @@ class Entry_fit(Entry_section):
             first.
 
             Args:
+                base_node (Node): Base Node of the FIT (with 'description'
+                    property)
                 subnode (Node): Generator node to process
                 depth (int): Current node depth (0 is the base 'fit' node)
                 in_images (bool): True if this is inside the 'images' node, so
                     that 'data' properties should be generated
             """
-            oper = self._get_operation(subnode)
+            oper = self._get_operation(base_node, subnode)
             if oper == OP_GEN_FDT_NODES:
                 _gen_fdt_nodes(subnode, depth, in_images)
 
@@ -412,7 +429,7 @@ class Entry_fit(Entry_section):
                 elif self.GetImage().generate and subnode.name.startswith('@'):
                     subnode_path = f'{rel_path}/{subnode.name}'
                     entry = self._entries.get(subnode_path)
-                    _gen_node(subnode, depth, in_images)
+                    _gen_node(base_node, subnode, depth, in_images)
                     if entry:
                         del self._entries[subnode_path]
                 else:
tools/binman/ftest.py +1 -1
index 106027d245..16463518aa 100644
@@ -5319,7 +5319,7 @@ fdt         fdtmap                Extract the devicetree blob from the fdtmap
         """Check handling of an FDT map when the section cannot be found"""
         with self.assertRaises(ValueError) as exc:
             self._DoReadFileDtb('224_fit_bad_oper.dts')
-        self.assertIn("Node '/binman/fit': Unknown operation 'unknown'",
+        self.assertIn("Node '/binman/fit': subnode 'images/@fdt-SEQ': Unknown operation 'unknown'",
                       str(exc.exception))
 
     def test_uses_expand_size(self):
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 17/25
#45 lore raw

Re: [PATCH v2 17/25] binman: Add a consistent way to report errors with fit

Quoted 4 lines · Simon Glass

Add a new function to handling reporting errors within a particular subnode of the FIT description. This can be used to make the format of these errors consistent.

Quoted 8 lines

---

(no changes since v1)

 tools/binman/etype/fit.py | 33 +++++++++++++++++++++++++--------
 tools/binman/ftest.py     |  2 +-
 2 files changed, 26 insertions(+), 9 deletions(-)

I don't like entry.Raise(msg) in general, if it were me I would define Exception subclasses and e.g. raise CustomError(self, subnode) for the various error conditions, but that's beside the point.

Quoted 11 lines
tools/binman/etype/fit.py +2 -0
index 70966028e8..50a9179f9f 100644
[...]
@@ -273,6 +274,20 @@ class Entry_fit(Entry_section):
 
         return tools.read_file(output_fname)
 
+    def _raise(self, base_node, node, msg):
+        """Raise an error with a paticular FIT subnode

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 288

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+
+        Args:
+            base_node (Node): Base Node of the FIT (with 'description' property)
+            node (Node): FIT subnode containing the error
+            msg (str): Message to report
+
+        Raises:
+            ValueError, as requested
+        """
+        rel_path = node.path[len(base_node.path) + 1:]

Can base_node be anything except self._node here, why not use that?

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 289

Quoted 1 line
tools/binman/etype/fit.py +1 -0
+        self.Raise(f"subnode '{rel_path}': {msg}")

The names are a bit inconsistent, but I guess you intend to change Entry.Raise() to Entry._raise() later on...

Quoted 5 lines
tools/binman/etype/fit.py +1 -0
+
     def _build_input(self):
         """Finish the FIT by adding the 'data' properties to it
 
[...]
#46 lore raw

Re: [PATCH v2 17/25] binman: Add a consistent way to report errors with fit

Hi Alper,

Quoted 15 lines · Alper Nebi Yasak
Quoted 4 lines · Simon Glass

Add a new function to handling reporting errors within a particular subnode of the FIT description. This can be used to make the format of these errors consistent.

Quoted 8 lines

---

(no changes since v1)

 tools/binman/etype/fit.py | 33 +++++++++++++++++++++++++--------
 tools/binman/ftest.py     |  2 +-
 2 files changed, 26 insertions(+), 9 deletions(-)
Quoted 5 lines

I don't like entry.Raise(msg) in general, if it were me I would define Exception subclasses and e.g. raise CustomError(self, subnode) for the various error conditions, but that's beside the point.

I have so far stuck with ValueError as I find the need to import classes to raise an exception a pain. But I suppose that would be OK.

Quoted 10 lines
Quoted 10 lines
tools/binman/etype/fit.py +0 -0
index 70966028e8..50a9179f9f 100644
[...]
@@ -273,6 +274,20 @@ class Entry_fit(Entry_section):
 
         return tools.read_file(output_fname)
 
Quoted 14 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 288

Quoted 12 lines
tools/binman/etype/fit.py +12 -0
+    def _raise(self, base_node, node, msg):
+        """Raise an error with a paticular FIT subnode
+
+        Args:
+            base_node (Node): Base Node of the FIT (with 'description' property)
+            node (Node): FIT subnode containing the error
+            msg (str): Message to report
+
+        Raises:
+            ValueError, as requested
+        """
+        rel_path = node.path[len(base_node.path) + 1:]

Can base_node be anything except self._node here, why not use that?

OK I fixed that.

Quoted 5 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 289

Quoted 2 lines
tools/binman/etype/fit.py +1 -0
+        self.Raise(f"subnode '{rel_path}': {msg}")

The names are a bit inconsistent, but I guess you intend to change Entry.Raise() to Entry._raise() later on...

Well I was planning to change Raise() to raise(), so I'll rename this raise_subnode()

Quoted 6 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 293

Quoted 6 lines
tools/binman/etype/fit.py +1 -0
+
     def _build_input(self):
         """Finish the FIT by adding the 'data' properties to it
 
[...]

Regards,
Simon

#47 lore raw

PATCH v2 18/25 binman: Update fit to use node instead of subnode

In reply to Simon Glass

It doesn't make sense to use 'subnode' as a function parameter since it is just a 'node' so far as the function is concerned. Update two functions to use 'node' instead.

---

(no changes since v1)

 tools/binman/etype/fit.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
tools/binman/etype/fit.py +8 -8
index 50a9179f9f..e1b056f56e 100644
@@ -332,7 +332,7 @@ class Entry_fit(Entry_section):
                 return
             fsw.property(pname, prop.bytes)
 
-        def _gen_fdt_nodes(subnode, depth, in_images):
+        def _gen_fdt_nodes(node, depth, in_images):
             """Generate FDT nodes
 
             This creates one node for each member of self._fdts using the
@@ -342,7 +342,7 @@ class Entry_fit(Entry_section):
             first.
 
             Args:
-                subnode (None): Generator node to process
+                node (None): Generator node to process
                 depth: Current node depth (0 is the base 'fit' node)
                 in_images: True if this is inside the 'images' node, so that
                     'data' properties should be generated
@@ -350,10 +350,10 @@ class Entry_fit(Entry_section):
             if self._fdts:
                 # Generate nodes for each FDT
                 for seq, fdt_fname in enumerate(self._fdts):
-                    node_name = subnode.name[1:].replace('SEQ', str(seq + 1))
+                    node_name = node.name[1:].replace('SEQ', str(seq + 1))
                     fname = tools.get_input_filename(fdt_fname + '.dtb')
                     with fsw.add_node(node_name):
-                        for pname, prop in subnode.props.items():
+                        for pname, prop in node.props.items():
                             val = prop.bytes.replace(
                                 b'NAME', tools.to_bytes(fdt_fname))
                             val = val.replace(
@@ -371,7 +371,7 @@ class Entry_fit(Entry_section):
                     else:
                         self.Raise("Generator node requires 'fit,fdt-list' property")
 
-        def _gen_node(base_node, subnode, depth, in_images):
+        def _gen_node(base_node, node, depth, in_images):
             """Generate nodes from a template
 
             This creates one node for each member of self._fdts using the
@@ -383,14 +383,14 @@ class Entry_fit(Entry_section):
             Args:
                 base_node (Node): Base Node of the FIT (with 'description'
                     property)
-                subnode (Node): Generator node to process
+                node (Node): Generator node to process
                 depth (int): Current node depth (0 is the base 'fit' node)
                 in_images (bool): True if this is inside the 'images' node, so
                     that 'data' properties should be generated
             """
-            oper = self._get_operation(base_node, subnode)
+            oper = self._get_operation(base_node, node)
             if oper == OP_GEN_FDT_NODES:
-                _gen_fdt_nodes(subnode, depth, in_images)
+                _gen_fdt_nodes(node, depth, in_images)
 
         def _add_node(base_node, depth, node):
             """Add nodes to the output FIT
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 18/25
#48 lore raw

Re: [PATCH v2 18/25] binman: Update fit to use node instead of subnode

Quoted 11 lines · Simon Glass

It doesn't make sense to use 'subnode' as a function parameter since it is just a 'node' so far as the function is concerned. Update two functions to use 'node' instead.

---

(no changes since v1)

 tools/binman/etype/fit.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
#49 lore raw

PATCH v2 19/25 binman: Keep a separate list of entries for fit

In reply to Simon Glass

The current implementation sets up the FIT entries but then deletes the 'generator' ones so they don't appear in the final image.

This is a bit clumsy. We cannot build the image more than once, since the generator entries are lost during the first build. Binman requires that calling BuildSectionData() multiple times returns a valid result each time.

Keep a separate, private list which includes the generator nodes and use that where needed, to correct this problem. Ensure that the missing list includes removed generator entries too.

---

(no changes since v1)

 tools/binman/etype/fit.py | 33 ++++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)
tools/binman/etype/fit.py +28 -5
index e1b056f56e..30b20a07a2 100644
@@ -163,10 +163,15 @@ class Entry_fit(Entry_section):
                 key: relative path to entry Node (from the base of the FIT)
                 value: Entry_section object comprising the contents of this
                     node
+            _priv_entries: Internal copy of _entries which includes 'generator'
+                entries which are used to create the FIT, but should not be
+                processed as real entries. This is set up once we have the
+                entries
         """
         super().__init__(section, etype, node)
         self._fit = None
         self._fit_props = {}
+        self._priv_entries = {}
 
         for pname, prop in self._node.props.items():
             if pname.startswith('fit,'):
@@ -239,6 +244,10 @@ class Entry_fit(Entry_section):
 
         _add_entries(self._node, 0, self._node)
 
+        # Keep a copy of all entries, including generator entries, since these
+        # removed from self._entries later.
+        self._priv_entries = dict(self._entries)
+
     def BuildSectionData(self, required):
         """Build FIT entry contents
 
@@ -415,11 +424,12 @@ class Entry_fit(Entry_section):
 
             has_images = depth == 2 and in_images
             if has_images:
-                entry = self._entries[rel_path]
+                entry = self._priv_entries[rel_path]
                 data = entry.GetData()
                 fsw.property('data', bytes(data))
 
             for subnode in node.subnodes:
+                subnode_path = f'{rel_path}/{subnode.name}'
                 if has_images and not (subnode.name.startswith('hash') or
                                        subnode.name.startswith('signature')):
                     # This subnode is a content node not meant to appear in
@@ -427,11 +437,11 @@ class Entry_fit(Entry_section):
                     # fsw.add_node() or _add_node() for it.
                     pass
                 elif self.GetImage().generate and subnode.name.startswith('@'):
-                    subnode_path = f'{rel_path}/{subnode.name}'
-                    entry = self._entries.get(subnode_path)
                     _gen_node(base_node, subnode, depth, in_images)
-                    if entry:
-                        del self._entries[subnode_path]
+                    # This is a generator (template) entry, so remove it from
+                    # the list of entries used by PackEntries(), etc. Otherwise
+                    # it will appear in the binman output
+                    to_remove.append(subnode_path)
                 else:
                     with fsw.add_node(subnode.name):
                         _add_node(base_node, depth + 1, subnode)
@@ -440,10 +450,16 @@ class Entry_fit(Entry_section):
         # entry node
         fsw = libfdt.FdtSw()
         fsw.finish_reservemap()
+        to_remove = []
         with fsw.add_node(''):
             _add_node(self._node, 0, self._node)
         fdt = fsw.as_fdt()
 
+        # Remove generator entries from the main list
+        for path in to_remove:
+            if path in self._entries:
+                del self._entries[path]
+
         # Pack this new FDT and scan it so we can add the data later
         fdt.pack()
         data = fdt.as_bytearray()
@@ -503,3 +519,10 @@ class Entry_fit(Entry_section):
     def AddBintools(self, btools):
         super().AddBintools(btools)
         self.mkimage = self.AddBintool(btools, 'mkimage')
+
+    def CheckMissing(self, missing_list):
+        # We must use our private entry list for this since generator notes
+        # which are removed from self._entries will otherwise not show up as
+        # missing
+        for entry in self._priv_entries.values():
+            entry.CheckMissing(missing_list)
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 19/25
#50 lore raw

Re: [PATCH v2 19/25] binman: Keep a separate list of entries for fit

Quoted 2 lines · Simon Glass

The current implementation sets up the FIT entries but then deletes the 'generator' ones so they don't appear in the final image.

They still show up in the fdtmap if I add one to rockchip-u-boot.dtsi:

  $ binman ls -i u-boot-rockchip.bin
  Name                Image-pos  Size     Entry-type    Offset
  -------------------------------------------------------------
  main-section                0   103520  section             0
    mkimage                   0    1a000  mkimage             0
    fit                   1a000    e8d74  fit             1a000
      u-boot              1a644    b1898  section           644
        u-boot-nodtb      1a644    b1898  u-boot-nodtb        0
      @atf-SEQ                0        0  section             0
        atf-bl31              0        0  atf-bl31            0
      @tee-SEQ                0        0  section             0
        tee-os                0        0  tee-os              0
      @fdt-SEQ                0        0  section             0
    fdtmap               102d74      79c  fdtmap         102d74

But not simple-bin.map:

ImagePos Offset Size Name
00000000 00000000 00103520 main-section

  00000000   00000000  0001a000  mkimage
  0001a000   0001a000  000e8d74  fit

0001a644 00000644 000b1898 u-boot
0001a644 00000000 000b1898 u-boot-nodtb
00102d74 00102d74 0000079c fdtmap

This happens in v1 as well, but I hadn't checked it then.

(The "fit" size and "fdtmap" offset are off by 0x10 too, but I'm not sure why yet.)

Quoted 4 lines

This is a bit clumsy. We cannot build the image more than once, since the generator entries are lost during the first build. Binman requires that calling BuildSectionData() multiple times returns a valid result each time.

I think the generator entries should be turned into concrete entries and be removed in _gen_entries(), so BuildSectionData() should work on the entries that were generated. This way the individual entries would show up in fdtmap and could be binman-extracted/replaced as well.

For FDTs we can generate blob entries for each file, for ELFs maybe we can split them into files like the script used to do (bl31_0x<addr>.bin) and do the same?

Maybe some new entry types, "data" for arbitrary data unrelated to a file whose contents we can set, or "elf-segment" that can extract a segment from an ELF file (but I don't like the information loss there).

Quoted 8 lines

Keep a separate, private list which includes the generator nodes and use that where needed, to correct this problem. Ensure that the missing list includes removed generator entries too.

---

(no changes since v1)

I'm more and more convinced that generator nodes needs a thorough redesign, but the patch is consistent with status quo, so:

Quoted 8 lines
 tools/binman/etype/fit.py | 33 ++++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)
tools/binman/etype/fit.py +0 -0
index e1b056f56e..30b20a07a2 100644

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 169

Quoted 7 lines
tools/binman/etype/fit.py +4 -0
@@ -163,10 +163,15 @@ class Entry_fit(Entry_section):
                 key: relative path to entry Node (from the base of the FIT)
                 value: Entry_section object comprising the contents of this
                     node
+            _priv_entries: Internal copy of _entries which includes 'generator'
+                entries which are used to create the FIT, but should not be
+                processed as real entries. This is set up once we have the
+                entries

Maybe this could be "_templates" or "_entry_generators" and only keep track of the generator entries.

Quoted 10 lines
tools/binman/etype/fit.py +1 -0
         """
         super().__init__(section, etype, node)
         self._fit = None
         self._fit_props = {}
+        self._priv_entries = {}
 
         for pname, prop in self._node.props.items():
             if pname.startswith('fit,'):
 
[...]
#51 lore raw

Re: [PATCH v2 19/25] binman: Keep a separate list of entries for fit

Hi Alper,

Quoted 28 lines · Alper Nebi Yasak
Quoted 2 lines · Simon Glass

The current implementation sets up the FIT entries but then deletes the 'generator' ones so they don't appear in the final image.

They still show up in the fdtmap if I add one to rockchip-u-boot.dtsi:

  $ binman ls -i u-boot-rockchip.bin
  Name                Image-pos  Size     Entry-type    Offset
  -------------------------------------------------------------
  main-section                0   103520  section             0
    mkimage                   0    1a000  mkimage             0
    fit                   1a000    e8d74  fit             1a000
      u-boot              1a644    b1898  section           644
        u-boot-nodtb      1a644    b1898  u-boot-nodtb        0
      @atf-SEQ                0        0  section             0
        atf-bl31              0        0  atf-bl31            0
      @tee-SEQ                0        0  section             0
        tee-os                0        0  tee-os              0
      @fdt-SEQ                0        0  section             0
    fdtmap               102d74      79c  fdtmap         102d74

But not simple-bin.map:

ImagePos Offset Size Name
00000000 00000000 00103520 main-section

  00000000   00000000  0001a000  mkimage
  0001a000   0001a000  000e8d74  fit
Quoted 8 lines

0001a644 00000644 000b1898 u-boot
0001a644 00000000 000b1898 u-boot-nodtb
00102d74 00102d74 0000079c fdtmap

This happens in v1 as well, but I hadn't checked it then.

(The "fit" size and "fdtmap" offset are off by 0x10 too, but I'm not sure why yet.)

Well let's fix this after the series. We need tests for the map and so on.

Quoted 10 lines
Quoted 5 lines

This is a bit clumsy. We cannot build the image more than once, since the generator entries are lost during the first build. Binman requires that calling BuildSectionData() multiple times returns a valid result each time.

I think the generator entries should be turned into concrete entries and be removed in _gen_entries(), so BuildSectionData() should work on the entries that were generated. This way the individual entries would show up in fdtmap and could be binman-extracted/replaced as well.

That makes sense, but I'm not sure how to implement it. The split-elf thing needs the contents of the entries which is not available at the start. It is likely available before BuildSectionData() is called, but not necessarily. So the template needs to hang around at least as long as that.

I think there is something we could do here, but it isn't quite clear to me. Perhaps we need a expand-nodes-based-on-contents phase in control.py ? Eek...

Quoted 4 lines

For FDTs we can generate blob entries for each file, for ELFs maybe we can split them into files like the script used to do (bl31_0x<addr>.bin) and do the same?

I'm not a big fan of adding files. Binman should be able to hold everything in memory. It does generate files for later inspection though, so yes we could add this feature.

Quoted 4 lines

Maybe some new entry types, "data" for arbitrary data unrelated to a file whose contents we can set, or "elf-segment" that can extract a segment from an ELF file (but I don't like the information loss there).

Yes I quite like the idea of new entry types. In fact I was hoping to turn split-elf into an entry type (one that generated multiple entries). But I decided to stop before doing that since we really need to gets the code in there, fix the bugs /move forward with some of the ideas you and others have.

Quoted 18 lines
Quoted 9 lines

Keep a separate, private list which includes the generator nodes and use that where needed, to correct this problem. Ensure that the missing list includes removed generator entries too.

---

(no changes since v1)

I'm more and more convinced that generator nodes needs a thorough redesign, but the patch is consistent with status quo, so:

Quoted 3 lines
 tools/binman/etype/fit.py | 33 ++++++++++++++++++++++++++++-----
 1 file changed, 28 insertions(+), 5 deletions(-)
Quoted 15 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 169

Quoted 12 lines
tools/binman/etype/fit.py +4 -0
index e1b056f56e..30b20a07a2 100644
@@ -163,10 +163,15 @@ class Entry_fit(Entry_section):
                 key: relative path to entry Node (from the base of the FIT)
                 value: Entry_section object comprising the contents of this
                     node
+            _priv_entries: Internal copy of _entries which includes 'generator'
+                entries which are used to create the FIT, but should not be
+                processed as real entries. This is set up once we have the
+                entries

Maybe this could be "_templates" or "_entry_generators" and only keep track of the generator entries.

Again I think we can revisit that. I feel it is a bit messy right now, too.

Quoted 11 lines

tools/binman/etype/fit.py class Entry_fit(Entry_section):

Quoted 11 lines
tools/binman/etype/fit.py +1 -0
         """
         super().__init__(section, etype, node)
         self._fit = None
         self._fit_props = {}
+        self._priv_entries = {}
 
         for pname, prop in self._node.props.items():
             if pname.startswith('fit,'):
 
[...]

Regards,
Simon

#52 lore raw

Re: [PATCH v2 19/25] binman: Keep a separate list of entries for fit

Quoted 17 lines · Simon Glass
Quoted 9 lines · Alper Nebi Yasak
Quoted 4 lines

This is a bit clumsy. We cannot build the image more than once, since the generator entries are lost during the first build. Binman requires that calling BuildSectionData() multiple times returns a valid result each time.

I think the generator entries should be turned into concrete entries and be removed in _gen_entries(), so BuildSectionData() should work on the entries that were generated. This way the individual entries would show up in fdtmap and could be binman-extracted/replaced as well.

That makes sense, but I'm not sure how to implement it. The split-elf thing needs the contents of the entries which is not available at the start. It is likely available before BuildSectionData() is called, but not necessarily. So the template needs to hang around at least as long as that.

Quoted 3 lines

I think there is something we could do here, but it isn't quite clear to me. Perhaps we need a expand-nodes-based-on-contents phase in control.py ? Eek...

I can't think of anything proper. I guess it could need architectural changes, but I need to read more of the code and get a better grasp of things.

Quoted 14 lines
Quoted 4 lines

For FDTs we can generate blob entries for each file, for ELFs maybe we can split them into files like the script used to do (bl31_0x<addr>.bin) and do the same?

I'm not a big fan of adding files. Binman should be able to hold everything in memory. It does generate files for later inspection though, so yes we could add this feature.

Quoted 4 lines

Maybe some new entry types, "data" for arbitrary data unrelated to a file whose contents we can set, or "elf-segment" that can extract a segment from an ELF file (but I don't like the information loss there).

Quoted 5 lines

Yes I quite like the idea of new entry types. In fact I was hoping to turn split-elf into an entry type (one that generated multiple entries). But I decided to stop before doing that since we really need to gets the code in there, fix the bugs /move forward with some of the ideas you and others have.

After this and the other fixes and such, I hope I can set aside some time and experiment more on the ideas I've been throwing at you.

Sorry if I ended up delaying this too much already...

#53 lore raw

PATCH v2 20/25 binman: Support splitting an ELF file into multiple nodes

In reply to Simon Glass

Some boards need to load an ELF file using the 'loadables' property, but the file has segments at different memory addresses. This means that it cannot be supplied as a flat binary.

Allow generating a separate node in the FIT for each segment in the ELF, with a different load address for each.

Also add checks that the fit,xxx directives are valid.

---

Changes in v2:
- Rewrite this to use the new FIT entry-type implementation
- Rename op-tee to tee-os

 tools/binman/entries.rst                     | 146 ++++++++++++
 tools/binman/etype/fit.py                    | 230 ++++++++++++++++++-
 tools/binman/ftest.py                        | 147 ++++++++++++
 tools/binman/test/226_fit_split_elf.dts      |  67 ++++++
 tools/binman/test/227_fit_bad_dir.dts        |   9 +
 tools/binman/test/228_fit_bad_dir_config.dts |   9 +
 6 files changed, 598 insertions(+), 10 deletions(-)
 create mode 100644 tools/binman/test/226_fit_split_elf.dts
 create mode 100644 tools/binman/test/227_fit_bad_dir.dts
 create mode 100644 tools/binman/test/228_fit_bad_dir_config.dts
tools/binman/entries.rst +146 -0
index 484cde5c80..a915f64d46 100644
@@ -612,6 +612,9 @@ gen-fdt-nodes
     Generate FDT nodes as above. This is the default if there is no
     `fit,operation` property.
 
+split-elf
+    Split an ELF file into a separate node for each segment.
+
 Generating nodes from an FDT list (gen-fdt-nodes)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -655,6 +658,149 @@ for each of your two files.
 Note that if no devicetree files are provided (with '-a of-list' as above)
 then no nodes will be generated.
 
+Generating nodes from an ELF file (split-elf)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This uses the node as a template to generate multiple nodes. The following
+special properties are available:
+
+split-elf
+    Split an ELF file into a separate node for each segment. This uses the
+    node as a template to generate multiple nodes. The following special
+    properties are available:
+
+    fit,load
+        Generates a `load = <...>` property with the load address of the
+        segmnet
+
+    fit,entry
+        Generates a `entry = <...>` property with the entry address of the
+        ELF. This is only produced for the first entry
+
+    fit,data
+        Generates a `data = <...>` property with the contents of the segment
+
+    fit,loadables
+        Generates a `loadable = <...>` property with a list of the generated
+        nodes (including all nodes if this operation is used multiple times)
+
+
+Here is an example showing ATF, TEE and a device tree all combined::
+
+    fit {
+        description = "test-desc";
+        #address-cells = <1>;
+        fit,fdt-list = "of-list";
+
+        images {
+            u-boot {
+                description = "U-Boot (64-bit)";
+                type = "standalone";
+                os = "U-Boot";
+                arch = "arm64";
+                compression = "none";
+                load = <CONFIG_SYS_TEXT_BASE>;
+                u-boot-nodtb {
+                };
+            };
+            @fdt-SEQ {
+                description = "fdt-NAME.dtb";
+                type = "flat_dt";
+                compression = "none";
+            };
+            @atf-SEQ {
+                fit,operation = "split-elf";
+                description = "ARM Trusted Firmware";
+                type = "firmware";
+                arch = "arm64";
+                os = "arm-trusted-firmware";
+                compression = "none";
+                fit,load;
+                fit,entry;
+                fit,data;
+
+                atf-bl31 {
+                };
+            };
+
+            @tee-SEQ {
+                fit,operation = "split-elf";
+                description = "TEE";
+                type = "tee";
+                arch = "arm64";
+                os = "tee";
+                compression = "none";
+                fit,load;
+                fit,entry;
+                fit,data;
+
+                tee-os {
+                };
+            };
+        };
+
+        configurations {
+            default = "@config-DEFAULT-SEQ";
+            @config-SEQ {
+                description = "conf-NAME.dtb";
+                fdt = "fdt-SEQ";
+                firmware = "u-boot";
+                fit,loadables;
+            };
+        };
+    };
+
+If ATF-BL31 is available, this generates a node for each segment in the
+ELF file, for example::
+
+    images {
+        atf-1 {
+            data = <...contents of first segment...>;
+            data-offset = <0x00000000>;
+            entry = <0x00040000>;
+            load = <0x00040000>;
+            compression = "none";
+            os = "arm-trusted-firmware";
+            arch = "arm64";
+            type = "firmware";
+            description = "ARM Trusted Firmware";
+        };
+        atf-2 {
+            data = <...contents of second segment...>;
+            load = <0xff3b0000>;
+            compression = "none";
+            os = "arm-trusted-firmware";
+            arch = "arm64";
+            type = "firmware";
+            description = "ARM Trusted Firmware";
+        };
+    };
+
+The same applies for OP-TEE if that is available.
+
+If each binary is not available, the relevant template node (@atf-SEQ or
+@tee-SEQ) is removed from the output.
+
+This also generates a `config-xxx` node for each device tree in `of-list`.
+Note that the U-Boot build system uses `-a of-list=$(CONFIG_OF_LIST)`
+so you can use `CONFIG_OF_LIST` to define that list. In this example it is
+set up for `firefly-rk3399` with a single device tree and the default set
+with `-a default-dt=$(CONFIG_DEFAULT_DEVICE_TREE)`, so the resulting output
+is::
+
+    configurations {
+        default = "config-1";
+        config-1 {
+            loadables = "atf-1", "atf-2", "atf-3", "tee-1", "tee-2";
+            description = "rk3399-firefly.dtb";
+            fdt = "fdt-1";
+            firmware = "u-boot";
+        };
+    };
+
+U-Boot SPL can then load the firmware (U-Boot proper) and all the loadables
+(ATF and TEE), then proceed with the boot.
+
 
 
 Entry: fmap: An entry which contains an Fmap section
tools/binman/etype/fit.py +220 -10

305 lines not on this page · raw message

tools/binman/ftest.py +147 -0

159 lines not on this page · raw message

tools/binman/test/226_fit_split_elf.dts new file +67 -0

67 lines not on this page · raw message

tools/binman/test/227_fit_bad_dir.dts new file +9 -0
new file mode 100644
index 0000000000..51f4816c4c
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+
+/dts-v1/;
+
+#include "226_fit_split_elf.dts"
+
+&atf {
+	fit,something = "bad";
+};
tools/binman/test/228_fit_bad_dir_config.dts new file +9 -0
new file mode 100644
index 0000000000..825a346c3e
@@ -0,0 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0+
+
+/dts-v1/;
+
+#include "226_fit_split_elf.dts"
+
+&config {
+	fit,config = "bad";
+};
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#54 lore raw

Re: [PATCH v2 20/25] binman: Support splitting an ELF file into multiple nodes

Quoted 41 lines · Simon Glass

Some boards need to load an ELF file using the 'loadables' property, but the file has segments at different memory addresses. This means that it cannot be supplied as a flat binary.

Allow generating a separate node in the FIT for each segment in the ELF, with a different load address for each.

Also add checks that the fit,xxx directives are valid.

---

Changes in v2:
- Rewrite this to use the new FIT entry-type implementation
- Rename op-tee to tee-os

 tools/binman/entries.rst                     | 146 ++++++++++++
 tools/binman/etype/fit.py                    | 230 ++++++++++++++++++-
 tools/binman/ftest.py                        | 147 ++++++++++++
 tools/binman/test/226_fit_split_elf.dts      |  67 ++++++
 tools/binman/test/227_fit_bad_dir.dts        |   9 +
 tools/binman/test/228_fit_bad_dir_config.dts |   9 +
 6 files changed, 598 insertions(+), 10 deletions(-)
 create mode 100644 tools/binman/test/226_fit_split_elf.dts
 create mode 100644 tools/binman/test/227_fit_bad_dir.dts
 create mode 100644 tools/binman/test/228_fit_bad_dir_config.dts

[...]

tools/binman/etype/fit.py +5 -0
index 30b20a07a2..63d552ed19 100644> > [...]
@@ -154,6 +159,149 @@ class Entry_fit(Entry_section):
 
     Note that if no devicetree files are provided (with '-a of-list' as above)
     then no nodes will be generated.
+
+    Generating nodes from an ELF file (split-elf)
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    This uses the node as a template to generate multiple nodes. The following

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 176

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+    special properties are available:
+
+    split-elf
+        Split an ELF file into a separate node for each segment. This uses the
+        node as a template to generate multiple nodes. The following special
+        properties are available:
+
+        fit,load
+            Generates a `load = <...>` property with the load address of the
+            segmnet

segmnet -> segment, missed it the first time around.

Quoted 27 lines
tools/binman/etype/fit.py +15 -5
+
+        fit,entry
+            Generates a `entry = <...>` property with the entry address of the
+            ELF. This is only produced for the first entry
+
+        fit,data
+            Generates a `data = <...>` property with the contents of the segment
+
+        fit,loadables
+            Generates a `loadable = <...>` property with a list of the generated
+            nodes (including all nodes if this operation is used multiple times)
 
[...]
@@ -363,11 +514,20 @@ class Entry_fit(Entry_section):
                     fname = tools.get_input_filename(fdt_fname + '.dtb')
                     with fsw.add_node(node_name):
                         for pname, prop in node.props.items():
-                            val = prop.bytes.replace(
-                                b'NAME', tools.to_bytes(fdt_fname))
-                            val = val.replace(
-                                b'SEQ', tools.to_bytes(str(seq + 1)))
-                            fsw.property(pname, val)
+                            if pname == 'fit,loadables':
+                                val = '\0'.join(self._loadables) + '\0'
+                                fsw.property('loadables', val.encode('utf-8'))
+                            elif pname == 'fit,operation':

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 530

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+                                pass
+                            elif pname.startswith('fit,'):
+                                self._raise(base_node, node,
+                                            f"Unknown directive '{pname}'")
+                            else:
+                                val = prop.bytes.replace(
+                                    b'NAME', tools.to_bytes(fdt_fname))
+                                val = val.replace(
+                                    b'SEQ', tools.to_bytes(str(seq + 1)))
+                                fsw.property(pname, val)

I get two distinct "loadables" properties with this config fragment, and fdtget and mkimage -l only considers the first one:

  @config-SEQ {
     description = "NAME.dtb";
     fdt = "fdt-SEQ";
     firmware = "atf-1";
     loadables = "u-boot";
     fit,loadables;
  };
  $ fdtget simple-bin.fit.fit /configurations/config-1 -p
  description
  fdt
  firmware
  loadables
  loadables
  $ fdtget simple-bin.fit.fit /configurations/config-1 loadables
  u-boot
  $ mkimage -l simple-bin.fit.fit
  FIT description: FIT image for U-Boot with bl31 (TF-A)
  Created:         Thu Feb 24 17:36:36 2022
   Image 0 (u-boot)
    Description:  U-Boot (64-bit)
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Standalone Program
    Compression:  uncompressed
    Data Size:    727192 Bytes = 710.15 KiB = 0.69 MiB
    Architecture: AArch64
    Load Address: 0x00200000
    Entry Point:  unavailable
   Image 1 (atf-1)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    139342 Bytes = 136.08 KiB = 0.13 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0x00040000
   Image 2 (atf-2)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    8024 Bytes = 7.84 KiB = 0.01 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0xff3b0000
   Image 3 (atf-3)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    8192 Bytes = 8.00 KiB = 0.01 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0xff8c0000
   Image 4 (fdt-1)
    Description:  fdt-rk3399-gru-bob
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Flat Device Tree
    Compression:  uncompressed
    Data Size:    69392 Bytes = 67.77 KiB = 0.07 MiB
    Architecture: Unknown Architecture
   Default Configuration: 'config-1'
   Configuration 0 (config-1)
    Description:  rk3399-gru-bob.dtb
    Kernel:       unavailable
    Firmware:     atf-1
    FDT:          fdt-1
    Loadables:    u-boot

I didn't test if my chromebook_kevin boots in this double-loadables configuration, though.

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 547

Quoted 13 lines
tools/binman/etype/fit.py +5 -1
                         # Add data for 'images' nodes (but not 'config')
                         if depth == 1 and in_images:
@@ -380,7 +540,43 @@ class Entry_fit(Entry_section):
                     else:
                         self.Raise("Generator node requires 'fit,fdt-list' property")
 
-        def _gen_node(base_node, node, depth, in_images):
+        def _gen_split_elf(base_node, node, elf_data, missing):
+            """Add nodes for the ELF file, one per group of contiguous segments
+
+            Args:
+                node (Node): Template node from the binman definition

node -> base_node

Quoted 14 lines
tools/binman/etype/fit.py +14 -0
+                node (Node): Node to replace (in the FIT being built)
+                data (bytes): ELF-format data to process (may be empty)
+                missing (bool): True if any of the data is missing
+
+            """
+            # If any pieces are missing, skip this. The missing entries will
+            # show an error
+            if not missing:
+                try:
+                    segments, entry = elf.read_loadable_segments(elf_data)
+                except ValueError as exc:
+                    self._raise(base_node, node,
+                                f'Failed to read ELF file: {str(exc)}')
+                for (seq, start, data) in segments:

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 571

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+                    node_name = node.name[1:].replace('SEQ', str(seq + 1))
+                    with fsw.add_node(node_name):
+                        loadables.append(node_name)
+                        for pname, prop in node.props.items():
+                            if not pname.startswith('fit,'):
+                                fsw.property(pname, prop.bytes)
+                            elif pname == 'fit,load':
+                                fsw.property_u32('load', start)
+                            elif pname == 'fit,entry':
+                                if not seq:

I'd prefer seq == 0 as it's a number.

Quoted 12 lines
tools/binman/etype/fit.py +8 -0
+                                    fsw.property_u32('entry', entry)
+                            elif pname == 'fit,data':
+                                fsw.property('data', bytes(data))
+                            elif pname != 'fit,operation':
+                                self._raise(base_node, node,
+                                            f"Unknown directive '{pname}'")
+
+        def _gen_node(base_node, node, depth, in_images, entry):
             """Generate nodes from a template
 
             This creates one node for each member of self._fdts using the
[...]
#55 lore raw

Re: [PATCH v2 20/25] binman: Support splitting an ELF file into multiple nodes

Hi Alper,

Quoted 97 lines · Alper Nebi Yasak
Quoted 41 lines · Simon Glass

Some boards need to load an ELF file using the 'loadables' property, but the file has segments at different memory addresses. This means that it cannot be supplied as a flat binary.

Allow generating a separate node in the FIT for each segment in the ELF, with a different load address for each.

Also add checks that the fit,xxx directives are valid.

---

Changes in v2:
- Rewrite this to use the new FIT entry-type implementation
- Rename op-tee to tee-os

 tools/binman/entries.rst                     | 146 ++++++++++++
 tools/binman/etype/fit.py                    | 230 ++++++++++++++++++-
 tools/binman/ftest.py                        | 147 ++++++++++++
 tools/binman/test/226_fit_split_elf.dts      |  67 ++++++
 tools/binman/test/227_fit_bad_dir.dts        |   9 +
 tools/binman/test/228_fit_bad_dir_config.dts |   9 +
 6 files changed, 598 insertions(+), 10 deletions(-)
 create mode 100644 tools/binman/test/226_fit_split_elf.dts
 create mode 100644 tools/binman/test/227_fit_bad_dir.dts
 create mode 100644 tools/binman/test/228_fit_bad_dir_config.dts

[...]

tools/binman/etype/fit.py +5 -0
index 30b20a07a2..63d552ed19 100644> > [...]
@@ -154,6 +159,149 @@ class Entry_fit(Entry_section):
 
     Note that if no devicetree files are provided (with '-a of-list' as above)
     then no nodes will be generated.
+
+    Generating nodes from an ELF file (split-elf)
+    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+    This uses the node as a template to generate multiple nodes. The following

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 176

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+    special properties are available:
+
+    split-elf
+        Split an ELF file into a separate node for each segment. This uses the
+        node as a template to generate multiple nodes. The following special
+        properties are available:
+
+        fit,load
+            Generates a `load = <...>` property with the load address of the
+            segmnet

segmnet -> segment, missed it the first time around.

Quoted 27 lines
tools/binman/etype/fit.py +15 -5
+
+        fit,entry
+            Generates a `entry = <...>` property with the entry address of the
+            ELF. This is only produced for the first entry
+
+        fit,data
+            Generates a `data = <...>` property with the contents of the segment
+
+        fit,loadables
+            Generates a `loadable = <...>` property with a list of the generated
+            nodes (including all nodes if this operation is used multiple times)
 
[...]
@@ -363,11 +514,20 @@ class Entry_fit(Entry_section):
                     fname = tools.get_input_filename(fdt_fname + '.dtb')
                     with fsw.add_node(node_name):
                         for pname, prop in node.props.items():
-                            val = prop.bytes.replace(
-                                b'NAME', tools.to_bytes(fdt_fname))
-                            val = val.replace(
-                                b'SEQ', tools.to_bytes(str(seq + 1)))
-                            fsw.property(pname, val)
+                            if pname == 'fit,loadables':
+                                val = '\0'.join(self._loadables) + '\0'
+                                fsw.property('loadables', val.encode('utf-8'))
+                            elif pname == 'fit,operation':

tools/binman/etype/fit.py class Entry_fit(Entry_section): · line 530

Quoted 10 lines
tools/binman/etype/fit.py +10 -0
+                                pass
+                            elif pname.startswith('fit,'):
+                                self._raise(base_node, node,
+                                            f"Unknown directive '{pname}'")
+                            else:
+                                val = prop.bytes.replace(
+                                    b'NAME', tools.to_bytes(fdt_fname))
+                                val = val.replace(
+                                    b'SEQ', tools.to_bytes(str(seq + 1)))
+                                fsw.property(pname, val)

I get two distinct "loadables" properties with this config fragment, and fdtget and mkimage -l only considers the first one:

Quoted 7 lines
  @config-SEQ {
     description = "NAME.dtb";
     fdt = "fdt-SEQ";
     firmware = "atf-1";
     loadables = "u-boot";
     fit,loadables;
  };

Yes that needs fixing, but I'll leave that case for now. It needs its own test.

Quoted 65 lines
  $ fdtget simple-bin.fit.fit /configurations/config-1 -p
  description
  fdt
  firmware
  loadables
  loadables
  $ fdtget simple-bin.fit.fit /configurations/config-1 loadables
  u-boot
  $ mkimage -l simple-bin.fit.fit
  FIT description: FIT image for U-Boot with bl31 (TF-A)
  Created:         Thu Feb 24 17:36:36 2022
   Image 0 (u-boot)
    Description:  U-Boot (64-bit)
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Standalone Program
    Compression:  uncompressed
    Data Size:    727192 Bytes = 710.15 KiB = 0.69 MiB
    Architecture: AArch64
    Load Address: 0x00200000
    Entry Point:  unavailable
   Image 1 (atf-1)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    139342 Bytes = 136.08 KiB = 0.13 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0x00040000
   Image 2 (atf-2)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    8024 Bytes = 7.84 KiB = 0.01 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0xff3b0000
   Image 3 (atf-3)
    Description:  ARM Trusted Firmware
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Firmware
    Compression:  uncompressed
    Data Size:    8192 Bytes = 8.00 KiB = 0.01 MiB
    Architecture: AArch64
    OS:           ARM Trusted Firmware
    Load Address: 0xff8c0000
   Image 4 (fdt-1)
    Description:  fdt-rk3399-gru-bob
    Created:      Thu Feb 24 17:36:36 2022
    Type:         Flat Device Tree
    Compression:  uncompressed
    Data Size:    69392 Bytes = 67.77 KiB = 0.07 MiB
    Architecture: Unknown Architecture
   Default Configuration: 'config-1'
   Configuration 0 (config-1)
    Description:  rk3399-gru-bob.dtb
    Kernel:       unavailable
    Firmware:     atf-1
    FDT:          fdt-1
    Loadables:    u-boot
Quoted 2 lines

I didn't test if my chromebook_kevin boots in this double-loadables configuration, though.

I wish we could get the kevin patches applied. What on earth is going on there? I haven't been able to test any of this on kevin.

[..]

Regards,
Simon

#56 lore raw

PATCH v2 21/25 rockchip: evb-rk3288: Drop raw-image support

In reply to Simon Glass

This boards uses SPL_FIT so does not need to support loading a raw image. Drop it to avoid binman trying to insert a symbol which has no value.

---

(no changes since v1)

 configs/evb-rk3288_defconfig | 1 +
 1 file changed, 1 insertion(+)
configs/evb-rk3288_defconfig +1 -0
index 97d4c14f65..6143c4d989 100644
@@ -24,6 +24,7 @@ CONFIG_USE_PREBOOT=y
 CONFIG_DEFAULT_FDT_FILE="rk3288-evb-rk808.dtb"
 CONFIG_SILENT_CONSOLE=y
 CONFIG_DISPLAY_BOARDINFO_LATE=y
+# CONFIG_SPL_RAW_IMAGE_SUPPORT is not set
 CONFIG_SPL_STACK_R=y
 CONFIG_SPL_STACK_R_MALLOC_SIMPLE_LEN=0x10000
 CONFIG_SPL_OPTEE_IMAGE=y
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#57 lore raw

PATCH v2 22/25 rockchip: Include binman script in 64-bit boards

In reply to Simon Glass

Include the rockchip-u-boot.dtsi file with 64-bit boards and enable binman so that these boards can also use it, rather than using special Makefile rules and scripts.

This does not change the Makefile nor remove any scripts, but sets it up so that this is possible.

---

(no changes since v1)

 arch/arm/dts/px30-u-boot.dtsi              | 2 ++
 arch/arm/dts/rk3308-u-boot.dtsi            | 2 ++
 arch/arm/dts/rk3326-odroid-go2-u-boot.dtsi | 2 ++
 arch/arm/dts/rk3328-u-boot.dtsi            | 2 ++
 arch/arm/dts/rk3368-u-boot.dtsi            | 1 +
 arch/arm/dts/rk3399-u-boot.dtsi            | 5 +++--
 arch/arm/dts/rk3568-u-boot.dtsi            | 2 ++
 arch/arm/mach-rockchip/Kconfig             | 6 ++++++
 8 files changed, 20 insertions(+), 2 deletions(-)
arch/arm/dts/px30-u-boot.dtsi +2 -0
index f102b2aef4..462eaf68f8 100644
@@ -3,6 +3,8 @@
  * (C) Copyright 2019 Rockchip Electronics Co., Ltd
  */
 
+#include "rockchip-u-boot.dtsi"
+
 / {
 	aliases {
 		mmc0 = &emmc;
arch/arm/dts/rk3308-u-boot.dtsi +2 -0
index 4bfad31fba..ab5bfc2ce9 100644
@@ -3,6 +3,8 @@
  *(C) Copyright 2019 Rockchip Electronics Co., Ltd
  */
 
+#include "rockchip-u-boot.dtsi"
+
 / {
 	aliases {
 		mmc0 = &emmc;
arch/arm/dts/rk3326-odroid-go2-u-boot.dtsi +2 -0
index 95f2652494..16c33735eb 100644
@@ -3,6 +3,8 @@
  * Copyright (c) 2020 Theobroma Systems Design und Consulting GmbH
  */
 
+#include "rockchip-u-boot.dtsi"
+
 / {
 	chosen {
 		u-boot,spl-boot-order = &sdmmc;
arch/arm/dts/rk3328-u-boot.dtsi +2 -0
index 1633558264..d4a7540a92 100644
@@ -3,6 +3,8 @@
  * (C) Copyright 2019 Rockchip Electronics Co., Ltd
  */
 
+#include "rockchip-u-boot.dtsi"
+
 / {
 	aliases {
 		mmc0 = &emmc;
arch/arm/dts/rk3368-u-boot.dtsi +1 -0
index 2767c2678d..b37da4e851 100644
@@ -3,6 +3,7 @@
  * Copyright (c) 2020 Theobroma Systems Design und Consulting GmbH
  */
 
+#include "rockchip-u-boot.dtsi"
 #include <dt-bindings/memory/rk3368-dmc.h>
 
 / {
arch/arm/dts/rk3399-u-boot.dtsi +3 -2
index 716b9a433a..fd4150e3f0 100644
@@ -60,8 +60,9 @@
 
 };
 
-#ifdef CONFIG_ROCKCHIP_SPI_IMAGE
 &binman {
+	multiple-images;
+#ifdef CONFIG_ROCKCHIP_SPI_IMAGE
 	rom {
 		filename = "u-boot.rom";
 		size = <0x400000>;
@@ -81,8 +82,8 @@
 		fdtmap {
 		};
 	};
-};
 #endif
+};
 
 &cru {
 	u-boot,dm-pre-reloc;
arch/arm/dts/rk3568-u-boot.dtsi +2 -0
index 5a80dda275..fa9b6ae23b 100644
@@ -3,6 +3,8 @@
  * (C) Copyright 2021 Rockchip Electronics Co., Ltd
  */
 
+#include "rockchip-u-boot.dtsi"
+
 / {
 	aliases {
 		mmc0 = &sdhci;
arch/arm/mach-rockchip/Kconfig +6 -0
index 92f35309e4..c8df65980f 100644
@@ -15,6 +15,7 @@ config ROCKCHIP_PX30
 	select DEBUG_UART_BOARD_INIT
 	imply ROCKCHIP_COMMON_BOARD
 	imply SPL_ROCKCHIP_COMMON_BOARD
+	imply BINMAN
 	help
 	  The Rockchip PX30 is a ARM-based SoC with a quad-core Cortex-A35
 	  including NEON and GPU, Mali-400 graphics, several DDR3 options
@@ -146,6 +147,7 @@ config ROCKCHIP_RK3308
 	imply SPL_SERIAL
 	imply TPL_SERIAL
 	imply SPL_SEPARATE_BSS
+	imply BINMAN
 	help
 	  The Rockchip RK3308 is a ARM-based Soc which embedded with quad
 	  Cortex-A35 and highly integrated audio interfaces.
@@ -167,6 +169,7 @@ config ROCKCHIP_RK3328
 	select ENABLE_ARM_SOC_BOOT0_HOOK
 	select DEBUG_UART_BOARD_INIT
 	select SYS_NS16550
+	imply BINMAN
 	help
 	  The Rockchip RK3328 is a ARM-based SoC with a quad-core Cortex-A53.
 	  including NEON and GPU, 1MB L2 cache, Mali-T7 graphics, two
@@ -186,6 +189,7 @@ config ROCKCHIP_RK3368
 	imply SPL_SERIAL
 	imply TPL_SERIAL
 	imply TPL_ROCKCHIP_COMMON_BOARD
+	imply BINMAN
 	help
 	  The Rockchip RK3368 is a ARM-based SoC with a octa-core (organised
 	  into a big and little cluster with 4 cores each) Cortex-A53 including
@@ -244,6 +248,7 @@ config ROCKCHIP_RK3399
 	imply TPL_ROCKCHIP_COMMON_BOARD
 	imply SYS_BOOTCOUNT_SINGLEWORD if BOOTCOUNT_LIMIT
 	imply CMD_BOOTCOUNT if BOOTCOUNT_LIMIT
+	imply BINMAN
 	help
 	  The Rockchip RK3399 is a ARM-based SoC with a dual-core Cortex-A72
 	  and quad-core Cortex-A53.
@@ -264,6 +269,7 @@ config ROCKCHIP_RK3568
 	select SYSCON
 	select BOARD_LATE_INIT
 	imply ROCKCHIP_COMMON_BOARD
+	imply BINMAN
 	help
 	  The Rockchip RK3568 is a ARM-based SoC with quad-core Cortex-A55,
 	  including NEON and GPU, 512K L3 cache, Mali-G52 based graphics,
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
Reviewed-by on v2 22/25
#58 lore raw

Re: [PATCH v2 22/25] rockchip: Include binman script in 64-bit boards

Quoted 12 lines · Simon Glass

Include the rockchip-u-boot.dtsi file with 64-bit boards and enable binman so that these boards can also use it, rather than using special Makefile rules and scripts.

This does not change the Makefile nor remove any scripts, but sets it up so that this is possible.

---

(no changes since v1)

Quoted 9 lines
 arch/arm/dts/px30-u-boot.dtsi              | 2 ++
 arch/arm/dts/rk3308-u-boot.dtsi            | 2 ++
 arch/arm/dts/rk3326-odroid-go2-u-boot.dtsi | 2 ++
 arch/arm/dts/rk3328-u-boot.dtsi            | 2 ++
 arch/arm/dts/rk3368-u-boot.dtsi            | 1 +
 arch/arm/dts/rk3399-u-boot.dtsi            | 5 +++--
 arch/arm/dts/rk3568-u-boot.dtsi            | 2 ++
 arch/arm/mach-rockchip/Kconfig             | 6 ++++++
 8 files changed, 20 insertions(+), 2 deletions(-)
#59 lore raw

PATCH v2 23/25 rockchip: Support building the all output files in binman

In reply to Simon Glass

Add the required binman images to replace the Makefile rules which are currently used. This includes subsuming:

- tpl/u-boot-tpl-rockchip.bin if TPL is enabled
- idbloader.img if either or both of SPL and TPL are enabled
- u-boot.itb if SPL_FIT is enabled
- u-boot-rockchip.bin if SPL is used, either using u-boot.itb when SPL_FIT is enabled or u-boot.img when it isn't

Note that the intermediate files are dropped with binman, since it producing everything in one pass. This means that tpl/u-boot-tpl-rockchip.bin is not created, for example.

Note that for some 32-bit rk3288 boards, rockchip-optee.dtsi is included.

---

Changes in v2:
- Rename op-tee to tee-os
- Drop use of .itb2

 arch/arm/dts/rockchip-u-boot.dtsi | 84 ++++++++++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 2 deletions(-)
arch/arm/dts/rockchip-u-boot.dtsi +82 -2
index eae3ee715d..64e4466489 100644
@@ -17,13 +17,93 @@
 		filename = "u-boot-rockchip.bin";
 		pad-byte = <0xff>;
 
-		blob {
-			filename = "idbloader.img";
+#ifdef CONFIG_TPL
+		mkimage {
+			args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+			u-boot-tpl {
+			};
+		};
+
+		u-boot-spl {
 		};
+#elif defined(CONFIG_SPL) /* SPL only */
+		mkimage {
+			args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+			u-boot-spl {
+			};
+		};
+#endif
+#if defined(CONFIG_SPL_FIT) && defined(CONFIG_ARM64)
+		fit: fit {
+			description = "FIT image for U-Boot with bl31 (TF-A)";
+			#address-cells = <1>;
+			fit,fdt-list = "of-list";
+			fit,external-offset = <CONFIG_FIT_EXTERNAL_OFFSET>;
+			images {
+				u-boot {
+					description = "U-Boot (64-bit)";
+					type = "standalone";
+					os = "U-Boot";
+					arch = "arm64";
+					compression = "none";
+					load = <CONFIG_SYS_TEXT_BASE>;
+					u-boot-nodtb {
+					};
+				};
 
+				@atf-SEQ {
+					fit,operation = "split-elf";
+					description = "ARM Trusted Firmware";
+					type = "firmware";
+					arch = "arm64";
+					os = "arm-trusted-firmware";
+					compression = "none";
+					fit,load;
+					fit,entry;
+					fit,data;
+
+					atf-bl31 {
+					};
+				};
+				@tee-SEQ {
+					fit,operation = "split-elf";
+					description = "TEE";
+					type = "tee";
+					arch = "arm64";
+					os = "tee";
+					compression = "none";
+					fit,load;
+					fit,entry;
+					fit,data;
+
+					tee-os {
+					};
+				};
+
+				@fdt-SEQ {
+					description = "fdt-NAME";
+					compression = "none";
+					type = "flat_dt";
+				};
+			};
+
+			configurations {
+				default = "@config-DEFAULT-SEQ";
+				@config-SEQ {
+					description = "NAME.dtb";
+					fdt = "fdt-SEQ";
+					firmware = "u-boot";
+					fit,loadables;
+				};
+			};
+		};
+#else
 		u-boot-img {
 			offset = <CONFIG_SPL_PAD_TO>;
 		};
+#endif /* CONFIG_ARM64 */
 	};
 };
 #endif
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#60 lore raw

Re: [PATCH v2 23/25] rockchip: Support building the all output files in binman

Quoted 1 line · Simon Glass

Good Evening,

Quoted 9 lines

Add the required binman images to replace the Makefile rules which are currently used. This includes subsuming:

- tpl/u-boot-tpl-rockchip.bin if TPL is enabled
- idbloader.img if either or both of SPL and TPL are enabled
- u-boot.itb if SPL_FIT is enabled
- u-boot-rockchip.bin if SPL is used, either using u-boot.itb when SPL_FIT is enabled or u-boot.img when it isn't

Quoted 3 lines

Note that the intermediate files are dropped with binman, since it producing everything in one pass. This means that tpl/u-boot-tpl-rockchip.bin is not created, for example.

I love this series, and look forward to it working.
I've tested this with rk3399, unfortunately it does not produce a functional u-boot.itb.
I originally thought it was, but then I realized it was placing it at an incorrect location.

Quoted 37 lines

Note that for some 32-bit rk3288 boards, rockchip-optee.dtsi is included.

---

Changes in v2:
- Rename op-tee to tee-os
- Drop use of .itb2

 arch/arm/dts/rockchip-u-boot.dtsi | 84 ++++++++++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 2 deletions(-)
arch/arm/dts/rockchip-u-boot.dtsi +13 -2
index eae3ee715d..64e4466489 100644
@@ -17,13 +17,93 @@
                filename = "u-boot-rockchip.bin";
                pad-byte = <0xff>;
 
-               blob {
-                       filename = "idbloader.img";
+#ifdef CONFIG_TPL
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-tpl {
+                       };
+               };
+
+               u-boot-spl {
                };
+#elif defined(CONFIG_SPL) /* SPL only */
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+

arch/arm/dts/rockchip-u-boot.dtsi line 43

Quoted 10 lines
arch/arm/dts/rockchip-u-boot.dtsi +10 -0
+                       u-boot-spl {
+                       };
+               };
+#endif
+#if defined(CONFIG_SPL_FIT) && defined(CONFIG_ARM64)
+               fit: fit {
+                       description = "FIT image for U-Boot with bl31 (TF-A)";
+                       #address-cells = <1>;
+                       fit,fdt-list = "of-list";
+                       fit,external-offset = <CONFIG_FIT_EXTERNAL_OFFSET>;

You need:
offset = <CONFIG_SPL_PAD_TO>;
here or the image is located at the wrong location.

Quoted 67 lines
arch/arm/dts/rockchip-u-boot.dtsi +59 -1
+                       images {
+                               u-boot {
+                                       description = "U-Boot (64-bit)";
+                                       type = "standalone";
+                                       os = "U-Boot";
+                                       arch = "arm64";
+                                       compression = "none";
+                                       load = <CONFIG_SYS_TEXT_BASE>;
+                                       u-boot-nodtb {
+                                       };
+                               };
 
+                               @atf-SEQ {
+                                       fit,operation = "split-elf";
+                                       description = "ARM Trusted Firmware";
+                                       type = "firmware";
+                                       arch = "arm64";
+                                       os = "arm-trusted-firmware";
+                                       compression = "none";
+                                       fit,load;
+                                       fit,entry;
+                                       fit,data;
+
+                                       atf-bl31 {
+                                       };
+                               };
+                               @tee-SEQ {
+                                       fit,operation = "split-elf";
+                                       description = "TEE";
+                                       type = "tee";
+                                       arch = "arm64";
+                                       os = "tee";
+                                       compression = "none";
+                                       fit,load;
+                                       fit,entry;
+                                       fit,data;
+
+                                       tee-os {
+                                       };
+                               };
+
+                               @fdt-SEQ {
+                                       description = "fdt-NAME";
+                                       compression = "none";
+                                       type = "flat_dt";
+                               };
+                       };
+
+                       configurations {
+                               default = "@config-DEFAULT-SEQ";
+                               @config-SEQ {
+                                       description = "NAME.dtb";
+                                       fdt = "fdt-SEQ";
+                                       firmware = "u-boot";
+                                       fit,loadables;
+                               };
+                       };
+               };
+#else
                u-boot-img {
                        offset = <CONFIG_SPL_PAD_TO>;
                };
+#endif /* CONFIG_ARM64 */
        };
 };
 #endif
--
Quoted 2 lines

2.35.1.574.g5d30c73bfb-goog

The image produced is not functional however, because it swaps the firmware node with the loadable-1 node.
Working image:
Configuration 0 (config_1)

  Description:  rk3399-pinephone-pro.dtb
  Kernel:       unavailable
  Firmware:     atf_1
  FDT:          fdt_1
  Loadables:    uboot
                atf_2
                atf_3

Non working image produced from this:
Configuration 0 (config-1)

  Description:  rk3399-rockpro64.dtb
  Kernel:       unavailable
  Firmware:     u-boot
  FDT:          fdt-1
  Loadables:    atf-1
                atf-2
                atf-3

Also, it still doesn't support passing two separate files to mkimage at the same time, required for proper support of rk33xx_SPI generation and for rk35xx in general.
Please see my standalone patch for what I hacked together to get that working:
http://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
Note: those images are not functional either due to this issue.

A wishlist item would be for it to produce the original idbloader.img and u-boot.itb files, at least for now.

Thanks!
Very Respectfully,
Peter Geis

#61 lore raw

Re: [PATCH v2 23/25] rockchip: Support building the all output files in binman

Quoted 31 lines · Peter Geis
Quoted 30 lines · Simon Glass
arch/arm/dts/rockchip-u-boot.dtsi +19 -2
index eae3ee715d..64e4466489 100644
@@ -17,13 +17,93 @@
                filename = "u-boot-rockchip.bin";
                pad-byte = <0xff>;
 
-               blob {
-                       filename = "idbloader.img";
+#ifdef CONFIG_TPL
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-tpl {
+                       };
+               };
+
+               u-boot-spl {
                };
+#elif defined(CONFIG_SPL) /* SPL only */
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-spl {
+                       };
+               };
+#endif
+#if defined(CONFIG_SPL_FIT) && defined(CONFIG_ARM64)
+               fit: fit {
Quoted 8 lines

arch/arm/dts/rockchip-u-boot.dtsi line 43

Quoted 4 lines
arch/arm/dts/rockchip-u-boot.dtsi +4 -0
+                       description = "FIT image for U-Boot with bl31 (TF-A)";
+                       #address-cells = <1>;
+                       fit,fdt-list = "of-list";
+                       fit,external-offset = <CONFIG_FIT_EXTERNAL_OFFSET>;

You need:
offset = <CONFIG_SPL_PAD_TO>;
here or the image is located at the wrong location.

Thanks for confirming this.

Quoted 14 lines

The image produced is not functional however, because it swaps the firmware node with the loadable-1 node.
Working image:
Configuration 0 (config_1)

  Description:  rk3399-pinephone-pro.dtb
  Kernel:       unavailable
  Firmware:     atf_1
  FDT:          fdt_1
  Loadables:    uboot
                atf_2
                atf_3

Non working image produced from this:
Configuration 0 (config-1)

Quoted 7 lines
  Description:  rk3399-rockpro64.dtb
  Kernel:       unavailable
  Firmware:     u-boot
  FDT:          fdt-1
  Loadables:    atf-1
                atf-2
                atf-3

And also this. I encountered the same things on rk3399-gru-kevin, mentioned them in my reply [1] to v1 but I guess Simon missed it.

[1] My reply to v1 of this patch
https://lore.kernel.org/u-boot/[email protected]/

Quoted 3 lines

Also, it still doesn't support passing two separate files to mkimage at the same time, required for proper support of rk33xx_SPI generation and for rk35xx in general.

I got v1 of this booting from SPI, but on a board that doesn't use TPL. I see in doc/boards/rockchip.rst that one would indeed pass TPL+SPL to mkimage as two different files. I don't know how they're processed or anything about the rksd/rkspi formats though.

Quoted 10 lines

Please see my standalone patch for what I hacked together to get that working:
http://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
Note: those images are not functional either due to this issue.

A wishlist item would be for it to produce the original idbloader.img and u-boot.itb files, at least for now.

Thanks!
Very Respectfully,
Peter Geis

#62 lore raw

Re: [PATCH v2 23/25] rockchip: Support building the all output files in binman

Quoted 66 lines · Alper Nebi Yasak
Quoted 31 lines · Peter Geis
Quoted 30 lines · Simon Glass
arch/arm/dts/rockchip-u-boot.dtsi +19 -2
index eae3ee715d..64e4466489 100644
@@ -17,13 +17,93 @@
                filename = "u-boot-rockchip.bin";
                pad-byte = <0xff>;
 
-               blob {
-                       filename = "idbloader.img";
+#ifdef CONFIG_TPL
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-tpl {
+                       };
+               };
+
+               u-boot-spl {
                };
+#elif defined(CONFIG_SPL) /* SPL only */
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-spl {
+                       };
+               };
+#endif
+#if defined(CONFIG_SPL_FIT) && defined(CONFIG_ARM64)
+               fit: fit {
Quoted 8 lines

arch/arm/dts/rockchip-u-boot.dtsi line 43

Quoted 4 lines
arch/arm/dts/rockchip-u-boot.dtsi +4 -0
+                       description = "FIT image for U-Boot with bl31 (TF-A)";
+                       #address-cells = <1>;
+                       fit,fdt-list = "of-list";
+                       fit,external-offset = <CONFIG_FIT_EXTERNAL_OFFSET>;

You need:
offset = <CONFIG_SPL_PAD_TO>;
here or the image is located at the wrong location.

Thanks for confirming this.

Quoted 14 lines

The image produced is not functional however, because it swaps the firmware node with the loadable-1 node.
Working image:
Configuration 0 (config_1)

  Description:  rk3399-pinephone-pro.dtb
  Kernel:       unavailable
  Firmware:     atf_1
  FDT:          fdt_1
  Loadables:    uboot
                atf_2
                atf_3

Non working image produced from this:
Configuration 0 (config-1)

Quoted 7 lines
  Description:  rk3399-rockpro64.dtb
  Kernel:       unavailable
  Firmware:     u-boot
  FDT:          fdt-1
  Loadables:    atf-1
                atf-2
                atf-3
Quoted 5 lines

And also this. I encountered the same things on rk3399-gru-kevin, mentioned them in my reply [1] to v1 but I guess Simon missed it.

[1] My reply to v1 of this patch
https://lore.kernel.org/u-boot/[email protected]/

Yes, that's the solution I ended up with as well.
Funnily enough, it seems loadables has a limit of five items, meaning the newest rk356x ATF images which have four binaries, u-boot, and optee trigger an error.

Quoted 9 lines
Quoted 4 lines

Also, it still doesn't support passing two separate files to mkimage at the same time, required for proper support of rk33xx_SPI generation and for rk35xx in general.

I got v1 of this booting from SPI, but on a board that doesn't use TPL. I see in doc/boards/rockchip.rst that one would indeed pass TPL+SPL to mkimage as two different files. I don't know how they're processed or anything about the rksd/rkspi formats though.

RKSPI mode splits the image up into 2048 chunks, and pads each chunk with 2048 of blank space.
For some reason the rk32/rk33 bootrom SPI code reads only 2048 out of each 4096 chunk.
Delightfully this seems to have been fixed in rk35xx.

Quoted 11 lines
Quoted 11 lines

Please see my standalone patch for what I hacked together to get that working:
http://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
Note: those images are not functional either due to this issue.

A wishlist item would be for it to produce the original idbloader.img and u-boot.itb files, at least for now.

Thanks!
Very Respectfully,
Peter Geis

#63 lore raw

Re: [PATCH v2 23/25] rockchip: Support building the all output files in binman

Hi Peter, Alper,

Quoted 90 lines · Peter Geis
Quoted 66 lines · Alper Nebi Yasak
Quoted 31 lines · Peter Geis
Quoted 30 lines · Simon Glass
arch/arm/dts/rockchip-u-boot.dtsi +19 -2
index eae3ee715d..64e4466489 100644
@@ -17,13 +17,93 @@
                filename = "u-boot-rockchip.bin";
                pad-byte = <0xff>;
 
-               blob {
-                       filename = "idbloader.img";
+#ifdef CONFIG_TPL
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-tpl {
+                       };
+               };
+
+               u-boot-spl {
                };
+#elif defined(CONFIG_SPL) /* SPL only */
+               mkimage {
+                       args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+
+                       u-boot-spl {
+                       };
+               };
+#endif
+#if defined(CONFIG_SPL_FIT) && defined(CONFIG_ARM64)
+               fit: fit {
Quoted 8 lines

arch/arm/dts/rockchip-u-boot.dtsi line 43

Quoted 4 lines
arch/arm/dts/rockchip-u-boot.dtsi +4 -0
+                       description = "FIT image for U-Boot with bl31 (TF-A)";
+                       #address-cells = <1>;
+                       fit,fdt-list = "of-list";
+                       fit,external-offset = <CONFIG_FIT_EXTERNAL_OFFSET>;

You need:
offset = <CONFIG_SPL_PAD_TO>;
here or the image is located at the wrong location.

Thanks for confirming this.

Quoted 14 lines

The image produced is not functional however, because it swaps the firmware node with the loadable-1 node.
Working image:
Configuration 0 (config_1)

  Description:  rk3399-pinephone-pro.dtb
  Kernel:       unavailable
  Firmware:     atf_1
  FDT:          fdt_1
  Loadables:    uboot
                atf_2
                atf_3

Non working image produced from this:
Configuration 0 (config-1)

Quoted 7 lines
  Description:  rk3399-rockpro64.dtb
  Kernel:       unavailable
  Firmware:     u-boot
  FDT:          fdt-1
  Loadables:    atf-1
                atf-2
                atf-3
Quoted 5 lines

And also this. I encountered the same things on rk3399-gru-kevin, mentioned them in my reply [1] to v1 but I guess Simon missed it.

[1] My reply to v1 of this patch
https://lore.kernel.org/u-boot/[email protected]/

Yes, that's the solution I ended up with as well.
Funnily enough, it seems loadables has a limit of five items, meaning the newest rk356x ATF images which have four binaries, u-boot, and optee trigger an error.

Quoted 9 lines
Quoted 4 lines

Also, it still doesn't support passing two separate files to mkimage at the same time, required for proper support of rk33xx_SPI generation and for rk35xx in general.

I got v1 of this booting from SPI, but on a board that doesn't use TPL. I see in doc/boards/rockchip.rst that one would indeed pass TPL+SPL to mkimage as two different files. I don't know how they're processed or anything about the rksd/rkspi formats though.

Quoted 13 lines

RKSPI mode splits the image up into 2048 chunks, and pads each chunk with 2048 of blank space.
For some reason the rk32/rk33 bootrom SPI code reads only 2048 out of each 4096 chunk.
Delightfully this seems to have been fixed in rk35xx.

Quoted 7 lines
Quoted 7 lines

Please see my standalone patch for what I hacked together to get that working:
http://patchwork.ozlabs.org/project/uboot/patch/[email protected]/
Note: those images are not functional either due to this issue.

A wishlist item would be for it to produce the original idbloader.img and u-boot.itb files, at least for now.

Thanks for all the info. I look forward to kevin being supported...but I should try jerry as well.

I hope to get this series applied soon and review the various follow-on patches. It would be great to sort things out this we as there are quite a few deficiencies in all of this, as you have found. But for now I will hold off on the rockchip-specific patches, until binman FIT is a bit better.

Regards,
Simon

#64 lore raw

PATCH v2 24/25 rockchip: Convert all boards to use binman

In reply to Simon Glass

Instead of the bash script, use binman to generate the FIT for arm64.

For 32-bit boards, use binman for all images, dropping the intermediate files.

With this change, only Zynq is now using SPL_FIT_GENERATOR so update the Kconfig rule accordingly.

Clean up the Makefile to the extent possible. Unfortunately, two boards do not use SPL_FRAMEWORK so don't enable the u-boot.img rule:

evb-rk3036
kylin-rk3036

So a small remnant remains.

---

(no changes since v1)

 Makefile     | 39 ++-------------------------------------
 boot/Kconfig |  2 +-
 2 files changed, 3 insertions(+), 38 deletions(-)
Makefile +2 -37
index 697cc51d67..7a4f95b898 100644
@@ -976,19 +976,8 @@ ifeq ($(CONFIG_MPC85xx)$(CONFIG_OF_SEPARATE),yy)
 INPUTS-y += u-boot-with-dtb.bin
 endif
 
-ifeq ($(CONFIG_ARCH_ROCKCHIP),y)
-# On ARM64 this target is produced by binman so we don't need this dep
-ifeq ($(CONFIG_ARM64),y)
-ifeq ($(CONFIG_SPL),y)
-# TODO: Get binman to generate this too
-INPUTS-y += u-boot-rockchip.bin
-endif
-else
-ifeq ($(CONFIG_SPL),y)
-# Generate these inputs for binman which will create the output files
-INPUTS-y += idbloader.img u-boot.img
-endif
-endif
+ifeq ($(CONFIG_ARCH_ROCKCHIP)_$(CONFIG_SPL_FRAMEWORK),y_)
+INPUTS-y += u-boot.img
 endif
 
 INPUTS-$(CONFIG_X86) += u-boot-x86-start16.bin u-boot-x86-reset16.bin \
@@ -1473,30 +1462,6 @@ OBJCOPYFLAGS_u-boot-with-spl.bin = -I binary -O binary \
 u-boot-with-spl.bin: $(SPL_IMAGE) $(SPL_PAYLOAD) FORCE
 	$(call if_changed,pad_cat)
 
-ifeq ($(CONFIG_ARCH_ROCKCHIP),y)
-
-# TPL + SPL
-ifeq ($(CONFIG_SPL)$(CONFIG_TPL),yy)
-MKIMAGEFLAGS_u-boot-tpl-rockchip.bin = -n $(CONFIG_SYS_SOC) -T rksd
-tpl/u-boot-tpl-rockchip.bin: tpl/u-boot-tpl.bin FORCE
-	$(call if_changed,mkimage)
-idbloader.img: tpl/u-boot-tpl-rockchip.bin spl/u-boot-spl.bin FORCE
-	$(call if_changed,cat)
-else
-MKIMAGEFLAGS_idbloader.img = -n $(CONFIG_SYS_SOC) -T rksd
-idbloader.img: spl/u-boot-spl.bin FORCE
-	$(call if_changed,mkimage)
-endif
-
-ifeq ($(CONFIG_ARM64),y)
-OBJCOPYFLAGS_u-boot-rockchip.bin = -I binary -O binary \
-	--pad-to=$(CONFIG_SPL_PAD_TO) --gap-fill=0xff
-u-boot-rockchip.bin: idbloader.img u-boot.itb FORCE
-	$(call if_changed,pad_cat)
-endif # CONFIG_ARM64
-
-endif # CONFIG_ARCH_ROCKCHIP
-
 ifeq ($(CONFIG_ARCH_LPC32XX)$(CONFIG_SPL),yy)
 MKIMAGEFLAGS_lpc32xx-spl.img = -T lpc32xximage -a $(CONFIG_SPL_TEXT_BASE)
 
boot/Kconfig +1 -1
index b83a4e8400..ae1a37ae92 100644
@@ -275,7 +275,7 @@ config SPL_FIT_SOURCE
 
 config USE_SPL_FIT_GENERATOR
 	bool "Use a script to generate the .its script"
-	default y if SPL_FIT && (!ARCH_SUNXI && !RISCV)
+	default y if SPL_FIT && ARCH_ZYNQMP
 
 config SPL_FIT_GENERATOR
 	string ".its file generator script for U-Boot FIT image"
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#65 lore raw

PATCH v2 25/25 rockchip: Drop the FIT generator script

In reply to Simon Glass

This is not used anymore. Drop it.

---

Changes in v2:
- Drop patches previously applied
- Add various suggestions from Alper Nebi Yasak
- Add patches to refactor binman's FIT support

 Makefile                               |   3 -
 arch/arm/mach-rockchip/make_fit_atf.py | 240 -------------------------
 boot/Kconfig                           |   1 -
 3 files changed, 244 deletions(-)
 delete mode 100755 arch/arm/mach-rockchip/make_fit_atf.py
Makefile +0 -3
index 7a4f95b898..f0b01a8f1b 100644
@@ -1352,9 +1352,6 @@ $(U_BOOT_ITS): $(subst ",,$(CONFIG_SPL_FIT_SOURCE))
 else
 ifneq ($(CONFIG_USE_SPL_FIT_GENERATOR),)
 U_BOOT_ITS := u-boot.its
-ifeq ($(CONFIG_SPL_FIT_GENERATOR),"arch/arm/mach-rockchip/make_fit_atf.py")
-U_BOOT_ITS_DEPS += u-boot
-endif
 $(U_BOOT_ITS): $(U_BOOT_ITS_DEPS) FORCE
 	$(srctree)/$(CONFIG_SPL_FIT_GENERATOR) \
 	$(patsubst %,arch/$(ARCH)/dts/%.dtb,$(subst ",,$(CONFIG_OF_LIST))) > $@
arch/arm/mach-rockchip/make_fit_atf.py deleted +0 -240

240 lines not on this page · raw message

boot/Kconfig +0 -1
index ae1a37ae92..1ae5b3a793 100644
@@ -280,7 +280,6 @@ config USE_SPL_FIT_GENERATOR
 config SPL_FIT_GENERATOR
 	string ".its file generator script for U-Boot FIT image"
 	depends on USE_SPL_FIT_GENERATOR
-	default "arch/arm/mach-rockchip/make_fit_atf.py" if SPL_LOAD_FIT && ARCH_ROCKCHIP
 	default "arch/arm/mach-zynqmp/mkimage_fit_atf.sh" if SPL_LOAD_FIT && ARCH_ZYNQMP
 	help
 	  Specifies a (platform specific) script file to generate the FIT
Signature · 1 line
2.35.1.574.g5d30c73bfb-goog
#66 lore raw

RFC PATCH 1/1 binman: support mkimage split files

In reply to Simon Glass

Good Evening,

I successfully tested your v2 patch series to create a bootable sdcard image out of the box for rockpro64-rk3399.
Unfortunately, rk356x and rk3399-spi modes are broken, due to the inability to pass multiple images to mkimage at the same time. rk3399-spi mode is already supported manually, see:
https://elixir.bootlin.com/u-boot/v2022.04-rc3/source/doc/board/rockchip/rockchip.rst#L182

rk356x is currently only supported manually, the image built by the old Makefile method is non functional. (u-boot-rockchip.bin)

Knowing absolutely nothing about python, I've hacked together something that works for splitting the image in the way mkimage expects. The file name passed to mkimage with the -d flag is:
./mkimage.simple-bin.mkimage.1:./mkimage.simple-bin.mkimage.2

I definitely don't expect this to be accepted as is, I just use it as an example of what we need to fully support this in binman. Adding the following allows me to build images automatically for rk356x:

mkimage {
	args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
	mkimage,separate_files;

	ddrl {
		type = "blob-ext";
		filename = "rk3568_ddr_1560MHz_v1.12.bin";
	};

	u-boot-spl {
	};
};

This is my first attempt to use in-reply-to, so I hope this works.

Thanks,
Peter Geis

---
 tools/binman/entry.py         | 43 ++++++++++++++++++++++++++---------
 tools/binman/etype/mkimage.py |  3 ++-
 2 files changed, 34 insertions(+), 12 deletions(-)
tools/binman/entry.py +32 -11
index 249f117ace56..48e552fc6af3 100644
@@ -114,6 +114,8 @@ class Entry(object):
         self.bintools = {}
         self.missing_bintools = []
         self.update_hash = True
+        self.fname_tmp = str()
+        self.index = 0
 
     @staticmethod
     def FindEntryClass(etype, expanded):
@@ -1134,7 +1136,7 @@ features to produce new behaviours.
         """
         self.update_hash = update_hash
 
-    def collect_contents_to_file(self, entries, prefix, fake_size=0):
+    def collect_contents_to_file(self, entries, prefix, fake_size=0, separate=False):
         """Put the contents of a list of entries into a file
 
         Args:
@@ -1152,13 +1154,32 @@ features to produce new behaviours.
                 str: Unique portion of filename (or None if no data)
         """
         data = b''
-        for entry in entries:
-            # First get the input data and put it in a file. If not available,
-            # try later.
-            if not entry.ObtainContents(fake_size=fake_size):
-                return None, None, None
-            data += entry.GetData()
-        uniq = self.GetUniqueName()
-        fname = tools.get_output_filename(f'{prefix}.{uniq}')
-        tools.write_file(fname, data)
-        return data, fname, uniq
+        if separate is False:
+            for entry in entries:
+                # First get the input data and put it in a file. If not available,
+                # try later.
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data += entry.GetData()
+            uniq = self.GetUniqueName()
+            fname = tools.get_output_filename(f'{prefix}.{uniq}')
+            tools.write_file(fname, data)
+            return data, fname, uniq
+        else:
+            for entry in entries:
+                self.index = (self.index + 1)
+                if (self.index > 2):
+                    print('BINMAN Warn: mkimage only supports a maximum of two separate files')
+                    break
+                # First get the input data and put it in a file. If not available,
+                # try later.
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data = entry.GetData()
+                uniq = self.GetUniqueName()
+                fname = tools.get_output_filename(f'{prefix}.{uniq}.{self.index}')
+                tools.write_file(fname, data)
+                self.fname_tmp = [''.join(self.fname_tmp),fname]
+            fname = ':'.join(self.fname_tmp)
+            uniq = self.GetUniqueName()
+            return data, fname, uniq
tools/binman/etype/mkimage.py +2 -1
index 5f6def2287f6..ce5f6b6b543a 100644
@@ -46,6 +46,7 @@ class Entry_mkimage(Entry):
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._mkimage_separate = fdt_util.GetBool(self._node, 'mkimage,separate_files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
@@ -53,7 +54,7 @@ class Entry_mkimage(Entry):
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
+            self._mkimage_entries.values(), 'mkimage', 1024, self._mkimage_separate)
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
Signature · 1 line
2.25.1
#67 lore raw

Re: [RFC] [PATCH] binman: support mkimage split files

Quoted 17 lines · Peter Geis

Good Evening,

I successfully tested your v2 patch series to create a bootable sdcard image out of the box for rockpro64-rk3399.
Unfortunately, rk356x and rk3399-spi modes are broken, due to the inability to pass multiple images to mkimage at the same time. rk3399-spi mode is already supported manually, see:
https://elixir.bootlin.com/u-boot/v2022.04-rc3/source/doc/board/rockchip/rockchip.rst#L182

rk356x is currently only supported manually, the image built by the old Makefile method is non functional. (u-boot-rockchip.bin)

Knowing absolutely nothing about python, I've hacked together something that works for splitting the image in the way mkimage expects. The file name passed to mkimage with the -d flag is:
./mkimage.simple-bin.mkimage.1:./mkimage.simple-bin.mkimage.2

Quoted 7 lines

I definitely don't expect this to be accepted as is, I just use it as an example of what we need to fully support this in binman. Adding the following allows me to build images automatically for rk356x:

mkimage {
	args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
	mkimage,separate_files;

Adding a property to toggle this sounds reasonable to me. The prefix might not be necessary, and I think dashes are preferred to underscores in property names.

Quoted 11 lines
	ddrl {
		type = "blob-ext";
		filename = "rk3568_ddr_1560MHz_v1.12.bin";
	};

	u-boot-spl {
	};
};

This is my first attempt to use in-reply-to, so I hope this works.

FYI, I see it as a reply to 00/25 of the series.

Quoted 65 lines

Thanks,
Peter Geis

---
 tools/binman/entry.py         | 43 ++++++++++++++++++++++++++---------
 tools/binman/etype/mkimage.py |  3 ++-
 2 files changed, 34 insertions(+), 12 deletions(-)
tools/binman/entry.py +22 -11
index 249f117ace56..48e552fc6af3 100644
@@ -114,6 +114,8 @@ class Entry(object):
         self.bintools = {}
         self.missing_bintools = []
         self.update_hash = True
+        self.fname_tmp = str()
+        self.index = 0
 
     @staticmethod
     def FindEntryClass(etype, expanded):
@@ -1134,7 +1136,7 @@ features to produce new behaviours.
         """
         self.update_hash = update_hash
 
-    def collect_contents_to_file(self, entries, prefix, fake_size=0):
+    def collect_contents_to_file(self, entries, prefix, fake_size=0, separate=False):
         """Put the contents of a list of entries into a file
 
         Args:
@@ -1152,13 +1154,32 @@ features to produce new behaviours.
                 str: Unique portion of filename (or None if no data)
         """
         data = b''
-        for entry in entries:
-            # First get the input data and put it in a file. If not available,
-            # try later.
-            if not entry.ObtainContents(fake_size=fake_size):
-                return None, None, None
-            data += entry.GetData()
-        uniq = self.GetUniqueName()
-        fname = tools.get_output_filename(f'{prefix}.{uniq}')
-        tools.write_file(fname, data)
-        return data, fname, uniq
+        if separate is False:
+            for entry in entries:
+                # First get the input data and put it in a file. If not available,
+                # try later.
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data += entry.GetData()
+            uniq = self.GetUniqueName()
+            fname = tools.get_output_filename(f'{prefix}.{uniq}')
+            tools.write_file(fname, data)
+            return data, fname, uniq
+        else:
+            for entry in entries:
+                self.index = (self.index + 1)
+                if (self.index > 2):
+                    print('BINMAN Warn: mkimage only supports a maximum of two separate files')
+                    break
+                # First get the input data and put it in a file. If not available,
+                # try later.

tools/binman/entry.py features to produce new behaviours. · line 1185

Quoted 10 lines
tools/binman/entry.py +10 -0
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data = entry.GetData()
+                uniq = self.GetUniqueName()
+                fname = tools.get_output_filename(f'{prefix}.{uniq}.{self.index}')
+                tools.write_file(fname, data)
+                self.fname_tmp = [''.join(self.fname_tmp),fname]
+            fname = ':'.join(self.fname_tmp)
+            uniq = self.GetUniqueName()
+            return data, fname, uniq

I would keep this function as-is and call it multiple times in the mkimage etype code below (once per subentry), and also do the mkimage-specific checks and 'file1:file2' argument joining there as well.

Quoted 21 lines
tools/binman/etype/mkimage.py +2 -1
index 5f6def2287f6..ce5f6b6b543a 100644
@@ -46,6 +46,7 @@ class Entry_mkimage(Entry):
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._mkimage_separate = fdt_util.GetBool(self._node, 'mkimage,separate_files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
@@ -53,7 +54,7 @@ class Entry_mkimage(Entry):
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
+            self._mkimage_entries.values(), 'mkimage', 1024, self._mkimage_separate)
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
#68 lore raw

Re: [RFC] [PATCH] binman: support mkimage split files

Quoted 23 lines · Alper Nebi Yasak
Quoted 21 lines · Peter Geis

Good Evening,

I successfully tested your v2 patch series to create a bootable sdcard image out of the box for rockpro64-rk3399.
Unfortunately, rk356x and rk3399-spi modes are broken, due to the inability to pass multiple images to mkimage at the same time. rk3399-spi mode is already supported manually, see:
https://elixir.bootlin.com/u-boot/v2022.04-rc3/source/doc/board/rockchip/rockchip.rst#L182

rk356x is currently only supported manually, the image built by the old Makefile method is non functional. (u-boot-rockchip.bin)

Knowing absolutely nothing about python, I've hacked together something that works for splitting the image in the way mkimage expects. The file name passed to mkimage with the -d flag is:
./mkimage.simple-bin.mkimage.1:./mkimage.simple-bin.mkimage.2

I definitely don't expect this to be accepted as is, I just use it as an example of what we need to fully support this in binman. Adding the following allows me to build images automatically for rk356x:

Quoted 7 lines
Quoted 3 lines
mkimage {
      args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
      mkimage,separate_files;

Adding a property to toggle this sounds reasonable to me. The prefix might not be necessary, and I think dashes are preferred to underscores in property names.

Thanks! I thought including mkimage as a prefix was necessary to make it clear what was happening, but on second thought it's in the mkimage node.

Quoted 7 lines
Quoted 7 lines

      ddrl {
              type = "blob-ext";
              filename = "rk3568_ddr_1560MHz_v1.12.bin";
      };
Quoted 7 lines
Quoted 5 lines
      u-boot-spl {
      };
};

This is my first attempt to use in-reply-to, so I hope this works.

FYI, I see it as a reply to 00/25 of the series.

I appreciate the confirmation.

Quoted 72 lines
Quoted 72 lines

Thanks,
Peter Geis

---
 tools/binman/entry.py         | 43 ++++++++++++++++++++++++++---------
 tools/binman/etype/mkimage.py |  3 ++-
 2 files changed, 34 insertions(+), 12 deletions(-)
tools/binman/entry.py +28 -11
index 249f117ace56..48e552fc6af3 100644
@@ -114,6 +114,8 @@ class Entry(object):
         self.bintools = {}
         self.missing_bintools = []
         self.update_hash = True
+        self.fname_tmp = str()
+        self.index = 0
 
     @staticmethod
     def FindEntryClass(etype, expanded):
@@ -1134,7 +1136,7 @@ features to produce new behaviours.
         """
         self.update_hash = update_hash
 
-    def collect_contents_to_file(self, entries, prefix, fake_size=0):
+    def collect_contents_to_file(self, entries, prefix, fake_size=0, separate=False):
         """Put the contents of a list of entries into a file
 
         Args:
@@ -1152,13 +1154,32 @@ features to produce new behaviours.
                 str: Unique portion of filename (or None if no data)
         """
         data = b''
-        for entry in entries:
-            # First get the input data and put it in a file. If not available,
-            # try later.
-            if not entry.ObtainContents(fake_size=fake_size):
-                return None, None, None
-            data += entry.GetData()
-        uniq = self.GetUniqueName()
-        fname = tools.get_output_filename(f'{prefix}.{uniq}')
-        tools.write_file(fname, data)
-        return data, fname, uniq
+        if separate is False:
+            for entry in entries:
+                # First get the input data and put it in a file. If not available,
+                # try later.
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data += entry.GetData()
+            uniq = self.GetUniqueName()
+            fname = tools.get_output_filename(f'{prefix}.{uniq}')
+            tools.write_file(fname, data)
+            return data, fname, uniq
+        else:
+            for entry in entries:
+                self.index = (self.index + 1)
+                if (self.index > 2):
+                    print('BINMAN Warn: mkimage only supports a maximum of two separate files')
+                    break
+                # First get the input data and put it in a file. If not available,
+                # try later.
+                if not entry.ObtainContents(fake_size=fake_size):
+                    return None, None, None
+                data = entry.GetData()
+                uniq = self.GetUniqueName()
+                fname = tools.get_output_filename(f'{prefix}.{uniq}.{self.index}')
+                tools.write_file(fname, data)
Quoted 8 lines

tools/binman/entry.py features to produce new behaviours. · line 1185

Quoted 4 lines
tools/binman/entry.py +4 -0
+                self.fname_tmp = [''.join(self.fname_tmp),fname]
+            fname = ':'.join(self.fname_tmp)
+            uniq = self.GetUniqueName()
+            return data, fname, uniq

I would keep this function as-is and call it multiple times in the mkimage etype code below (once per subentry), and also do the mkimage-specific checks and 'file1:file2' argument joining there as well.

Hmm, thinking about it I think I can pull this off.
My first (non public) attempt was something similar to this, which just hardcoded the first file name.

I'm not kidding when I said all the python I know I learned to make this happen, so it's a learning event.
Thanks for the ideas!

Quoted 22 lines
Quoted 22 lines
tools/binman/etype/mkimage.py +2 -1
index 5f6def2287f6..ce5f6b6b543a 100644
@@ -46,6 +46,7 @@ class Entry_mkimage(Entry):
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._mkimage_separate = fdt_util.GetBool(self._node, 'mkimage,separate_files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
@@ -53,7 +54,7 @@ class Entry_mkimage(Entry):
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy
         data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
+            self._mkimage_entries.values(), 'mkimage', 1024, self._mkimage_separate)
         if data is None:
             return False
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
#69 lore raw

PATCH v2 1/1 binman: support mkimage separate files

mkimage has the ability to process two files at the same time. This is necessary for rk356x support as both TPL and SPL need to be hashed individually in the resulting header.
It also eases support for rkspi as mkimage handles everything for making the requisite file for maskrom loading.

Add a new flag "separate_files" for mkimage handling to gather the files separately rather than combining them before passing them to mkimage.

---
Changelog:
v2:
I've managed to move this all into mkimage.py as per Alper's suggestion. I've added an example to the readme portion of the function. mkimage,separate_files is now separate-files.

 tools/binman/etype/mkimage.py | 41 ++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)
tools/binman/etype/mkimage.py +38 -3
index 5f6def2287f6..0a86f904a2b7 100644
@@ -17,6 +17,7 @@ class Entry_mkimage(Entry):
     Properties / Entry arguments:
         - datafile: Filename for -d argument
         - args: Other arguments to pass
+        - separate-files: Boolean flag for passing files individually.
 
     The data passed to mkimage is collected from subnodes of the mkimage node,
     e.g.::
@@ -42,20 +43,54 @@ class Entry_mkimage(Entry):
             };
         };
 
+    This calls mkimage to create a rksd image with a standalone ram init
+    binary file and u-boot-spl.bin as individual input files. The output from
+    mkimage then becomes part of the image produced by binman.
+
+        mkimage {
+            args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+            separate-files;
+
+            ddrl {
+                type = "blob-ext";
+                filename = "rk3568_ddr_1560MHz_v1.12.bin";
+            };
+
+            u-boot-spl {
+            };
+        };
+
     """
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._separate = fdt_util.GetBool(self._node, 'separate-files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
+        self.index = 0
+        self.fname_tmp = str()
 
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy
-        data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
-        if data is None:
+        if self._separate is False:
+          data, input_fname, uniq = self.collect_contents_to_file(
+              self._mkimage_entries.values(), 'mkimage', 1024)
+          if data is None:
             return False
+        else:
+          for entry in self._mkimage_entries.values():
+            self.index = (self.index + 1)
+            if (self.index > 2):
+              print('BINMAN Warn: mkimage only supports a maximum of two separate files')
+              return False
+            input_fname = ['mkimage', str(self.index)]
+            data, input_fname, uniq = self.collect_contents_to_file(
+                    [entry], ".".join(input_fname), 1024)
+            if data is None:
+              return False
+            self.fname_tmp = [''.join(self.fname_tmp),input_fname]
+          input_fname = ":".join(self.fname_tmp)
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
                                 output_fname) is not None:
Signature · 1 line
2.25.1
#70 lore raw

Re: [PATCH v2] binman: support mkimage separate files

Hi Peter,

Quoted 6 lines · Peter Geis

mkimage has the ability to process two files at the same time. This is necessary for rk356x support as both TPL and SPL need to be hashed individually in the resulting header.
It also eases support for rkspi as mkimage handles everything for making the requisite file for maskrom loading.

This makes me wonder if we should move that functoinality out of mkimage and into binman?

Quoted 61 lines

Add a new flag "separate_files" for mkimage handling to gather the files separately rather than combining them before passing them to mkimage.

---
Changelog:
v2:
I've managed to move this all into mkimage.py as per Alper's suggestion. I've added an example to the readme portion of the function. mkimage,separate_files is now separate-files.

 tools/binman/etype/mkimage.py | 41 ++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)
tools/binman/etype/mkimage.py +21 -0
index 5f6def2287f6..0a86f904a2b7 100644
@@ -17,6 +17,7 @@ class Entry_mkimage(Entry):
     Properties / Entry arguments:
         - datafile: Filename for -d argument
         - args: Other arguments to pass
+        - separate-files: Boolean flag for passing files individually.
 
     The data passed to mkimage is collected from subnodes of the mkimage node,
     e.g.::
@@ -42,20 +43,54 @@ class Entry_mkimage(Entry):
             };
         };
 
+    This calls mkimage to create a rksd image with a standalone ram init
+    binary file and u-boot-spl.bin as individual input files. The output from
+    mkimage then becomes part of the image produced by binman.
+
+        mkimage {
+            args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+            separate-files;
+
+            ddrl {
+                type = "blob-ext";
+                filename = "rk3568_ddr_1560MHz_v1.12.bin";
+            };
+
+            u-boot-spl {
+            };
+        };
+
     """
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._separate = fdt_util.GetBool(self._node, 'separate-files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
+        self.index = 0
+        self.fname_tmp = str()
 
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 82

Quoted 10 lines
tools/binman/etype/mkimage.py +6 -3
-        data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
-        if data is None:
+        if self._separate is False:
+          data, input_fname, uniq = self.collect_contents_to_file(
+              self._mkimage_entries.values(), 'mkimage', 1024)
+          if data is None:
             return False
+        else:
+          for entry in self._mkimage_entries.values():

We can do:

for index, entry in enumerate(self._mkimage_entries.values()):

then you don't need self.index

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 85

Quoted 3 lines
tools/binman/etype/mkimage.py +3 -0
+            self.index = (self.index + 1)
+            if (self.index > 2):
+              print('BINMAN Warn: mkimage only supports a maximum of two separate files')

Can we use self.Raise() here instead? It seems like a fatal error. Also this check should go in ReadNode() since we don't want to die this late in the picture if we know it is wrong upfront. (BTW I am moving the node-reading code to ReadNode() in my v3 series as suggested by Alper).

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 89

Quoted 4 lines
tools/binman/etype/mkimage.py +4 -0
+              return False
+            input_fname = ['mkimage', str(self.index)]
+            data, input_fname, uniq = self.collect_contents_to_file(
+                    [entry], ".".join(input_fname), 1024)

I suppose we can just use

data = entry.GetData()

here?

Quoted 10 lines
tools/binman/etype/mkimage.py +4 -1
+            if data is None:
+              return False
+            self.fname_tmp = [''.join(self.fname_tmp),input_fname]
+          input_fname = ":".join(self.fname_tmp)
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
                                 output_fname) is not None:
--

2.25.1

Looks OK to me, needs a test or two.

Regards,
Simon

#71 lore raw

Re: [PATCH v2] binman: support mkimage separate files

Quoted 2 lines · Simon Glass

Hi Peter,

Good Morning,

Quoted 11 lines · Peter Geis
Quoted 6 lines

mkimage has the ability to process two files at the same time. This is necessary for rk356x support as both TPL and SPL need to be hashed individually in the resulting header.
It also eases support for rkspi as mkimage handles everything for making the requisite file for maskrom loading.

This makes me wonder if we should move that functoinality out of mkimage and into binman?

Rockchip rk32 and rk33 maskrom loading from SPI has a bug that causes it to only load 2048 bytes out of each 4096 byte chunk. RKSPI splits the TPL/SPL (the portion directly loaded by maskrom) into 2048 chunks then pads each chunk with blank space so the image can load correctly.
While it could be moved to binman, as far as I'm aware this is a corner case and I don't know any other chip that would need this functionality.

Quoted 75 lines
Quoted 62 lines

Add a new flag "separate_files" for mkimage handling to gather the files separately rather than combining them before passing them to mkimage.

---
Changelog:
v2:
I've managed to move this all into mkimage.py as per Alper's suggestion. I've added an example to the readme portion of the function. mkimage,separate_files is now separate-files.

 tools/binman/etype/mkimage.py | 41 ++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)
tools/binman/etype/mkimage.py +21 -0
index 5f6def2287f6..0a86f904a2b7 100644
@@ -17,6 +17,7 @@ class Entry_mkimage(Entry):
     Properties / Entry arguments:
         - datafile: Filename for -d argument
         - args: Other arguments to pass
+        - separate-files: Boolean flag for passing files individually.
 
     The data passed to mkimage is collected from subnodes of the mkimage node,
     e.g.::
@@ -42,20 +43,54 @@ class Entry_mkimage(Entry):
             };
         };
 
+    This calls mkimage to create a rksd image with a standalone ram init
+    binary file and u-boot-spl.bin as individual input files. The output from
+    mkimage then becomes part of the image produced by binman.
+
+        mkimage {
+            args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+            separate-files;
+
+            ddrl {
+                type = "blob-ext";
+                filename = "rk3568_ddr_1560MHz_v1.12.bin";
+            };
+
+            u-boot-spl {
+            };
+        };
+
     """
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._separate = fdt_util.GetBool(self._node, 'separate-files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
+        self.index = 0
+        self.fname_tmp = str()
 
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 82

Quoted 10 lines
tools/binman/etype/mkimage.py +6 -3
-        data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
-        if data is None:
+        if self._separate is False:
+          data, input_fname, uniq = self.collect_contents_to_file(
+              self._mkimage_entries.values(), 'mkimage', 1024)
+          if data is None:
             return False
+        else:
+          for entry in self._mkimage_entries.values():

We can do:

Quoted 3 lines

for index, entry in enumerate(self._mkimage_entries.values()):

then you don't need self.index

Thanks!

Quoted 10 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 85

Quoted 4 lines
tools/binman/etype/mkimage.py +3 -0
+            self.index = (self.index + 1)
+            if (self.index > 2):
+              print('BINMAN Warn: mkimage only supports a maximum of two separate files')

Can we use self.Raise() here instead? It seems like a fatal error. Also this check should go in ReadNode() since we don't want to die this late in the picture if we know it is wrong upfront. (BTW I am moving the node-reading code to ReadNode() in my v3 series as suggested by Alper).

Certainly, this really would be a fatal error.

Quoted 9 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 89

Quoted 5 lines
tools/binman/etype/mkimage.py +4 -0
+              return False
+            input_fname = ['mkimage', str(self.index)]
+            data, input_fname, uniq = self.collect_contents_to_file(
+                    [entry], ".".join(input_fname), 1024)

I suppose we can just use

data = entry.GetData()

We don't actually use data directly here, collect_contents_to_file collects the data into separate files and passes the file name back. data is just used to tell if that function failed, the file names are what we care about.
Really as far as I can tell collect_contents_to_file doesn't need to pass data back at all, because input_fname and uniq would be returned False as well and either could be used for this check. uniq is also used later on (I checked, each time returns the same value so clobbering it in each iteration doesn't cause issues).

Quoted 14 lines

here?

Quoted 10 lines
tools/binman/etype/mkimage.py +4 -1
+            if data is None:
+              return False
+            self.fname_tmp = [''.join(self.fname_tmp),input_fname]
+          input_fname = ":".join(self.fname_tmp)
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
                                 output_fname) is not None:
--

2.25.1

Quoted 4 lines

Looks OK to me, needs a test or two.

Regards,
Simon

Honestly, if you can implement this better than I did in your series, please do. As I said previously, all the python I know now I learned to make this happen, so I imagine it is not optimal.

Very Respectfully,
Peter

#72 lore raw

Re: [PATCH v2] binman: support mkimage separate files

Quoted 10 lines · Peter Geis
Quoted 9 lines · Simon Glass
Quoted 5 lines · Peter Geis

mkimage has the ability to process two files at the same time. This is necessary for rk356x support as both TPL and SPL need to be hashed individually in the resulting header.
It also eases support for rkspi as mkimage handles everything for making the requisite file for maskrom loading.

This makes me wonder if we should move that functoinality out of mkimage and into binman?

I think we should have entry types for the formats mkimage supports, even if they're just stubs that run 'mkimage -T <type>'. Primarily because I don't see 'mkimage' as a meaningful 'type' of an entry, but I don't know if there's a common format to all the types it supports. Creating stub entries would also let us switch to native implementations one by one if we want that later.

Quoted 8 lines

Rockchip rk32 and rk33 maskrom loading from SPI has a bug that causes it to only load 2048 bytes out of each 4096 byte chunk. RKSPI splits the TPL/SPL (the portion directly loaded by maskrom) into 2048 chunks then pads each chunk with blank space so the image can load correctly.
While it could be moved to binman, as far as I'm aware this is a corner case and I don't know any other chip that would need this functionality.

Do you know if rk35xx chips have the same bug? Does rkspi also do the weird chunking for those when it maybe doesn't need to?

Quoted 49 lines
Quoted 49 lines
Quoted 49 lines

Add a new flag "separate_files" for mkimage handling to gather the files separately rather than combining them before passing them to mkimage.

---
Changelog:
v2:
I've managed to move this all into mkimage.py as per Alper's suggestion. I've added an example to the readme portion of the function. mkimage,separate_files is now separate-files.

 tools/binman/etype/mkimage.py | 41 ++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)
tools/binman/etype/mkimage.py +18 -0
index 5f6def2287f6..0a86f904a2b7 100644
@@ -17,6 +17,7 @@ class Entry_mkimage(Entry):
     Properties / Entry arguments:
         - datafile: Filename for -d argument
         - args: Other arguments to pass
+        - separate-files: Boolean flag for passing files individually.
 
     The data passed to mkimage is collected from subnodes of the mkimage node,
     e.g.::
@@ -42,20 +43,54 @@ class Entry_mkimage(Entry):
             };
         };
 
+    This calls mkimage to create a rksd image with a standalone ram init
+    binary file and u-boot-spl.bin as individual input files. The output from
+    mkimage then becomes part of the image produced by binman.
+
+        mkimage {
+            args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+            separate-files;
+
+            ddrl {
+                type = "blob-ext";
+                filename = "rk3568_ddr_1560MHz_v1.12.bin";
+            };
+
+            u-boot-spl {
+            };
+        };
+
     """
     def __init__(self, section, etype, node):
Quoted 8 lines
Quoted 8 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 72

Quoted 8 lines
tools/binman/etype/mkimage.py +3 -0
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._separate = fdt_util.GetBool(self._node, 'separate-files')
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
+        self.index = 0
+        self.fname_tmp = str()

These two wouldn't be useful for anything except the function below, so should be defined there as local variables.

Quoted 27 lines
Quoted 14 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 82

Quoted 13 lines
tools/binman/etype/mkimage.py +6 -3
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy
-        data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
-        if data is None:
+        if self._separate is False:
+          data, input_fname, uniq = self.collect_contents_to_file(
+              self._mkimage_entries.values(), 'mkimage', 1024)
+          if data is None:
             return False
+        else:
+          for entry in self._mkimage_entries.values():
Quoted 5 lines

We can do:

for index, entry in enumerate(self._mkimage_entries.values()):

then you don't need self.index

Thanks!

Quoted 5 lines
Quoted 4 lines
tools/binman/etype/mkimage.py +3 -0
+            self.index = (self.index + 1)
+            if (self.index > 2):
+              print('BINMAN Warn: mkimage only supports a maximum of two separate files')
Quoted 7 lines
Quoted 5 lines

Can we use self.Raise() here instead? It seems like a fatal error. Also this check should go in ReadNode() since we don't want to die this late in the picture if we know it is wrong upfront. (BTW I am moving the node-reading code to ReadNode() in my v3 series as suggested by Alper).

Certainly, this really would be a fatal error.

(The "BINMAN Warn:" prefix should also be dropped with self.Raise())

Quoted 6 lines
Quoted 6 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 89

Quoted 6 lines
tools/binman/etype/mkimage.py +4 -0
+              return False
+            input_fname = ['mkimage', str(self.index)]
+            data, input_fname, uniq = self.collect_contents_to_file(
+                    [entry], ".".join(input_fname), 1024)

This could have f"mkimage.{index}" as the prefix argument instead.

Quoted 5 lines
Quoted 4 lines

I suppose we can just use

data = entry.GetData()

Quoted 9 lines

We don't actually use data directly here, collect_contents_to_file collects the data into separate files and passes the file name back. data is just used to tell if that function failed, the file names are what we care about.
Really as far as I can tell collect_contents_to_file doesn't need to pass data back at all, because input_fname and uniq would be returned False as well and either could be used for this check. uniq is also used later on (I checked, each time returns the same value so clobbering it in each iteration doesn't cause issues).

Yeah, I think collect_contents_to_file() is better here for the one-entry case as well. The alternative is doing exactly everything it does manually.

Quoted 7 lines
Quoted 7 lines

here?

Quoted 4 lines
tools/binman/etype/mkimage.py +4 -0
+            if data is None:
+              return False
+            self.fname_tmp = [''.join(self.fname_tmp),input_fname]
+          input_fname = ":".join(self.fname_tmp)

Instead of these, I'd set input_fnames = [] before the for loop, use input_fnames.append(input_fname) inside the loop to collect the names, then ":".join(input_fnames) to build the argument.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
tools/binman/etype/mkimage.py +0 -1
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
                                 output_fname) is not None:
--

2.25.1

Quoted 8 lines
Quoted 4 lines

Looks OK to me, needs a test or two.

Regards,
Simon

Honestly, if you can implement this better than I did in your series, please do. As I said previously, all the python I know now I learned to make this happen, so I imagine it is not optimal.

Well, you're 90% there already (with the other 90% being the tests...), but if you don't feel like working on this, tell me and I can do so in a few days.

#73 lore raw

Re: [PATCH v2] binman: support mkimage separate files

Quoted 29 lines · Alper Nebi Yasak
Quoted 10 lines · Peter Geis
Quoted 9 lines · Simon Glass
Quoted 5 lines · Peter Geis

mkimage has the ability to process two files at the same time. This is necessary for rk356x support as both TPL and SPL need to be hashed individually in the resulting header.
It also eases support for rkspi as mkimage handles everything for making the requisite file for maskrom loading.

This makes me wonder if we should move that functoinality out of mkimage and into binman?

I think we should have entry types for the formats mkimage supports, even if they're just stubs that run 'mkimage -T <type>'. Primarily because I don't see 'mkimage' as a meaningful 'type' of an entry, but I don't know if there's a common format to all the types it supports. Creating stub entries would also let us switch to native implementations one by one if we want that later.

Quoted 8 lines

Rockchip rk32 and rk33 maskrom loading from SPI has a bug that causes it to only load 2048 bytes out of each 4096 byte chunk. RKSPI splits the TPL/SPL (the portion directly loaded by maskrom) into 2048 chunks then pads each chunk with blank space so the image can load correctly.
While it could be moved to binman, as far as I'm aware this is a corner case and I don't know any other chip that would need this functionality.

Quoted 2 lines

Do you know if rk35xx chips have the same bug? Does rkspi also do the weird chunking for those when it maybe doesn't need to?

From my limited testing, no rk35xx seems to have fixed this bug. Thus this series gives me hope for "unified" images for SPI and MMC where the offsets are the same, and the only thing that changes is the TPL/SPL actually loaded in place.

Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines

Add a new flag "separate_files" for mkimage handling to gather the files separately rather than combining them before passing them to mkimage.

---
Changelog:
v2:
I've managed to move this all into mkimage.py as per Alper's suggestion. I've added an example to the readme portion of the function. mkimage,separate_files is now separate-files.

 tools/binman/etype/mkimage.py | 41 ++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)
tools/binman/etype/mkimage.py +19 -0
index 5f6def2287f6..0a86f904a2b7 100644
@@ -17,6 +17,7 @@ class Entry_mkimage(Entry):
     Properties / Entry arguments:
         - datafile: Filename for -d argument
         - args: Other arguments to pass
+        - separate-files: Boolean flag for passing files individually.
 
     The data passed to mkimage is collected from subnodes of the mkimage node,
     e.g.::
@@ -42,20 +43,54 @@ class Entry_mkimage(Entry):
             };
         };
 
+    This calls mkimage to create a rksd image with a standalone ram init
+    binary file and u-boot-spl.bin as individual input files. The output from
+    mkimage then becomes part of the image produced by binman.
+
+        mkimage {
+            args = "-n", CONFIG_SYS_SOC, "-T", "rksd";
+            separate-files;
+
+            ddrl {
+                type = "blob-ext";
+                filename = "rk3568_ddr_1560MHz_v1.12.bin";
+            };
+
+            u-boot-spl {
+            };
+        };
+
     """
     def __init__(self, section, etype, node):
         super().__init__(section, etype, node)
         self._args = fdt_util.GetArgs(self._node, 'args')
+        self._separate = fdt_util.GetBool(self._node, 'separate-files')
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 72

Quoted 5 lines
tools/binman/etype/mkimage.py +2 -0
         self._mkimage_entries = OrderedDict()
         self.align_default = None
         self.ReadEntries()
+        self.index = 0
+        self.fname_tmp = str()

These two wouldn't be useful for anything except the function below, so should be defined there as local variables.

Thanks!

Quoted 40 lines
Quoted 28 lines
Quoted 15 lines
Quoted 4 lines
tools/binman/etype/mkimage.py +0 -0
     def ObtainContents(self):
         # Use a non-zero size for any fake files to keep mkimage happy

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 82

Quoted 10 lines
tools/binman/etype/mkimage.py +6 -3
-        data, input_fname, uniq = self.collect_contents_to_file(
-            self._mkimage_entries.values(), 'mkimage', 1024)
-        if data is None:
+        if self._separate is False:
+          data, input_fname, uniq = self.collect_contents_to_file(
+              self._mkimage_entries.values(), 'mkimage', 1024)
+          if data is None:
             return False
+        else:
+          for entry in self._mkimage_entries.values():
Quoted 5 lines

We can do:

for index, entry in enumerate(self._mkimage_entries.values()):

then you don't need self.index

Thanks!

Quoted 5 lines
Quoted 4 lines
tools/binman/etype/mkimage.py +3 -0
+            self.index = (self.index + 1)
+            if (self.index > 2):
+              print('BINMAN Warn: mkimage only supports a maximum of two separate files')
Quoted 7 lines
Quoted 5 lines

Can we use self.Raise() here instead? It seems like a fatal error. Also this check should go in ReadNode() since we don't want to die this late in the picture if we know it is wrong upfront. (BTW I am moving the node-reading code to ReadNode() in my v3 series as suggested by Alper).

Certainly, this really would be a fatal error.

(The "BINMAN Warn:" prefix should also be dropped with self.Raise())

Quoted 2 lines
Quoted 2 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 89

Quoted 4 lines
tools/binman/etype/mkimage.py +4 -0
+              return False
+            input_fname = ['mkimage', str(self.index)]
+            data, input_fname, uniq = self.collect_contents_to_file(
+                    [entry], ".".join(input_fname), 1024)

This could have f"mkimage.{index}" as the prefix argument instead.

I have no idea what that means.

Quoted 16 lines
Quoted 6 lines
Quoted 5 lines

I suppose we can just use

data = entry.GetData()

Quoted 9 lines

We don't actually use data directly here, collect_contents_to_file collects the data into separate files and passes the file name back. data is just used to tell if that function failed, the file names are what we care about.
Really as far as I can tell collect_contents_to_file doesn't need to pass data back at all, because input_fname and uniq would be returned False as well and either could be used for this check. uniq is also used later on (I checked, each time returns the same value so clobbering it in each iteration doesn't cause issues).

Quoted 3 lines

Yeah, I think collect_contents_to_file() is better here for the one-entry case as well. The alternative is doing exactly everything it does manually.

Yeah V1 actually implemented that all in a modified collect_contents_to_file(), so this is much simpler already.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

here?

Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

tools/binman/etype/mkimage.py class Entry_mkimage(Entry): · line 93

Quoted 4 lines
tools/binman/etype/mkimage.py +4 -0
+            if data is None:
+              return False
+            self.fname_tmp = [''.join(self.fname_tmp),input_fname]
+          input_fname = ":".join(self.fname_tmp)

Instead of these, I'd set input_fnames = [] before the for loop, use input_fnames.append(input_fname) inside the loop to collect the names, then ":".join(input_fnames) to build the argument.

Ah, there's the little python details I don't know about yet.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
tools/binman/etype/mkimage.py +0 -1
         output_fname = tools.get_output_filename('mkimage-out.%s' % uniq)
         if self.mkimage.run_cmd('-d', input_fname, *self._args,
                                 output_fname) is not None:
--

2.25.1

Looks OK to me, needs a test or two.

Regards,
Simon

Quoted 7 lines
Quoted 3 lines

Honestly, if you can implement this better than I did in your series, please do. As I said previously, all the python I know now I learned to make this happen, so I imagine it is not optimal.

Well, you're 90% there already (with the other 90% being the tests...), but if you don't feel like working on this, tell me and I can do so in a few days.

If you would like to, please do!
Otherwise, I will continue stumbling around with it once this series settles down.

Thanks,
Peter