binman: Template fixes and improvements
v2 cover binman: Template fixes and improvements
With the basic template feature in place, some problems have come to light.
Firstly, keeping the template around while processing entries seems unnecessary and perhaps confusing, so this is removed.
Secondly this series aims to support phandles in a more intuitive way, rather than just ignoring them in templates. It includes an experimental patch to copy phandles from template so that it is possible to so something like:
template {
some_node: some-node {
};
};
image {
insert-template = <&template>;
};with the some_node phandle being copied to the 'image' node, to result in:
image {
insert-template = <&template>; some_node: some-node {
};};
Changes in v2:
- Use a 'done' variable to reduce code duplication
- Handle phandle copying property and report duplicates
- Refine support for phandles to deal with duplicates
- Add a test that deals with duplicate phandles
Simon Glass (6):
binman: Produce a template-file after processing
dtoc: Make properties dirty when purging them
dtoc: Add some debugging when copying nodes
fdt: Allow copying phandles into templates
binman: Remove templates after use
binman: Support templates containing phandles
tools/binman/binman.rst | 26 +++++++- tools/binman/control.py | 32 ++++++++- tools/binman/ftest.py | 43 ++++++++++++ tools/binman/test/291_template_phandle.dts | 51 +++++++++++++++ .../binman/test/292_template_phandle_dup.dts | 65 +++++++++++++++++++ tools/dtoc/fdt.py | 34 +++++++--- tools/dtoc/test/dtoc_test_copy.dts | 6 +- tools/dtoc/test_fdt.py | 40 +++++++++--- 8 files changed, 274 insertions(+), 23 deletions(-) create mode 100644 tools/binman/test/291_template_phandle.dts create mode 100644 tools/binman/test/292_template_phandle_dup.dts
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v2 1/6 binman: Produce a template-file after processing
This file aids debugging when binman fails to get far enough to write out the final devicetree file. Write it immediate after template processing.
---
(no changes since v1)
tools/binman/binman.rst | 4 ++++ tools/binman/control.py | 14 ++++++++++++-- tools/binman/ftest.py | 9 +++++++++ 3 files changed, 25 insertions(+), 2 deletions(-)
tools/binman/binman.rst +4 -0
@@ -1256,6 +1256,10 @@ Properties in the template node are inserted into the destination node if they do not exist there. In the example above, `some-property` is added to each of `spi-image` and `mmc-image`. +The initial devicetree produced by the templating process is written to the +`u-boot.dtb.tmpl1` file. This can be useful to see what is going on if there is +a failure before the final `u-boot.dtb.out` file is written. + Note that template nodes are not removed from the binman description at present.
tools/binman/control.py +12 -2
@@ -486,6 +486,9 @@ def _ProcessTemplates(parent): Args: parent: Binman node to process (typically /binman) + Returns: + bool: True if any templates were processed + Search though each target node looking for those with an 'insert-template' property. Use that as a list of references to template nodes to use to adjust the target node.
@@ -498,11 +501,15 @@ def _ProcessTemplates(parent): See 'Templates' in the Binman documnentation for details. """ + found = False for node in parent.subnodes: tmpl = fdt_util.GetPhandleList(node, 'insert-template') if tmpl: node.copy_subnodes_from_phandles(tmpl) - _ProcessTemplates(node) + found = True + + found |= _ProcessTemplates(node) + return found def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): """Prepare the images to be processed and select the device tree
@@ -546,7 +553,10 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): raise ValueError("Device tree '%s' does not have a 'binman' " "node" % dtb_fname) - _ProcessTemplates(node) + if _ProcessTemplates(node): + dtb.Sync(True) + fname = tools.get_output_filename('u-boot.dtb.tmpl1') + tools.write_file(fname, dtb.GetContents()) images = _ReadImageDesc(node, use_expanded)
tools/binman/ftest.py +9 -0
@@ -6793,6 +6793,15 @@ fdt fdtmap Extract the devicetree blob from the fdtmap second = U_BOOT_DATA + b'#' + VGA_DATA + U_BOOT_DTB_DATA self.assertEqual(U_BOOT_IMG_DATA + first + second, data) + dtb_fname1 = tools.get_output_filename('u-boot.dtb.tmpl1') + self.assertTrue(os.path.exists(dtb_fname1)) + dtb = fdt.Fdt.FromData(tools.read_file(dtb_fname1)) + dtb.Scan() + node1 = dtb.GetNode('/binman/template') + self.assertTrue(node1) + vga = dtb.GetNode('/binman/first/intel-vga') + self.assertTrue(vga) + def testTemplateBlobMulti(self): """Test using a template with 'multiple-images' enabled""" TestFunctional._MakeInputFile('my-blob.bin', b'blob')
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v2 1/6] binman: Produce a template-file after processing
This file aids debugging when binman fails to get far enough to write out the final devicetree file. Write it immediate after template processing.
---
(no changes since v1)
tools/binman/binman.rst | 4 ++++ tools/binman/control.py | 14 ++++++++++++-- tools/binman/ftest.py | 9 +++++++++ 3 files changed, 25 insertions(+), 2 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 2/6 dtoc: Make properties dirty when purging them
In reply to Simon GlassWithout the 'dirty' flag properties are not written back to the devicetree when synced. This means that new properties copied over to a node are not always written out.
Fix this and add a test.
---
(no changes since v1)
tools/dtoc/fdt.py | 1 + tools/dtoc/test/dtoc_test_copy.dts | 6 ++++-- tools/dtoc/test_fdt.py | 16 +++++++++++++--- 3 files changed, 18 insertions(+), 5 deletions(-)
tools/dtoc/fdt.py +1 -0
@@ -272,6 +272,7 @@ class Prop: the FDT is synced """ self._offset = None + self.dirty = True class Node: """A device tree node
tools/dtoc/test/dtoc_test_copy.dts +4 -2
@@ -37,11 +37,12 @@ new-prop; }; - second1 { + second1: second1 { new-prop; }; second4 { + use_second1 = <&second1>; }; }; };
@@ -65,12 +66,13 @@ }; second: second { - second1 { + second_1_bad: second1 { some-prop; }; second2 { some-prop; + use_second1_bad = <&second_1_bad>; }; }; };
tools/dtoc/test_fdt.py +13 -3
@@ -308,7 +308,7 @@ class TestNode(unittest.TestCase): def test_copy_node(self): """Test copy_node() function""" - def do_copy_checks(dtb, dst, expect_none): + def do_copy_checks(dtb, dst, second1_ph_val, expect_none): self.assertEqual( ['/dest/base', '/dest/first@0', '/dest/existing'], [n.path for n in dst.subnodes])
@@ -365,12 +365,22 @@ class TestNode(unittest.TestCase): ['second1', 'second2', 'second3', 'second4'], [n.name for n in second.subnodes]) + # Check the 'second_1_bad' phandle is not copied over + second1 = second.FindNode('second1') + self.assertTrue(second1) + sph = second1.props.get('phandle') + self.assertTrue(sph) + self.assertEqual(second1_ph_val, sph.bytes) + + dtb = fdt.FdtScan(find_dtb_file('dtoc_test_copy.dts')) tmpl = dtb.GetNode('/base') dst = dtb.GetNode('/dest') + second1_ph_val = (dtb.GetNode('/dest/base/second/second1'). + props['phandle'].bytes) dst.copy_node(tmpl) - do_copy_checks(dtb, dst, expect_none=True) + do_copy_checks(dtb, dst, second1_ph_val, expect_none=True) dtb.Sync(auto_resize=True)
@@ -378,7 +388,7 @@ class TestNode(unittest.TestCase): new_dtb = fdt.Fdt.FromData(dtb.GetContents()) new_dtb.Scan() dst = new_dtb.GetNode('/dest') - do_copy_checks(new_dtb, dst, expect_none=False) + do_copy_checks(new_dtb, dst, second1_ph_val, expect_none=False) def test_copy_subnodes_from_phandles(self): """Test copy_node() function"""
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v2 2/6] dtoc: Make properties dirty when purging them
Without the 'dirty' flag properties are not written back to the devicetree when synced. This means that new properties copied over to a node are not always written out.
Fix this and add a test.
---
(no changes since v1)
tools/dtoc/fdt.py | 1 + tools/dtoc/test/dtoc_test_copy.dts | 6 ++++-- tools/dtoc/test_fdt.py | 16 +++++++++++++--- 3 files changed, 18 insertions(+), 5 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 3/6 dtoc: Add some debugging when copying nodes
In reply to Simon GlassShow the operations being performed, when debugging is enabled.
Convert a mistaken 'print' in test_copy_subnodes_from_phandles() while we are here.
---
Changes in v2:
- Use a 'done' variable to reduce code duplication
tools/dtoc/fdt.py | 5 +++++ tools/dtoc/test_fdt.py | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-)
tools/dtoc/fdt.py +5 -0
@@ -249,6 +249,7 @@ class Prop: """ if self.dirty: node = self._node + tout.debug(f'sync {node.path}: {self.name}') fdt_obj = node._fdt._fdt_obj node_name = fdt_obj.get_name(node._offset) if node_name and node_name != node.name:
@@ -716,9 +717,13 @@ class Node: 'phandle' property is not copied since this might result in two nodes with the same phandle, thus making phandle references ambiguous. """ + tout.debug(f'copy to {self.path}: {src.path}') for name, src_prop in src.props.items(): + done = False if name != 'phandle' and name not in self.props: self.props[name] = Prop(self, None, name, src_prop.bytes) + done = True + tout.debug(f" {name}{'' if done else ' - ignored'}") def copy_node(self, src): """Copy a node and all its subnodes into this node
tools/dtoc/test_fdt.py +2 -1
@@ -32,6 +32,7 @@ from dtoc.fdt import Type, BytesToValue import libfdt from u_boot_pylib import test_util from u_boot_pylib import tools +from u_boot_pylib import tout #pylint: disable=protected-access
@@ -414,7 +415,7 @@ class TestNode(unittest.TestCase): # Make sure that the phandle for 'over' is not copied over = dst.FindNode('over') - print('keys', over.props.keys()) + tout.debug(f'keys: {over.props.keys()}') self.assertNotIn('phandle', over.props.keys()) # Check the merged properties, first the base ones in '/dest'
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v2 3/6] dtoc: Add some debugging when copying nodes
Show the operations being performed, when debugging is enabled.
Convert a mistaken 'print' in test_copy_subnodes_from_phandles() while we are here.
---
Changes in v2:
- Use a 'done' variable to reduce code duplication
tools/dtoc/fdt.py | 5 +++++ tools/dtoc/test_fdt.py | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-)
Applied to u-boot-dm, thanks!
PATCH v2 4/6 fdt: Allow copying phandles into templates
In reply to Simon GlassAllow phandles to be copied over from a template. This can potentially cause duplicate phandles, so detect this and report an error.
---
Changes in v2:
- Handle phandle copying property and report duplicates
tools/dtoc/fdt.py | 28 +++++++++++++++++++--------- tools/dtoc/test_fdt.py | 21 +++++++++++++++++---- 2 files changed, 36 insertions(+), 13 deletions(-)
tools/dtoc/fdt.py +19 -9
@@ -337,6 +337,11 @@ class Node: self.props = self._fdt.GetProps(self) phandle = fdt_obj.get_phandle(self.Offset()) if phandle: + dup = self._fdt.phandle_to_node.get(phandle) + if dup: + raise ValueError( + f'Duplicate phandle {phandle} in nodes {dup.path} and {self.path}') + self._fdt.phandle_to_node[phandle] = self offset = fdt_obj.first_subnode(self.Offset(), QUIET_NOTFOUND)
@@ -707,11 +712,12 @@ class Node: prop.Sync(auto_resize) return added - def merge_props(self, src): + def merge_props(self, src, copy_phandles): """Copy missing properties (except 'phandle') from another node Args: src (Node): Node containing properties to copy + copy_phandles (bool): True to copy phandle properties in nodes Adds properties which are present in src but not in this node. Any 'phandle' property is not copied since this might result in two nodes
@@ -720,21 +726,24 @@ class Node: tout.debug(f'copy to {self.path}: {src.path}') for name, src_prop in src.props.items(): done = False - if name != 'phandle' and name not in self.props: - self.props[name] = Prop(self, None, name, src_prop.bytes) - done = True + if name not in self.props: + if copy_phandles or name != 'phandle': + self.props[name] = Prop(self, None, name, src_prop.bytes) + done = True tout.debug(f" {name}{'' if done else ' - ignored'}") - def copy_node(self, src): + def copy_node(self, src, copy_phandles=False): """Copy a node and all its subnodes into this node Args: src (Node): Node to copy + copy_phandles (bool): True to copy phandle properties in nodes Returns: Node: Resulting destination node - This works recursively. + This works recursively, with copy_phandles being set to True for the + recursive calls The new node is put before all other nodes. If the node already exists, just its subnodes and properties are copied, placing them before
@@ -746,12 +755,12 @@ class Node: dst.move_to_first() else: dst = self.insert_subnode(src.name) - dst.merge_props(src) + dst.merge_props(src, copy_phandles) # Process in reverse order so that they appear correctly in the result, # since copy_node() puts the node first in the list for node in reversed(src.subnodes): - dst.copy_node(node) + dst.copy_node(node, True) return dst def copy_subnodes_from_phandles(self, phandle_list):
@@ -774,7 +783,7 @@ class Node: dst = self.copy_node(node) tout.debug(f'merge props from {parent.path} to {dst.path}') - self.merge_props(parent) + self.merge_props(parent, False) class Fdt:
@@ -835,6 +844,7 @@ class Fdt: TODO([email protected]): Implement the 'root' parameter """ + self.phandle_to_node = {} self._cached_offsets = True self._root = self.Node(self, None, 0, '/', '/') self._root.Scan()
tools/dtoc/test_fdt.py +17 -4
@@ -340,8 +340,8 @@ class TestNode(unittest.TestCase): over = dtb.GetNode('/dest/base/over') self.assertTrue(over) - # Make sure that the phandle for 'over' is not copied - self.assertNotIn('phandle', over.props.keys()) + # Make sure that the phandle for 'over' is copied + self.assertIn('phandle', over.props.keys()) second = dtb.GetNode('/dest/base/second') self.assertTrue(second)
@@ -349,7 +349,7 @@ class TestNode(unittest.TestCase): [n.name for n in chk.subnodes]) self.assertEqual(chk, over.parent) self.assertEqual( - {'bootph-all', 'compatible', 'reg', 'low-power'}, + {'bootph-all', 'compatible', 'reg', 'low-power', 'phandle'}, over.props.keys()) if expect_none:
@@ -385,9 +385,22 @@ class TestNode(unittest.TestCase): dtb.Sync(auto_resize=True) - # Now check that the FDT looks correct + # Now check the resulting FDT. It should have duplicate phandles since + # 'over' has been copied to 'dest/base/over' but still exists in its old + # place new_dtb = fdt.Fdt.FromData(dtb.GetContents()) + with self.assertRaises(ValueError) as exc: + new_dtb.Scan() + self.assertIn( + 'Duplicate phandle 1 in nodes /dest/base/over and /base/over', + str(exc.exception)) + + # Remove the source nodes for the copy + new_dtb.GetNode('/base').Delete() + + # Now it should scan OK new_dtb.Scan() + dst = new_dtb.GetNode('/dest') do_copy_checks(new_dtb, dst, second1_ph_val, expect_none=False)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v2 5/6 binman: Remove templates after use
In reply to Simon GlassIt is not necessary to keep templates around after they have been processed. They can cause confusion and potentially duplicate phandles.
Remove them.
Use the same means of detecting a template node in _ReadImageDesc so that the two places are consistent.
---
(no changes since v1)
tools/binman/binman.rst | 8 +++++--- tools/binman/control.py | 14 +++++++++++++- tools/binman/ftest.py | 7 +++++++ 3 files changed, 25 insertions(+), 4 deletions(-)
tools/binman/binman.rst +5 -3
@@ -1256,11 +1256,13 @@ Properties in the template node are inserted into the destination node if they do not exist there. In the example above, `some-property` is added to each of `spi-image` and `mmc-image`. +Note that template nodes are removed from the binman description after +processing and before binman builds the image descriptions. + The initial devicetree produced by the templating process is written to the `u-boot.dtb.tmpl1` file. This can be useful to see what is going on if there is -a failure before the final `u-boot.dtb.out` file is written. - -Note that template nodes are not removed from the binman description at present. +a failure before the final `u-boot.dtb.out` file is written. A second +`u-boot.dtb.tmpl2` file is written when the templates themselves are removed. Updating an ELF file
tools/binman/control.py +13 -1
@@ -57,7 +57,7 @@ def _ReadImageDesc(binman_node, use_expanded): images = OrderedDict() if 'multiple-images' in binman_node.props: for node in binman_node.subnodes: - if 'template' not in node.name: + if not node.name.startswith('template'): images[node.name] = Image(node.name, node, use_expanded=use_expanded) else:
@@ -511,6 +511,13 @@ def _ProcessTemplates(parent): found |= _ProcessTemplates(node) return found +def _RemoveTemplates(parent): + """Remove any templates in the binman description + """ + for node in parent.subnodes: + if node.name.startswith('template'): + node.Delete() + def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): """Prepare the images to be processed and select the device tree
@@ -558,6 +565,11 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): fname = tools.get_output_filename('u-boot.dtb.tmpl1') tools.write_file(fname, dtb.GetContents()) + _RemoveTemplates(node) + dtb.Sync(True) + fname = tools.get_output_filename('u-boot.dtb.tmpl2') + tools.write_file(fname, dtb.GetContents()) + images = _ReadImageDesc(node, use_expanded) if select_images:
tools/binman/ftest.py +7 -0
@@ -6802,6 +6802,13 @@ fdt fdtmap Extract the devicetree blob from the fdtmap vga = dtb.GetNode('/binman/first/intel-vga') self.assertTrue(vga) + dtb_fname2 = tools.get_output_filename('u-boot.dtb.tmpl2') + self.assertTrue(os.path.exists(dtb_fname2)) + dtb2 = fdt.Fdt.FromData(tools.read_file(dtb_fname2)) + dtb2.Scan() + node2 = dtb2.GetNode('/binman/template') + self.assertFalse(node2) + def testTemplateBlobMulti(self): """Test using a template with 'multiple-images' enabled""" TestFunctional._MakeInputFile('my-blob.bin', b'blob')
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v2 5/6] binman: Remove templates after use
It is not necessary to keep templates around after they have been processed. They can cause confusion and potentially duplicate phandles.
Remove them.
Use the same means of detecting a template node in _ReadImageDesc so that the two places are consistent.
---
(no changes since v1)
tools/binman/binman.rst | 8 +++++--- tools/binman/control.py | 14 +++++++++++++- tools/binman/ftest.py | 7 +++++++ 3 files changed, 25 insertions(+), 4 deletions(-)
Applied to u-boot-dm, thanks!
PATCH v2 6/6 binman: Support templates containing phandles
In reply to Simon GlassThis provides support for phandles to be copied over from templates. This is not quite safe, since if the template is instantiated twice (i.e. in two different nodes), then duplicate phandles will be found. This will result in an error.
---
Changes in v2:
- Refine support for phandles to deal with duplicates
- Add a test that deals with duplicate phandles
tools/binman/binman.rst | 18 +++++ tools/binman/control.py | 4 ++ tools/binman/ftest.py | 27 ++++++++ tools/binman/test/291_template_phandle.dts | 51 +++++++++++++++ .../binman/test/292_template_phandle_dup.dts | 65 +++++++++++++++++++ 5 files changed, 165 insertions(+) create mode 100644 tools/binman/test/291_template_phandle.dts create mode 100644 tools/binman/test/292_template_phandle_dup.dts
tools/binman/binman.rst +18 -0
@@ -1264,6 +1264,24 @@ The initial devicetree produced by the templating process is written to the a failure before the final `u-boot.dtb.out` file is written. A second `u-boot.dtb.tmpl2` file is written when the templates themselves are removed. +Dealing with phandles +--------------------- + +Templates can contain phandles and these are copied to the destination node. +However this should be used with care, since if a template is instantiated twice +then the phandle will be copied twice, resulting in a devicetree with duplicate +phandles, i.e. the same phandle used by two different nodes. Binman detects this +situation and produces an error, for example:: + + Duplicate phandle 1 in nodes /binman/image/fit/images/atf/atf-bl31 and + /binman/image-2/fit/images/atf/atf-bl31 + +In this case an atf-bl31 node containing a phandle has been copied into two +different target nodes, resulting in the same phandle for each. See +testTemplatePhandleDup() for the test case. + +The solution is typically to put the phandles in the corresponding target nodes +(one for each) and remove the phandle from the template. Updating an ELF file ====================
tools/binman/control.py +4 -0
@@ -567,6 +567,10 @@ def PrepareImagesAndDtbs(dtb_fname, select_images, update_fdt, use_expanded): _RemoveTemplates(node) dtb.Sync(True) + + # Rescan the dtb to pick up the new phandles + dtb.Scan() + node = _FindBinmanNode(dtb) fname = tools.get_output_filename('u-boot.dtb.tmpl2') tools.write_file(fname, dtb.GetContents())
tools/binman/ftest.py +27 -0
@@ -6900,6 +6900,33 @@ fdt fdtmap Extract the devicetree blob from the fdtmap # Move to next spl_data = content[:0x18] + def testTemplatePhandle(self): + """Test using a template in a node containing a phandle""" + entry_args = { + 'atf-bl31-path': 'bl31.elf', + } + data = self._DoReadFileDtb('291_template_phandle.dts', + entry_args=entry_args) + fname = tools.get_output_filename('image.bin') + out = tools.run('dumpimage', '-l', fname) + + # We should see the FIT description and one for each of the two images + lines = out.splitlines() + descs = [line.split()[-1] for line in lines if 'escription' in line] + self.assertEqual(['test-desc', 'atf', 'fdt'], descs) + + def testTemplatePhandleDup(self): + """Test using a template in a node containing a phandle""" + entry_args = { + 'atf-bl31-path': 'bl31.elf', + } + with self.assertRaises(ValueError) as e: + self._DoReadFileDtb('292_template_phandle_dup.dts', + entry_args=entry_args) + self.assertIn( + 'Duplicate phandle 1 in nodes /binman/image/fit/images/atf/atf-bl31 and /binman/image-2/fit/images/atf/atf-bl31', + str(e.exception)) + if __name__ == "__main__": unittest.main()
tools/binman/test/291_template_phandle.dts new file +51 -0
51 lines not on this page · raw message
tools/binman/test/292_template_phandle_dup.dts new file +65 -0
65 lines not on this page · raw message
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v2 6/6] binman: Support templates containing phandles
This provides support for phandles to be copied over from templates. This is not quite safe, since if the template is instantiated twice (i.e. in two different nodes), then duplicate phandles will be found. This will result in an error.
---
Changes in v2:
- Refine support for phandles to deal with duplicates
- Add a test that deals with duplicate phandles
tools/binman/binman.rst | 18 +++++ tools/binman/control.py | 4 ++ tools/binman/ftest.py | 27 ++++++++ tools/binman/test/291_template_phandle.dts | 51 +++++++++++++++ .../binman/test/292_template_phandle_dup.dts | 65 +++++++++++++++++++ 5 files changed, 165 insertions(+) create mode 100644 tools/binman/test/291_template_phandle.dts create mode 100644 tools/binman/test/292_template_phandle_dup.dts
Applied to u-boot-dm, thanks!