Skip to contentSkip to conversation

binman: Support compressing files in parallel

PATCH In review Simon Glass 1 participant 13 messages Jul 6, 2021 – Jul 22, 2021

lore
Share & tools
lore
#0 lore raw

cover binman: Support compressing files in parallel

When building an image with large files it makes sense to compress them in parallel. This series adds support for this as well a few other performance-related tweaks. It also includes support for timing of the different types of operation (e.g. compression), for debugging purposes.

Simon Glass (6):
binman: Put compressed data into separate files
binman: Support multithreading for building images
binman: Split node-reading out from constructor in files
binman: Use bytearray instead of string
patman: Use bytearray instead of string
binman: Add basic support for debugging performance

 tools/binman/binman.rst          | 18 ++++++
 tools/binman/cmdline.py          |  4 ++
 tools/binman/control.py          |  7 +++
 tools/binman/etype/blob.py       |  5 ++
 tools/binman/etype/collection.py |  2 +-
 tools/binman/etype/files.py      |  3 +
 tools/binman/etype/section.py    | 40 ++++++++++++--
 tools/binman/ftest.py            | 41 ++++++++++++--
 tools/binman/image.py            |  3 +
 tools/binman/state.py            | 95 ++++++++++++++++++++++++++++++++
 tools/patman/cros_subprocess.py  |  6 +-
 tools/patman/tools.py            |  9 ++-
 12 files changed, 218 insertions(+), 15 deletions(-)
Signature · 1 line
2.32.0.93.g670b81a890-goog
#1 lore raw

PATCH 1/6 binman: Put compressed data into separate files

At present compression uses the same temporary file for all invocations. With multithreading this causes the data to become corrupted. Use a different filename each time.

---

 tools/patman/tools.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
tools/patman/tools.py +7 -2
index ec95a543bd9..877e37cd8da 100644
@@ -466,6 +466,9 @@ def Compress(indata, algo, with_header=True):
     This requires 'lz4' and 'lzma_alone' tools. It also requires an output
     directory to be previously set up, by calling PrepareOutputDir().
 
+    Care is taken to use unique temporary files so that this function can be
+    called from multiple threads.
+
     Args:
         indata: Input data to compress
         algo: Algorithm to use ('none', 'gzip', 'lz4' or 'lzma')
@@ -475,14 +478,16 @@ def Compress(indata, algo, with_header=True):
     """
     if algo == 'none':
         return indata
-    fname = GetOutputFilename('%s.comp.tmp' % algo)
+    fname = tempfile.NamedTemporaryFile(prefix='%s.comp.tmp' % algo,
+                                        dir=outdir).name
     WriteFile(fname, indata)
     if algo == 'lz4':
         data = Run('lz4', '--no-frame-crc', '-B4', '-5', '-c', fname,
                    binary=True)
     # cbfstool uses a very old version of lzma
     elif algo == 'lzma':
-        outfname = GetOutputFilename('%s.comp.otmp' % algo)
+        outfname = tempfile.NamedTemporaryFile(prefix='%s.comp.otmp' % algo,
+                                               dir=outdir).name
         Run('lzma_alone', 'e', fname, outfname, '-lc1', '-lp0', '-pb0', '-d8')
         data = ReadFile(outfname)
     elif algo == 'gzip':
Signature · 1 line
2.32.0.93.g670b81a890-goog
#2 lore raw

Re: [PATCH 1/6] binman: Put compressed data into separate files

At present compression uses the same temporary file for all invocations. With multithreading this causes the data to become corrupted. Use a different filename each time.

---

 tools/patman/tools.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

Applied to u-boot-dm, thanks!

#3 lore raw

PATCH 2/6 binman: Support multithreading for building images

In reply to Simon Glass

Some images may take a while to build, e.g. if they are large and use slow compression. Support compiling sections in parallel to speed things up.

---

 tools/binman/binman.rst       | 18 ++++++++++++++++++
 tools/binman/cmdline.py       |  4 ++++
 tools/binman/control.py       |  4 ++++
 tools/binman/etype/section.py | 36 ++++++++++++++++++++++++++++++++---
 tools/binman/ftest.py         | 33 ++++++++++++++++++++++++++++----
 tools/binman/image.py         |  3 +++
 tools/binman/state.py         | 23 ++++++++++++++++++++++
 7 files changed, 114 insertions(+), 7 deletions(-)
tools/binman/binman.rst +18 -0
index bc635aa00a5..09e7b571982 100644
@@ -1142,6 +1142,22 @@ adds a -v<level> option to the call to binman::
    make BINMAN_VERBOSE=5
 
 
+Building sections in parallel
+-----------------------------
+
+By default binman uses multiprocessing to speed up compilation of large images.
+This works at a section level, with one thread for each entry in the section.
+This can speed things up if the entries are large and use compression.
+
+This feature can be disabled with the '-T' flag, which defaults to a suitable
+value for your machine. This depends on the Python version, e.g on v3.8 it uses
+12 threads on an 8-core machine. See ConcurrentFutures_ for more details.
+
+The special value -T0 selects single-threaded mode, useful for debugging during
+development, since dealing with exceptions and problems in threads is more
+difficult. This avoids any use of ThreadPoolExecutor.
+
+
 History / Credits
 -----------------
 
@@ -1190,3 +1206,5 @@ Some ideas:
 --
 Simon Glass <[email protected]>
 7/7/2016
+
+.. _ConcurrentFutures: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
tools/binman/cmdline.py +4 -0
index 95f9ba27fbd..d6156df408b 100644
@@ -32,6 +32,10 @@ controlled by a description in the board device tree.'''
         default=False, help='Display the README file')
     parser.add_argument('--toolpath', type=str, action='append',
         help='Add a path to the directories containing tools')
+    parser.add_argument('-T', '--threads', type=int,
+          default=None, help='Number of threads to use (0=single-thread)')
+    parser.add_argument('--test-section-timeout', action='store_true',
+          help='Use a zero timeout for section multi-threading (for testing)')
     parser.add_argument('-v', '--verbosity', default=1,
         type=int, help='Control verbosity: 0=silent, 1=warnings, 2=notices, '
         '3=info, 4=detail, 5=debug')
tools/binman/control.py +4 -0
index f57e34daaaa..b2113b6e64f 100644
@@ -628,9 +628,13 @@ def Binman(args):
             tools.PrepareOutputDir(args.outdir, args.preserve)
             tools.SetToolPaths(args.toolpath)
             state.SetEntryArgs(args.entry_arg)
+            state.SetThreads(args.threads)
 
             images = PrepareImagesAndDtbs(dtb_fname, args.image,
                                           args.update_fdt, use_expanded)
+            if args.test_section_timeout:
+                # Set the first image to timeout, used in testThreadTimeout()
+                images[list(images.keys())[0]].test_section_timeout = True
             missing = False
             for image in images.values():
                 missing |= ProcessImage(image, args.update_fdt, args.map,
tools/binman/etype/section.py +33 -3
index c3bac026c14..92d3f3add4c 100644
@@ -9,10 +9,12 @@ images to be created.
 """
 
 from collections import OrderedDict
+import concurrent.futures
 import re
 import sys
 
 from binman.entry import Entry
+from binman import state
 from dtoc import fdt_util
 from patman import tools
 from patman import tout
@@ -525,15 +527,43 @@ class Entry_section(Entry):
     def GetEntryContents(self):
         """Call ObtainContents() for each entry in the section
         """
+        def _CheckDone(entry):
+            if not entry.ObtainContents():
+                next_todo.append(entry)
+            return entry
+
         todo = self._entries.values()
         for passnum in range(3):
+            threads = state.GetThreads()
             next_todo = []
-            for entry in todo:
-                if not entry.ObtainContents():
-                    next_todo.append(entry)
+
+            if threads == 0:
+                for entry in todo:
+                    _CheckDone(entry)
+            else:
+                with concurrent.futures.ThreadPoolExecutor(
+                        max_workers=threads) as executor:
+                    future_to_data = {
+                        entry: executor.submit(_CheckDone, entry)
+                        for entry in todo}
+                    timeout = 60
+                    if self.GetImage().test_section_timeout:
+                        timeout = 0
+                    done, not_done = concurrent.futures.wait(
+                        future_to_data.values(), timeout=timeout)
+                    # Make sure we check the result, so any exceptions are
+                    # generated. Check the results in entry order, since tests
+                    # may expect earlier entries to fail first.
+                    for entry in todo:
+                        job = future_to_data[entry]
+                        job.result()
+                    if not_done:
+                        self.Raise('Timed out obtaining contents')
+
             todo = next_todo
             if not todo:
                 break
+
         if todo:
             self.Raise('Internal error: Could not complete processing of contents: remaining %s' %
                        todo)
tools/binman/ftest.py +29 -4
index 14daa69a69b..ceaa6a8737c 100644
@@ -315,7 +315,8 @@ class TestFunctional(unittest.TestCase):
     def _DoTestFile(self, fname, debug=False, map=False, update_dtb=False,
                     entry_args=None, images=None, use_real_dtb=False,
                     use_expanded=False, verbosity=None, allow_missing=False,
-                    extra_indirs=None):
+                    extra_indirs=None, threads=None,
+                    test_section_timeout=False):
         """Run binman with a given test file
 
         Args:
@@ -338,6 +339,8 @@ class TestFunctional(unittest.TestCase):
             allow_missing: Set the '--allow-missing' flag so that missing
                 external binaries just produce a warning instead of an error
             extra_indirs: Extra input directories to add using -I
+            threads: Number of threads to use (None for default, 0 for
+                single-threaded)
         """
         args = []
         if debug:
@@ -349,6 +352,10 @@ class TestFunctional(unittest.TestCase):
         if self.toolpath:
             for path in self.toolpath:
                 args += ['--toolpath', path]
+        if threads is not None:
+            args.append('-T%d' % threads)
+        if test_section_timeout:
+            args.append('--test-section-timeout')
         args += ['build', '-p', '-I', self._indir, '-d', self.TestFile(fname)]
         if map:
             args.append('-m')
@@ -381,7 +388,7 @@ class TestFunctional(unittest.TestCase):
             fname: Filename of .dts file to read
             outfile: Output filename for compiled device-tree binary
 
-        Returns:
+        #Returns:
             Contents of device-tree binary
         """
         tmpdir = tempfile.mkdtemp(prefix='binmant.')
@@ -419,7 +426,7 @@ class TestFunctional(unittest.TestCase):
 
     def _DoReadFileDtb(self, fname, use_real_dtb=False, use_expanded=False,
                        map=False, update_dtb=False, entry_args=None,
-                       reset_dtbs=True, extra_indirs=None):
+                       reset_dtbs=True, extra_indirs=None, threads=None):
         """Run binman and return the resulting image
 
         This runs binman with a given test file and then reads the resulting
@@ -446,6 +453,8 @@ class TestFunctional(unittest.TestCase):
                 function. If reset_dtbs is True, then the original test dtb
                 is written back before this function finishes
             extra_indirs: Extra input directories to add using -I
+            threads: Number of threads to use (None for default, 0 for
+                single-threaded)
 
         Returns:
             Tuple:
@@ -470,7 +479,8 @@ class TestFunctional(unittest.TestCase):
         try:
             retcode = self._DoTestFile(fname, map=map, update_dtb=update_dtb,
                     entry_args=entry_args, use_real_dtb=use_real_dtb,
-                    use_expanded=use_expanded, extra_indirs=extra_indirs)
+                    use_expanded=use_expanded, extra_indirs=extra_indirs,
+                    threads=threads)
             self.assertEqual(0, retcode)
             out_dtb_fname = tools.GetOutputFilename('u-boot.dtb.out')
 
@@ -4609,6 +4619,21 @@ class TestFunctional(unittest.TestCase):
         self.assertIn('Expected __bss_size symbol in vpl/u-boot-vpl',
                       str(e.exception))
 
+    def testSectionsSingleThread(self):
+        """Test sections without multithreading"""
+        data = self._DoReadFileDtb('055_sections.dts', threads=0)[0]
+        expected = (U_BOOT_DATA + tools.GetBytes(ord('!'), 12) +
+                    U_BOOT_DATA + tools.GetBytes(ord('a'), 12) +
+                    U_BOOT_DATA + tools.GetBytes(ord('&'), 4))
+        self.assertEqual(expected, data)
+
+    def testThreadTimeout(self):
+        """Test handling a thread that takes too long"""
+        with self.assertRaises(ValueError) as e:
+            self._DoTestFile('055_sections.dts', test_section_timeout=True)
+        self.assertIn("Node '/binman/section@0': Timed out obtaining contents",
+                      str(e.exception))
+
 
 if __name__ == "__main__":
     unittest.main()
tools/binman/image.py +3 -0
index 10778f47fe9..cdc58b39a40 100644
@@ -36,6 +36,8 @@ class Image(section.Entry_section):
         fdtmap_data: Contents of the fdtmap when loading from a file
         allow_repack: True to add properties to allow the image to be safely
             repacked later
+        test_section_timeout: Use a zero timeout for section multi-threading
+            (for testing)
 
     Args:
         copy_to_orig: Copy offset/size to orig_offset/orig_size after reading
@@ -74,6 +76,7 @@ class Image(section.Entry_section):
         self.allow_repack = False
         self._ignore_missing = ignore_missing
         self.use_expanded = use_expanded
+        self.test_section_timeout = False
         if not test:
             self.ReadNode()
 
tools/binman/state.py +23 -0
index e4a3abd2805..809429fbd9d 100644
@@ -7,6 +7,7 @@
 
 import hashlib
 import re
+import threading
 
 from dtoc import fdt
 import os
@@ -56,6 +57,9 @@ allow_entry_expansion = True
 # to the new ones, the compressed size increases, etc.
 allow_entry_contraction = False
 
+# Number of threads to use for binman (None means machine-dependent)
+num_threads = None
+
 def GetFdtForEtype(etype):
     """Get the Fdt object for a particular device-tree entry
 
@@ -421,3 +425,22 @@ def AllowEntryContraction():
             raised
     """
     return allow_entry_contraction
+
+def SetThreads(threads):
+    """Set the number of threads to use when building sections
+
+    Args:
+        threads: Number of threads to use (None for default, 0 for
+            single-threaded)
+    """
+    global num_threads
+
+    num_threads = threads
+
+def GetThreads():
+    """Get the number of threads to use when building sections
+
+    Returns:
+        Number of threads to use (None for default, 0 for single-threaded)
+    """
+    return num_threads
Signature · 1 line
2.32.0.93.g670b81a890-goog
#4 lore raw

Re: [PATCH 2/6] binman: Support multithreading for building images

Some images may take a while to build, e.g. if they are large and use slow compression. Support compiling sections in parallel to speed things up.

---

 tools/binman/binman.rst       | 18 ++++++++++++++++++
 tools/binman/cmdline.py       |  4 ++++
 tools/binman/control.py       |  4 ++++
 tools/binman/etype/section.py | 36 ++++++++++++++++++++++++++++++++---
 tools/binman/ftest.py         | 33 ++++++++++++++++++++++++++++----
 tools/binman/image.py         |  3 +++
 tools/binman/state.py         | 23 ++++++++++++++++++++++
 7 files changed, 114 insertions(+), 7 deletions(-)

Applied to u-boot-dm, thanks!

#5 lore raw

PATCH 3/6 binman: Split node-reading out from constructor in files

In reply to Simon Glass

The constructor should not read the node information. Move it to the ReadNode() method instead. This allows this etype to be subclassed.

---

 tools/binman/etype/files.py | 3 +++
 1 file changed, 3 insertions(+)
tools/binman/etype/files.py +3 -0
index 5db36abef0b..9b04a496a85 100644
@@ -34,6 +34,9 @@ class Entry_files(Entry_section):
         from binman import state
 
         super().__init__(section, etype, node)
+
+    def ReadNode(self):
+        super().ReadNode()
         self._pattern = fdt_util.GetString(self._node, 'pattern')
         if not self._pattern:
             self.Raise("Missing 'pattern' property")
Signature · 1 line
2.32.0.93.g670b81a890-goog
#6 lore raw

Re: [PATCH 3/6] binman: Split node-reading out from constructor in files

The constructor should not read the node information. Move it to the ReadNode() method instead. This allows this etype to be subclassed.

---

 tools/binman/etype/files.py | 3 +++
 1 file changed, 3 insertions(+)

Applied to u-boot-dm, thanks!

#7 lore raw

PATCH 4/6 binman: Use bytearray instead of string

In reply to Simon Glass

This is faster if data is being concatenated. Update the section and collection etypes.

---

 tools/binman/etype/collection.py | 2 +-
 tools/binman/etype/section.py    | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
tools/binman/etype/collection.py +1 -1
index 1625575fe98..442b40b48b3 100644
@@ -40,7 +40,7 @@ class Entry_collection(Entry):
         """
         # Join up all the data
         self.Info('Getting contents, required=%s' % required)
-        data = b''
+        data = bytearray()
         for entry_phandle in self.content:
             entry_data = self.section.GetContentsByPhandle(entry_phandle, self,
                                                            required)
tools/binman/etype/section.py +2 -2
index 92d3f3add4c..e2949fc9163 100644
@@ -166,7 +166,7 @@ class Entry_section(Entry):
         pad_byte = (entry._pad_byte if isinstance(entry, Entry_section)
                     else self._pad_byte)
 
-        data = b''
+        data = bytearray()
         # Handle padding before the entry
         if entry.pad_before:
             data += tools.GetBytes(self._pad_byte, entry.pad_before)
@@ -200,7 +200,7 @@ class Entry_section(Entry):
         Returns:
             Contents of the section (bytes)
         """
-        section_data = b''
+        section_data = bytearray()
 
         for entry in self._entries.values():
             entry_data = entry.GetData(required)
Signature · 1 line
2.32.0.93.g670b81a890-goog
#8 lore raw

Re: [PATCH 4/6] binman: Use bytearray instead of string

This is faster if data is being concatenated. Update the section and collection etypes.

---

 tools/binman/etype/collection.py | 2 +-
 tools/binman/etype/section.py    | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

Applied to u-boot-dm, thanks!

#9 lore raw

PATCH 5/6 patman: Use bytearray instead of string

In reply to Simon Glass

If the process outputs a lot of data on stdout this can be quite slow, since the bytestring is regenerated each time. Use a bytearray instead.

---

 tools/patman/cros_subprocess.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
tools/patman/cros_subprocess.py +3 -3
index efd0a5aaf72..fdd51386856 100644
@@ -169,11 +169,11 @@ class Popen(subprocess.Popen):
                 self.stdin.close()
         if self.stdout:
             read_set.append(self.stdout)
-            stdout = b''
+            stdout = bytearray()
         if self.stderr and self.stderr != self.stdout:
             read_set.append(self.stderr)
-            stderr = b''
-        combined = b''
+            stderr = bytearray()
+        combined = bytearray()
 
         input_offset = 0
         while read_set or write_set:
Signature · 1 line
2.32.0.93.g670b81a890-goog
#10 lore raw

Re: [PATCH 5/6] patman: Use bytearray instead of string

If the process outputs a lot of data on stdout this can be quite slow, since the bytestring is regenerated each time. Use a bytearray instead.

---

 tools/patman/cros_subprocess.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Applied to u-boot-dm, thanks!

#11 lore raw

PATCH 6/6 binman: Add basic support for debugging performance

In reply to Simon Glass

One of binman's attributes is that it is extremely fast, at least for a Python program. Add some simple timing around operations that might take a while, such as reading an image and compressing it. This should help to maintain the performance as new features are added.

This is for debugging purposes only.

---

 tools/binman/control.py    |  3 ++
 tools/binman/etype/blob.py |  5 +++
 tools/binman/ftest.py      |  8 +++++
 tools/binman/state.py      | 72 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 88 insertions(+)
tools/binman/control.py +3 -0
index b2113b6e64f..dcba02ff7f8 100644
@@ -646,6 +646,9 @@ def Binman(args):
 
             if missing:
                 tout.Warning("\nSome images are invalid")
+
+            # Use this to debug the time take to pack the image
+            #state.TimingShow()
         finally:
             tools.FinaliseOutputDir()
     finally:
tools/binman/etype/blob.py +5 -0
index 018f8c9a319..fae86ca3ec0 100644
@@ -6,6 +6,7 @@
 #
 
 from binman.entry import Entry
+from binman import state
 from dtoc import fdt_util
 from patman import tools
 from patman import tout
@@ -59,8 +60,12 @@ class Entry_blob(Entry):
         the data in chunks and avoid reading it all at once. For now
         this seems like an unnecessary complication.
         """
+        state.TimingStart('read')
         indata = tools.ReadFile(self._pathname)
+        state.TimingAccum('read')
+        state.TimingStart('compress')
         data = self.CompressData(indata)
+        state.TimingAccum('compress')
         self.SetContents(data)
         return True
 
tools/binman/ftest.py +8 -0
index ceaa6a8737c..d93361778ec 100644
@@ -4634,6 +4634,14 @@ class TestFunctional(unittest.TestCase):
         self.assertIn("Node '/binman/section@0': Timed out obtaining contents",
                       str(e.exception))
 
+    def testTiming(self):
+        """Test output of timing information"""
+        data = self._DoReadFile('055_sections.dts')
+        with test_util.capture_sys_output() as (stdout, stderr):
+            state.TimingShow()
+        self.assertIn('read:', stdout.getvalue())
+        self.assertIn('compress:', stdout.getvalue())
+
 
 if __name__ == "__main__":
     unittest.main()
tools/binman/state.py +72 -0

89 lines not on this page · raw message

Signature · 1 line
2.32.0.93.g670b81a890-goog
#12 lore raw

Re: [PATCH 6/6] binman: Add basic support for debugging performance

One of binman's attributes is that it is extremely fast, at least for a Python program. Add some simple timing around operations that might take a while, such as reading an image and compressing it. This should help to maintain the performance as new features are added.

This is for debugging purposes only.

---

 tools/binman/control.py    |  3 ++
 tools/binman/etype/blob.py |  5 +++
 tools/binman/ftest.py      |  8 +++++
 tools/binman/state.py      | 72 ++++++++++++++++++++++++++++++++++++++
 4 files changed, 88 insertions(+)

Applied to u-boot-dm, thanks!