binman: Support compressing files in parallel
cover binman: Support compressing files in parallel
When building an image with large files it makes sense to compress them in parallel. This series adds support for this as well a few other performance-related tweaks. It also includes support for timing of the different types of operation (e.g. compression), for debugging purposes.
Simon Glass (6):
binman: Put compressed data into separate files
binman: Support multithreading for building images
binman: Split node-reading out from constructor in files
binman: Use bytearray instead of string
patman: Use bytearray instead of string
binman: Add basic support for debugging performance
tools/binman/binman.rst | 18 ++++++ tools/binman/cmdline.py | 4 ++ tools/binman/control.py | 7 +++ tools/binman/etype/blob.py | 5 ++ tools/binman/etype/collection.py | 2 +- tools/binman/etype/files.py | 3 + tools/binman/etype/section.py | 40 ++++++++++++-- tools/binman/ftest.py | 41 ++++++++++++-- tools/binman/image.py | 3 + tools/binman/state.py | 95 ++++++++++++++++++++++++++++++++ tools/patman/cros_subprocess.py | 6 +- tools/patman/tools.py | 9 ++- 12 files changed, 218 insertions(+), 15 deletions(-)
Signature · 1 line
2.32.0.93.g670b81a890-goog
PATCH 1/6 binman: Put compressed data into separate files
At present compression uses the same temporary file for all invocations. With multithreading this causes the data to become corrupted. Use a different filename each time.
---
tools/patman/tools.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
tools/patman/tools.py +7 -2
@@ -466,6 +466,9 @@ def Compress(indata, algo, with_header=True): This requires 'lz4' and 'lzma_alone' tools. It also requires an output directory to be previously set up, by calling PrepareOutputDir(). + Care is taken to use unique temporary files so that this function can be + called from multiple threads. + Args: indata: Input data to compress algo: Algorithm to use ('none', 'gzip', 'lz4' or 'lzma')
@@ -475,14 +478,16 @@ def Compress(indata, algo, with_header=True): """ if algo == 'none': return indata - fname = GetOutputFilename('%s.comp.tmp' % algo) + fname = tempfile.NamedTemporaryFile(prefix='%s.comp.tmp' % algo, + dir=outdir).name WriteFile(fname, indata) if algo == 'lz4': data = Run('lz4', '--no-frame-crc', '-B4', '-5', '-c', fname, binary=True) # cbfstool uses a very old version of lzma elif algo == 'lzma': - outfname = GetOutputFilename('%s.comp.otmp' % algo) + outfname = tempfile.NamedTemporaryFile(prefix='%s.comp.otmp' % algo, + dir=outdir).name Run('lzma_alone', 'e', fname, outfname, '-lc1', '-lp0', '-pb0', '-d8') data = ReadFile(outfname) elif algo == 'gzip':
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 1/6] binman: Put compressed data into separate files
At present compression uses the same temporary file for all invocations. With multithreading this causes the data to become corrupted. Use a different filename each time.
---
tools/patman/tools.py | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 2/6 binman: Support multithreading for building images
In reply to Simon GlassSome images may take a while to build, e.g. if they are large and use slow compression. Support compiling sections in parallel to speed things up.
---
tools/binman/binman.rst | 18 ++++++++++++++++++ tools/binman/cmdline.py | 4 ++++ tools/binman/control.py | 4 ++++ tools/binman/etype/section.py | 36 ++++++++++++++++++++++++++++++++--- tools/binman/ftest.py | 33 ++++++++++++++++++++++++++++---- tools/binman/image.py | 3 +++ tools/binman/state.py | 23 ++++++++++++++++++++++ 7 files changed, 114 insertions(+), 7 deletions(-)
tools/binman/binman.rst +18 -0
@@ -1142,6 +1142,22 @@ adds a -v<level> option to the call to binman:: make BINMAN_VERBOSE=5 +Building sections in parallel +----------------------------- + +By default binman uses multiprocessing to speed up compilation of large images. +This works at a section level, with one thread for each entry in the section. +This can speed things up if the entries are large and use compression. + +This feature can be disabled with the '-T' flag, which defaults to a suitable +value for your machine. This depends on the Python version, e.g on v3.8 it uses +12 threads on an 8-core machine. See ConcurrentFutures_ for more details. + +The special value -T0 selects single-threaded mode, useful for debugging during +development, since dealing with exceptions and problems in threads is more +difficult. This avoids any use of ThreadPoolExecutor. + + History / Credits -----------------
@@ -1190,3 +1206,5 @@ Some ideas: -- Simon Glass <[email protected]> 7/7/2016 + +.. _ConcurrentFutures: https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor
tools/binman/cmdline.py +4 -0
@@ -32,6 +32,10 @@ controlled by a description in the board device tree.''' default=False, help='Display the README file') parser.add_argument('--toolpath', type=str, action='append', help='Add a path to the directories containing tools') + parser.add_argument('-T', '--threads', type=int, + default=None, help='Number of threads to use (0=single-thread)') + parser.add_argument('--test-section-timeout', action='store_true', + help='Use a zero timeout for section multi-threading (for testing)') parser.add_argument('-v', '--verbosity', default=1, type=int, help='Control verbosity: 0=silent, 1=warnings, 2=notices, ' '3=info, 4=detail, 5=debug')
tools/binman/control.py +4 -0
@@ -628,9 +628,13 @@ def Binman(args): tools.PrepareOutputDir(args.outdir, args.preserve) tools.SetToolPaths(args.toolpath) state.SetEntryArgs(args.entry_arg) + state.SetThreads(args.threads) images = PrepareImagesAndDtbs(dtb_fname, args.image, args.update_fdt, use_expanded) + if args.test_section_timeout: + # Set the first image to timeout, used in testThreadTimeout() + images[list(images.keys())[0]].test_section_timeout = True missing = False for image in images.values(): missing |= ProcessImage(image, args.update_fdt, args.map,
tools/binman/etype/section.py +33 -3
@@ -9,10 +9,12 @@ images to be created. """ from collections import OrderedDict +import concurrent.futures import re import sys from binman.entry import Entry +from binman import state from dtoc import fdt_util from patman import tools from patman import tout
@@ -525,15 +527,43 @@ class Entry_section(Entry): def GetEntryContents(self): """Call ObtainContents() for each entry in the section """ + def _CheckDone(entry): + if not entry.ObtainContents(): + next_todo.append(entry) + return entry + todo = self._entries.values() for passnum in range(3): + threads = state.GetThreads() next_todo = [] - for entry in todo: - if not entry.ObtainContents(): - next_todo.append(entry) + + if threads == 0: + for entry in todo: + _CheckDone(entry) + else: + with concurrent.futures.ThreadPoolExecutor( + max_workers=threads) as executor: + future_to_data = { + entry: executor.submit(_CheckDone, entry) + for entry in todo} + timeout = 60 + if self.GetImage().test_section_timeout: + timeout = 0 + done, not_done = concurrent.futures.wait( + future_to_data.values(), timeout=timeout) + # Make sure we check the result, so any exceptions are + # generated. Check the results in entry order, since tests + # may expect earlier entries to fail first. + for entry in todo: + job = future_to_data[entry] + job.result() + if not_done: + self.Raise('Timed out obtaining contents') + todo = next_todo if not todo: break + if todo: self.Raise('Internal error: Could not complete processing of contents: remaining %s' % todo)
tools/binman/ftest.py +29 -4
@@ -315,7 +315,8 @@ class TestFunctional(unittest.TestCase): def _DoTestFile(self, fname, debug=False, map=False, update_dtb=False, entry_args=None, images=None, use_real_dtb=False, use_expanded=False, verbosity=None, allow_missing=False, - extra_indirs=None): + extra_indirs=None, threads=None, + test_section_timeout=False): """Run binman with a given test file Args:
@@ -338,6 +339,8 @@ class TestFunctional(unittest.TestCase): allow_missing: Set the '--allow-missing' flag so that missing external binaries just produce a warning instead of an error extra_indirs: Extra input directories to add using -I + threads: Number of threads to use (None for default, 0 for + single-threaded) """ args = [] if debug:
@@ -349,6 +352,10 @@ class TestFunctional(unittest.TestCase): if self.toolpath: for path in self.toolpath: args += ['--toolpath', path] + if threads is not None: + args.append('-T%d' % threads) + if test_section_timeout: + args.append('--test-section-timeout') args += ['build', '-p', '-I', self._indir, '-d', self.TestFile(fname)] if map: args.append('-m')
@@ -381,7 +388,7 @@ class TestFunctional(unittest.TestCase): fname: Filename of .dts file to read outfile: Output filename for compiled device-tree binary - Returns: + #Returns: Contents of device-tree binary """ tmpdir = tempfile.mkdtemp(prefix='binmant.')
@@ -419,7 +426,7 @@ class TestFunctional(unittest.TestCase): def _DoReadFileDtb(self, fname, use_real_dtb=False, use_expanded=False, map=False, update_dtb=False, entry_args=None, - reset_dtbs=True, extra_indirs=None): + reset_dtbs=True, extra_indirs=None, threads=None): """Run binman and return the resulting image This runs binman with a given test file and then reads the resulting
@@ -446,6 +453,8 @@ class TestFunctional(unittest.TestCase): function. If reset_dtbs is True, then the original test dtb is written back before this function finishes extra_indirs: Extra input directories to add using -I + threads: Number of threads to use (None for default, 0 for + single-threaded) Returns: Tuple:
@@ -470,7 +479,8 @@ class TestFunctional(unittest.TestCase): try: retcode = self._DoTestFile(fname, map=map, update_dtb=update_dtb, entry_args=entry_args, use_real_dtb=use_real_dtb, - use_expanded=use_expanded, extra_indirs=extra_indirs) + use_expanded=use_expanded, extra_indirs=extra_indirs, + threads=threads) self.assertEqual(0, retcode) out_dtb_fname = tools.GetOutputFilename('u-boot.dtb.out')
@@ -4609,6 +4619,21 @@ class TestFunctional(unittest.TestCase): self.assertIn('Expected __bss_size symbol in vpl/u-boot-vpl', str(e.exception)) + def testSectionsSingleThread(self): + """Test sections without multithreading""" + data = self._DoReadFileDtb('055_sections.dts', threads=0)[0] + expected = (U_BOOT_DATA + tools.GetBytes(ord('!'), 12) + + U_BOOT_DATA + tools.GetBytes(ord('a'), 12) + + U_BOOT_DATA + tools.GetBytes(ord('&'), 4)) + self.assertEqual(expected, data) + + def testThreadTimeout(self): + """Test handling a thread that takes too long""" + with self.assertRaises(ValueError) as e: + self._DoTestFile('055_sections.dts', test_section_timeout=True) + self.assertIn("Node '/binman/section@0': Timed out obtaining contents", + str(e.exception)) + if __name__ == "__main__": unittest.main()
tools/binman/image.py +3 -0
@@ -36,6 +36,8 @@ class Image(section.Entry_section): fdtmap_data: Contents of the fdtmap when loading from a file allow_repack: True to add properties to allow the image to be safely repacked later + test_section_timeout: Use a zero timeout for section multi-threading + (for testing) Args: copy_to_orig: Copy offset/size to orig_offset/orig_size after reading
@@ -74,6 +76,7 @@ class Image(section.Entry_section): self.allow_repack = False self._ignore_missing = ignore_missing self.use_expanded = use_expanded + self.test_section_timeout = False if not test: self.ReadNode()
tools/binman/state.py +23 -0
@@ -7,6 +7,7 @@ import hashlib import re +import threading from dtoc import fdt import os
@@ -56,6 +57,9 @@ allow_entry_expansion = True # to the new ones, the compressed size increases, etc. allow_entry_contraction = False +# Number of threads to use for binman (None means machine-dependent) +num_threads = None + def GetFdtForEtype(etype): """Get the Fdt object for a particular device-tree entry
@@ -421,3 +425,22 @@ def AllowEntryContraction(): raised """ return allow_entry_contraction + +def SetThreads(threads): + """Set the number of threads to use when building sections + + Args: + threads: Number of threads to use (None for default, 0 for + single-threaded) + """ + global num_threads + + num_threads = threads + +def GetThreads(): + """Get the number of threads to use when building sections + + Returns: + Number of threads to use (None for default, 0 for single-threaded) + """ + return num_threads
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 2/6] binman: Support multithreading for building images
Some images may take a while to build, e.g. if they are large and use slow compression. Support compiling sections in parallel to speed things up.
---
tools/binman/binman.rst | 18 ++++++++++++++++++ tools/binman/cmdline.py | 4 ++++ tools/binman/control.py | 4 ++++ tools/binman/etype/section.py | 36 ++++++++++++++++++++++++++++++++--- tools/binman/ftest.py | 33 ++++++++++++++++++++++++++++---- tools/binman/image.py | 3 +++ tools/binman/state.py | 23 ++++++++++++++++++++++ 7 files changed, 114 insertions(+), 7 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 3/6 binman: Split node-reading out from constructor in files
In reply to Simon GlassThe constructor should not read the node information. Move it to the ReadNode() method instead. This allows this etype to be subclassed.
---
tools/binman/etype/files.py | 3 +++ 1 file changed, 3 insertions(+)
tools/binman/etype/files.py +3 -0
@@ -34,6 +34,9 @@ class Entry_files(Entry_section): from binman import state super().__init__(section, etype, node) + + def ReadNode(self): + super().ReadNode() self._pattern = fdt_util.GetString(self._node, 'pattern') if not self._pattern: self.Raise("Missing 'pattern' property")
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 3/6] binman: Split node-reading out from constructor in files
The constructor should not read the node information. Move it to the ReadNode() method instead. This allows this etype to be subclassed.
---
tools/binman/etype/files.py | 3 +++ 1 file changed, 3 insertions(+)
Applied to u-boot-dm, thanks!
PATCH 4/6 binman: Use bytearray instead of string
In reply to Simon GlassThis is faster if data is being concatenated. Update the section and collection etypes.
---
tools/binman/etype/collection.py | 2 +- tools/binman/etype/section.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
tools/binman/etype/collection.py +1 -1
@@ -40,7 +40,7 @@ class Entry_collection(Entry): """ # Join up all the data self.Info('Getting contents, required=%s' % required) - data = b'' + data = bytearray() for entry_phandle in self.content: entry_data = self.section.GetContentsByPhandle(entry_phandle, self, required)
tools/binman/etype/section.py +2 -2
@@ -166,7 +166,7 @@ class Entry_section(Entry): pad_byte = (entry._pad_byte if isinstance(entry, Entry_section) else self._pad_byte) - data = b'' + data = bytearray() # Handle padding before the entry if entry.pad_before: data += tools.GetBytes(self._pad_byte, entry.pad_before)
@@ -200,7 +200,7 @@ class Entry_section(Entry): Returns: Contents of the section (bytes) """ - section_data = b'' + section_data = bytearray() for entry in self._entries.values(): entry_data = entry.GetData(required)
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 4/6] binman: Use bytearray instead of string
This is faster if data is being concatenated. Update the section and collection etypes.
---
tools/binman/etype/collection.py | 2 +- tools/binman/etype/section.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 5/6 patman: Use bytearray instead of string
In reply to Simon GlassIf the process outputs a lot of data on stdout this can be quite slow, since the bytestring is regenerated each time. Use a bytearray instead.
---
tools/patman/cros_subprocess.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
tools/patman/cros_subprocess.py +3 -3
@@ -169,11 +169,11 @@ class Popen(subprocess.Popen): self.stdin.close() if self.stdout: read_set.append(self.stdout) - stdout = b'' + stdout = bytearray() if self.stderr and self.stderr != self.stdout: read_set.append(self.stderr) - stderr = b'' - combined = b'' + stderr = bytearray() + combined = bytearray() input_offset = 0 while read_set or write_set:
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 5/6] patman: Use bytearray instead of string
If the process outputs a lot of data on stdout this can be quite slow, since the bytestring is regenerated each time. Use a bytearray instead.
---
tools/patman/cros_subprocess.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Applied to u-boot-dm, thanks!
PATCH 6/6 binman: Add basic support for debugging performance
In reply to Simon GlassOne of binman's attributes is that it is extremely fast, at least for a Python program. Add some simple timing around operations that might take a while, such as reading an image and compressing it. This should help to maintain the performance as new features are added.
This is for debugging purposes only.
---
tools/binman/control.py | 3 ++ tools/binman/etype/blob.py | 5 +++ tools/binman/ftest.py | 8 +++++ tools/binman/state.py | 72 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 88 insertions(+)
tools/binman/control.py +3 -0
@@ -646,6 +646,9 @@ def Binman(args): if missing: tout.Warning("\nSome images are invalid") + + # Use this to debug the time take to pack the image + #state.TimingShow() finally: tools.FinaliseOutputDir() finally:
tools/binman/etype/blob.py +5 -0
@@ -6,6 +6,7 @@ # from binman.entry import Entry +from binman import state from dtoc import fdt_util from patman import tools from patman import tout
@@ -59,8 +60,12 @@ class Entry_blob(Entry): the data in chunks and avoid reading it all at once. For now this seems like an unnecessary complication. """ + state.TimingStart('read') indata = tools.ReadFile(self._pathname) + state.TimingAccum('read') + state.TimingStart('compress') data = self.CompressData(indata) + state.TimingAccum('compress') self.SetContents(data) return True
tools/binman/ftest.py +8 -0
@@ -4634,6 +4634,14 @@ class TestFunctional(unittest.TestCase): self.assertIn("Node '/binman/section@0': Timed out obtaining contents", str(e.exception)) + def testTiming(self): + """Test output of timing information""" + data = self._DoReadFile('055_sections.dts') + with test_util.capture_sys_output() as (stdout, stderr): + state.TimingShow() + self.assertIn('read:', stdout.getvalue()) + self.assertIn('compress:', stdout.getvalue()) + if __name__ == "__main__": unittest.main()
tools/binman/state.py +72 -0
89 lines not on this page · raw message
Signature · 1 line
2.32.0.93.g670b81a890-goog
Re: [PATCH 6/6] binman: Add basic support for debugging performance
One of binman's attributes is that it is extremely fast, at least for a Python program. Add some simple timing around operations that might take a while, such as reading an image and compressing it. This should help to maintain the performance as new features are added.
This is for debugging purposes only.
---
tools/binman/control.py | 3 ++ tools/binman/etype/blob.py | 5 +++ tools/binman/ftest.py | 8 +++++ tools/binman/state.py | 72 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 88 insertions(+)
Applied to u-boot-dm, thanks!