Add some missing buildman features and deprecate MAKEALL
v5 cover Add some missing buildman features and deprecate MAKEALL
Buildman has been around for a little over a year and is used by a fair number of U-Boot developers. However quite a few people still use MAKEALL.
Buildman was intended to replace MAKEALL, so perhaps now is a good time to start that process.
The reasons to deprecate MAKEALL are:
- We don't want to maintain two build systems
- Buildman is typically faster
- Buildman has a lot more features
This series adds a few features to buildman to fill some gaps, adds some information into the README on how to migrate from MAKEALL, and adds a deprecation message to MAKEALL.
Changes in v5:
- Update test for output options
- Update test for verbose option
- Add new patch to add an 'and' operator for board selection
- Drop patch to search for *cc instead of *gcc for the compiler
Changes in v4:
- Fix missing import in builder.py
- Fix typo in commit message
- Add new patch to remove unused non-incremental build method code
- Add new patch to allow the config file to be specified
- Add new patch to show a message when there are no errors
- Add new path to add a few more toolchain examples to the README
Changes in v3:
- Add new patch to build current source tree
- Add new patch to move BuilderThread code to its own file
- Add new patch to sort command line options
- Add new patch to refactor output options
- Add new patch to add a verbose option
Changes in v2:
- Add new patch to fix existing typos
- Minor changes to the text
Simon Glass (13):
buildman: Fix a few typos
buildman: Add some notes about moving from MAKEALL
buildman: Allow building of current source tree
buildman: Move BuilderThread code to its own file
buildman: Sort command line options
buildman: Refactor output options
buildman: Add verbose option to display errors as they happen
buildman: Remove unused non-incremental build method code
buildman: Add an option to specify the buildman config file
buildman: Add a message indicating there are no errors
buildman: Add a few more toolchain examples to the README
buildman: Introduce an 'and' operator for board selection
RFC: Deprecate MAKEALL
MAKEALL | 4 + tools/buildman/README | 159 ++++++++++-- tools/buildman/board.py | 144 +++++++++-- tools/buildman/builder.py | 544 ++++++---------------------------------- tools/buildman/builderthread.py | 434 ++++++++++++++++++++++++++++++++ tools/buildman/buildman.py | 16 +- tools/buildman/control.py | 101 +++++--- tools/buildman/test.py | 56 ++++- 8 files changed, 905 insertions(+), 553 deletions(-) create mode 100644 tools/buildman/builderthread.py
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 1/13 buildman: Fix a few typos
There are several typos in the README - fix them.
---
Changes in v5: None
Changes in v4: None
Changes in v3: None
Changes in v2:
- Add new patch to fix existing typos
tools/buildman/README | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
tools/buildman/README +6 -6
@@ -71,8 +71,8 @@ directory. It clones this repository into a copy for each thread, and the threads do not affect the state of your git repository. Any checkouts done by the thread affect only the working directory for that thread. -Buildman automatically selects the correct toolchain for each board. You -must supply suitable toolchains, but buildman takes care of selecting the +Buildman automatically selects the correct tool chain for each board. You +must supply suitable tool chains, but buildman takes care of selecting the right one. Buildman always builds a branch, and always builds the upstream commit as
@@ -287,7 +287,7 @@ If it can't detect the upstream branch, try checking out the branch, and doing something like 'git branch --set-upstream <branch> upstream/master' or something similar. -As an exmmple: +As an example: Dry run, so not doing much. But I would do this:
@@ -339,7 +339,7 @@ Building 18 commits for 1059 boards (4 threads, 1 job per thread) 528 36 124 /19062 1:13:30 : SIMPC8313_SP This means that it is building 19062 board/commit combinations. So far it -has managed to succesfully build 528. Another 36 have built with warnings, +has managed to successfully build 528. Another 36 have built with warnings, and 124 more didn't build at all. Buildman expects to complete the process in an hour and 15 minutes. Use this time to buy a faster computer.
@@ -413,7 +413,7 @@ again. At commit 16, the error moves - you can see that the old error at line 120 is fixed, but there is a new one at line 126. This is probably only because -we added some code and moved the broken line futher down the file. +we added some code and moved the broken line father down the file. If many boards have the same error, then -e will display the error only once. This makes the output as concise as possible.
@@ -491,7 +491,7 @@ You can also use -d to see a detailed size breakdown for each board. This list is sorted in order from largest growth to largest reduction. It is possible to go a little further with the -B option (--bloat). This -shows where U-Boot has bloted, breaking the size change down to the function +shows where U-Boot has bloated, breaking the size change down to the function level. Example output is below: $ ./tools/buildman/buildman -b us-mem4 -sSdB
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 2/13 buildman: Add some notes about moving from MAKEALL
In reply to Simon GlassFor those used to MAKEALL, buildman seems strange. Add some notes to ease the transition.
---
Changes in v5: None
Changes in v4: None
Changes in v3: None
Changes in v2:
- Minor changes to the text
tools/buildman/README | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+)
tools/buildman/README +92 -0
@@ -3,6 +3,8 @@ # SPDX-License-Identifier: GPL-2.0+ # +(Please read 'How to change from MAKEALL' if you are used to that tool) + What is this? =============
@@ -663,6 +665,96 @@ Other options Buildman has various other command line options. Try --help to see them. +How to change from MAKEALL +========================== + +Buildman includes most of the features of MAKEALL and is generally faster +and easier to use. In particular it builds entire branches: if a particular +commit introduces an error in a particular board, buildman can easily show +you this, even if a later commit fixes that error. + +The reasons to deprecate MAKEALL are: +- We don't want to maintain two build systems +- Buildman is typically faster +- Buildman has a lot more features + +But still, many people will be sad to lose MAKEALL. If you are used to +MAKEALL, here are a few pointers. + +First you need to set up your tool chains - see the 'Setting up' section +for details. Once you have your required toolchain(s) detected then you are +ready to go. + +Buildman works on entire branches, so the normal use is: + + ./tools/buildman/buildman -b <branch_name> <list of things to build> + +followed by (afterwards, or perhaps concurrently in another terminal): + + ./tools/buildman/buildman -b <branch_name> -s <list of things to build> + +to see the results of the build. Rather than showing you all the output, +buildman just shows a summary, with red indicating that a commit introduced +an error and green indicating that a commit fixed an error. Use the -e +flag to see the full errors. + +You don't need to stick around on that branch while buildman is running. It +checks out its own copy of the source code, so you can change branches, +add commits, etc. without affecting the build in progress. + +The <list of things to build> can include board names, architectures or the +like. There are no flags to disambiguate since ambiguities are rare. Using +the examples from MAKEALL: + +Examples: + - build all Power Architecture boards: + MAKEALL -a powerpc + MAKEALL --arch powerpc + MAKEALL powerpc + ** buildman -b <branch> powerpc + - build all PowerPC boards manufactured by vendor "esd": + MAKEALL -a powerpc -v esd + ** buildman -b <branch> esd + - build all PowerPC boards manufactured either by "keymile" or "siemens": + MAKEALL -a powerpc -v keymile -v siemens + ** buildman -b <branch> keymile siemens + - build all Freescale boards with MPC83xx CPUs, plus all 4xx boards: + MAKEALL -c mpc83xx -v freescale 4xx + ** buildman -b <branch> mpc83xx freescale 4xx + +Buildman automatically tries to use all the CPUs in your machine. If you +are building a lot of boards it will use one thread for every CPU core +it detects in your machine. This is like MAKEALL's BUILD_NBUILDS option. +You can use the -T flag to change the number of threads. If you are only +building a few boards, buildman will automatically run make with the -j +flag to increase the number of concurrent make tasks. It isn't normally +that helpful to fiddle with this option, but if you use the BUILD_NCPUS +option in MAKEALL then -j is the equivalent in buildman. + +Buildman puts its output in ../<branch_name> by default but you can change +this with the -o option. Buildman normally does out-of-tree builds: use -i +to disable that if you really want to. But be careful that once you have +used -i you pollute buildman's copies of the source tree, and you will need +to remove the build directory (normally ../<branch_name>) to run buildman +in normal mode (without -i). + +Buildman doesn't keep the output result normally, but use the -k option to +do this. + +Please read 'Theory of Operation' a few times as it will make a lot of +things clearer. + +Some options you might like are: + + -B shows which functions are growing/shrinking in which commit - great + for finding code bloat. + -S shows image sizes for each commit (just an overall summary) + -u shows boards that you haven't built yet + --step 0 will build just the upstream commit and the last commit of your + branch. This is often a quick sanity check that your branch doesn't + break anything. But note this does not check bisectability! + + TODO ====
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 3/13 buildman: Allow building of current source tree
In reply to Simon GlassOriginally buildman had some support for building the current source tree. However this was dropped before it was submitted, as part of the effort to make it faster when building entire branches.
Reinstate this support. If no -b option is given, buildman will build the current source tree.
---
Changes in v5: None
Changes in v4: None
Changes in v3:
- Add new patch to build current source tree
Changes in v2: None
tools/buildman/builder.py | 71 +++++++++++++++++++++++++----------- tools/buildman/control.py | 91 ++++++++++++++++++++++++++++++----------------- 2 files changed, 108 insertions(+), 54 deletions(-)
tools/buildman/builder.py +50 -21
@@ -248,21 +248,34 @@ class BuilderThread(threading.Thread): if self.toolchain: # Checkout the right commit - if commit_upto is not None: + if self.builder.commits: commit = self.builder.commits[commit_upto] if self.builder.checkout: git_dir = os.path.join(work_dir, '.git') gitutil.Checkout(commit.hash, git_dir, work_dir, force=True) else: - commit = self.builder.commit # Ick, fix this for BuildCommits() + commit = 'current' # Set up the environment and command line env = self.toolchain.MakeEnvironment() Mkdir(out_dir) args = [] + cwd = work_dir if not self.builder.in_tree: - args.append('O=build') + if commit_upto is None: + # In this case we are building in the original source + # directory (i.e. the current directory where buildman + # is invoked. The output directory is set to this + # thread's selected work directory. + # + # Symlinks can confuse U-Boot's Makefile since + # we may use '..' in our path, so remove them. + work_dir = os.path.realpath(work_dir) + args.append('O=%s/build' % work_dir) + cwd = None + else: + args.append('O=build') args.append('-s') if self.builder.num_jobs is not None: args.extend(['-j', str(self.builder.num_jobs)])
@@ -272,14 +285,14 @@ class BuilderThread(threading.Thread): # If we need to reconfigure, do that now if do_config: - result = self.Make(commit, brd, 'distclean', work_dir, + result = self.Make(commit, brd, 'distclean', cwd, 'distclean', *args, env=env) - result = self.Make(commit, brd, 'config', work_dir, + result = self.Make(commit, brd, 'config', cwd, *(args + config_args), env=env) config_out = result.combined do_config = False # No need to configure next time if result.return_code == 0: - result = self.Make(commit, brd, 'build', work_dir, *args, + result = self.Make(commit, brd, 'build', cwd, *args, env=env) result.stdout = config_out + result.stdout else:
@@ -478,8 +491,10 @@ class BuilderThread(threading.Thread): self.builder.out_queue.put(result) else: # Just build the currently checked-out build - result = self.RunCommit(None, True) - result.commit_upto = self.builder.upto + result, request_config = self.RunCommit(None, brd, work_dir, True, + True, self.builder.force_build_failures) + result.commit_upto = 0 + self._WriteResult(result, job.keep_outputs) self.builder.out_queue.put(result) def run(self):
@@ -491,12 +506,16 @@ class BuilderThread(threading.Thread): alive = True while True: job = self.builder.queue.get() + if self.builder.active and alive: + self.RunJob(job) + ''' try: if self.builder.active and alive: self.RunJob(job) except Exception as err: alive = False print err + ''' self.builder.queue.task_done()
@@ -763,10 +782,13 @@ class Builder: Args: commit_upto: Commit number to use (0..self.count-1) """ - commit = self.commits[commit_upto] - subject = commit.subject.translate(trans_valid_chars) - commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1, - self.commit_count, commit.hash, subject[:20])) + if self.commits: + commit = self.commits[commit_upto] + subject = commit.subject.translate(trans_valid_chars) + commit_dir = ('%02d_of_%02d_g%s_%s' % (commit_upto + 1, + self.commit_count, commit.hash, subject[:20])) + else: + commit_dir = 'current' output_dir = os.path.join(self.base_dir, commit_dir) return output_dir
@@ -1308,14 +1330,18 @@ class Builder: show_detail: Show detail for each board show_bloat: Show detail for each function """ - self.commit_count = len(commits) + self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected) for commit_upto in range(0, self.commit_count, self._step): board_dict, err_lines = self.GetResultSummary(board_selected, commit_upto, read_func_sizes=show_bloat) - msg = '%02d: %s' % (commit_upto + 1, commits[commit_upto].subject) + if commits: + msg = '%02d: %s' % (commit_upto + 1, + commits[commit_upto].subject) + else: + msg = 'current' print self.col.Color(self.col.BLUE, msg) self.PrintResultSummary(board_selected, board_dict, err_lines if show_errors else [], show_sizes, show_detail,
@@ -1330,7 +1356,7 @@ class Builder: commits: Selected commits to build """ # First work out how many commits we will build - count = (len(commits) + self._step - 1) / self._step + count = (self.commit_count + self._step - 1) / self._step self.count = len(board_selected) * count self.upto = self.warned = self.fail = 0 self._timestamps = collections.deque()
@@ -1377,13 +1403,14 @@ class Builder: """ return os.path.join(self._working_dir, '%02d' % thread_num) - def _PrepareThread(self, thread_num): + def _PrepareThread(self, thread_num, setup_git): """Prepare the working directory for a thread. This clones or fetches the repo into the thread's work directory. Args: thread_num: Thread number (0, 1, ...) + setup_git: True to set up a git repo clone """ thread_dir = self.GetThreadDir(thread_num) Mkdir(thread_dir)
@@ -1392,7 +1419,7 @@ class Builder: # Clone the repo if it doesn't already exist # TODO(sjg at chromium): Perhaps some git hackery to symlink instead, so # we have a private index but uses the origin repo's contents? - if self.git_dir: + if setup_git and self.git_dir: src_dir = os.path.abspath(self.git_dir) if os.path.exists(git_dir): gitutil.Fetch(git_dir, thread_dir)
@@ -1400,17 +1427,18 @@ class Builder: print 'Cloning repo for thread %d' % thread_num gitutil.Clone(src_dir, thread_dir) - def _PrepareWorkingSpace(self, max_threads): + def _PrepareWorkingSpace(self, max_threads, setup_git): """Prepare the working directory for use. Set up the git repo for each thread. Args: max_threads: Maximum number of threads we expect to need. + setup_git: True to set up a git repo clone """ Mkdir(self._working_dir) for thread in range(max_threads): - self._PrepareThread(thread) + self._PrepareThread(thread, setup_git) def _PrepareOutputSpace(self): """Get the output directories ready to receive files.
@@ -1437,12 +1465,13 @@ class Builder: show_errors: True to show summarised error/warning info keep_outputs: True to save build output files """ - self.commit_count = len(commits) + self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected) Mkdir(self.base_dir) - self._PrepareWorkingSpace(min(self.num_threads, len(board_selected))) + self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)), + commits is not None) self._PrepareOutputSpace() self.SetupBuild(board_selected, commits) self.ProcessResult(None)
tools/buildman/control.py +58 -33
140 lines not on this page · raw message
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 4/13 buildman: Move BuilderThread code to its own file
In reply to Simon GlassThe builder.py file is getting too long, so split out some code.
---
Changes in v5: None
Changes in v4:
- Fix missing import in builder.py
Changes in v3:
- Add new patch to move BuilderThread code to its own file
Changes in v2: None
tools/buildman/builder.py | 439 +--------------------------------------- tools/buildman/builderthread.py | 434 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 442 insertions(+), 431 deletions(-) create mode 100644 tools/buildman/builderthread.py
tools/buildman/builder.py +8 -431
@@ -6,7 +6,6 @@ # import collections -import errno from datetime import datetime, timedelta import glob import os
@@ -15,9 +14,9 @@ import Queue import shutil import string import sys -import threading import time +import builderthread import command import gitutil import terminal
@@ -97,428 +96,6 @@ OUTCOME_OK, OUTCOME_WARNING, OUTCOME_ERROR, OUTCOME_UNKNOWN = range(4) trans_valid_chars = string.maketrans("/: ", "---") -def Mkdir(dirname): - """Make a directory if it doesn't already exist. - - Args: - dirname: Directory to create - """ - try: - os.mkdir(dirname) - except OSError as err: - if err.errno == errno.EEXIST: - pass - else: - raise - -class BuilderJob: - """Holds information about a job to be performed by a thread - - Members: - board: Board object to build - commits: List of commit options to build. - """ - def __init__(self): - self.board = None - self.commits = [] - - -class ResultThread(threading.Thread): - """This thread processes results from builder threads. - - It simply passes the results on to the builder. There is only one - result thread, and this helps to serialise the build output. - """ - def __init__(self, builder): - """Set up a new result thread - - Args: - builder: Builder which will be sent each result - """ - threading.Thread.__init__(self) - self.builder = builder - - def run(self): - """Called to start up the result thread. - - We collect the next result job and pass it on to the build. - """ - while True: - result = self.builder.out_queue.get() - self.builder.ProcessResult(result) - self.builder.out_queue.task_done() - - -class BuilderThread(threading.Thread): - """This thread builds U-Boot for a particular board. - - An input queue provides each new job. We run 'make' to build U-Boot - and then pass the results on to the output queue. - - Members: - builder: The builder which contains information we might need - thread_num: Our thread number (0-n-1), used to decide on a - temporary directory - """ - def __init__(self, builder, thread_num): - """Set up a new builder thread""" - threading.Thread.__init__(self) - self.builder = builder - self.thread_num = thread_num - - def Make(self, commit, brd, stage, cwd, *args, **kwargs): - """Run 'make' on a particular commit and board. - - The source code will already be checked out, so the 'commit' - argument is only for information. - - Args: - commit: Commit object that is being built - brd: Board object that is being built - stage: Stage of the build. Valid stages are: - distclean - can be called to clean source - config - called to configure for a board - build - the main make invocation - it does the build - args: A list of arguments to pass to 'make' - kwargs: A list of keyword arguments to pass to command.RunPipe() - - Returns: - CommandResult object - """ - return self.builder.do_make(commit, brd, stage, cwd, *args, - **kwargs) - - def RunCommit(self, commit_upto, brd, work_dir, do_config, force_build, - force_build_failures): - """Build a particular commit. - - If the build is already done, and we are not forcing a build, we skip - the build and just return the previously-saved results. - - Args: - commit_upto: Commit number to build (0...n-1) - brd: Board object to build - work_dir: Directory to which the source will be checked out - do_config: True to run a make <board>_defconfig on the source - force_build: Force a build even if one was previously done - force_build_failures: Force a bulid if the previous result showed - failure - - Returns: - tuple containing: - - CommandResult object containing the results of the build - - boolean indicating whether 'make config' is still needed - """ - # Create a default result - it will be overwritte by the call to - # self.Make() below, in the event that we do a build. - result = command.CommandResult() - result.return_code = 0 - if self.builder.in_tree: - out_dir = work_dir - else: - out_dir = os.path.join(work_dir, 'build') - - # Check if the job was already completed last time - done_file = self.builder.GetDoneFile(commit_upto, brd.target) - result.already_done = os.path.exists(done_file) - will_build = (force_build or force_build_failures or - not result.already_done) - if result.already_done and will_build: - # Get the return code from that build and use it - with open(done_file, 'r') as fd: - result.return_code = int(fd.readline()) - err_file = self.builder.GetErrFile(commit_upto, brd.target) - if os.path.exists(err_file) and os.stat(err_file).st_size: - result.stderr = 'bad' - elif not force_build: - # The build passed, so no need to build it again - will_build = False - - if will_build: - # We are going to have to build it. First, get a toolchain - if not self.toolchain: - try: - self.toolchain = self.builder.toolchains.Select(brd.arch) - except ValueError as err: - result.return_code = 10 - result.stdout = '' - result.stderr = str(err) - # TODO(sjg at chromium.org): This gets swallowed, but needs - # to be reported. - - if self.toolchain: - # Checkout the right commit - if self.builder.commits: - commit = self.builder.commits[commit_upto] - if self.builder.checkout: - git_dir = os.path.join(work_dir, '.git') - gitutil.Checkout(commit.hash, git_dir, work_dir, - force=True) - else: - commit = 'current' - - # Set up the environment and command line - env = self.toolchain.MakeEnvironment() - Mkdir(out_dir) - args = [] - cwd = work_dir - if not self.builder.in_tree: - if commit_upto is None: - # In this case we are building in the original source - # directory (i.e. the current directory where buildman - # is invoked. The output directory is set to this - # thread's selected work directory. - # - # Symlinks can confuse U-Boot's Makefile since - # we may use '..' in our path, so remove them. - work_dir = os.path.realpath(work_dir) - args.append('O=%s/build' % work_dir) - cwd = None - else: - args.append('O=build') - args.append('-s') - if self.builder.num_jobs is not None: - args.extend(['-j', str(self.builder.num_jobs)]) - config_args = ['%s_defconfig' % brd.target] - config_out = '' - args.extend(self.builder.toolchains.GetMakeArguments(brd)) - - # If we need to reconfigure, do that now - if do_config: - result = self.Make(commit, brd, 'distclean', cwd, - 'distclean', *args, env=env) - result = self.Make(commit, brd, 'config', cwd, - *(args + config_args), env=env) - config_out = result.combined - do_config = False # No need to configure next time - if result.return_code == 0: - result = self.Make(commit, brd, 'build', cwd, *args, - env=env) - result.stdout = config_out + result.stdout - else: - result.return_code = 1 - result.stderr = 'No tool chain for %s\n' % brd.arch - result.already_done = False - - result.toolchain = self.toolchain - result.brd = brd - result.commit_upto = commit_upto - result.out_dir = out_dir - return result, do_config - - def _WriteResult(self, result, keep_outputs): - """Write a built result to the output directory. - - Args: - result: CommandResult object containing result to write - keep_outputs: True to store the output binaries, False - to delete them - """ - # Fatal error - if result.return_code < 0: - return - - # Aborted? - if result.stderr and 'No child processes' in result.stderr: - return - - if result.already_done: - return - - # Write the output and stderr - output_dir = self.builder._GetOutputDir(result.commit_upto) - Mkdir(output_dir) - build_dir = self.builder.GetBuildDir(result.commit_upto, - result.brd.target) - Mkdir(build_dir) - - outfile = os.path.join(build_dir, 'log') - with open(outfile, 'w') as fd: - if result.stdout: - fd.write(result.stdout) - - errfile = self.builder.GetErrFile(result.commit_upto, - result.brd.target) - if result.stderr: - with open(errfile, 'w') as fd: - fd.write(result.stderr) - elif os.path.exists(errfile): - os.remove(errfile) - - if result.toolchain: - # Write the build result and toolchain information. - done_file = self.builder.GetDoneFile(result.commit_upto, - result.brd.target) - with open(done_file, 'w') as fd: - fd.write('%s' % result.return_code) - with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: - print >>fd, 'gcc', result.toolchain.gcc - print >>fd, 'path', result.toolchain.path - print >>fd, 'cross', result.toolchain.cross - print >>fd, 'arch', result.toolchain.arch - fd.write('%s' % result.return_code) - - with open(os.path.join(build_dir, 'toolchain'), 'w') as fd: - print >>fd, 'gcc', result.toolchain.gcc - print >>fd, 'path', result.toolchain.path - - # Write out the image and function size information and an objdump - env = result.toolchain.MakeEnvironment() - lines = [] - for fname in ['u-boot', 'spl/u-boot-spl']: - cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname] - nm_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - if nm_result.stdout: - nm = self.builder.GetFuncSizesFile(result.commit_upto, - result.brd.target, fname) - with open(nm, 'w') as fd: - print >>fd, nm_result.stdout, - - cmd = ['%sobjdump' % self.toolchain.cross, '-h', fname] - dump_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - rodata_size = '' - if dump_result.stdout: - objdump = self.builder.GetObjdumpFile(result.commit_upto, - result.brd.target, fname) - with open(objdump, 'w') as fd: - print >>fd, dump_result.stdout, - for line in dump_result.stdout.splitlines(): - fields = line.split() - if len(fields) > 5 and fields[1] == '.rodata': - rodata_size = fields[2] - - cmd = ['%ssize' % self.toolchain.cross, fname] - size_result = command.RunPipe([cmd], capture=True, - capture_stderr=True, cwd=result.out_dir, - raise_on_error=False, env=env) - if size_result.stdout: - lines.append(size_result.stdout.splitlines()[1] + ' ' + - rodata_size) - - # Write out the image sizes file. This is similar to the output - # of binutil's 'size' utility, but it omits the header line and - # adds an additional hex value at the end of each line for the - # rodata size - if len(lines): - sizes = self.builder.GetSizesFile(result.commit_upto, - result.brd.target) - with open(sizes, 'w') as fd: - print >>fd, '\n'.join(lines) - - # Now write the actual build output - if keep_outputs: - patterns = ['u-boot', '*.bin', 'u-boot.dtb', '*.map', - 'include/autoconf.mk', 'spl/u-boot-spl', - 'spl/u-boot-spl.bin'] - for pattern in patterns: - file_list = glob.glob(os.path.join(result.out_dir, pattern)) - for fname in file_list: - shutil.copy(fname, build_dir) - - - def RunJob(self, job): - """Run a single job - - A job consists of a building a list of commits for a particular board. - - Args: - job: Job to build - """ - brd = job.board - work_dir = self.builder.GetThreadDir(self.thread_num) - self.toolchain = None - if job.commits: - # Run 'make board_defconfig' on the first commit - do_config = True - commit_upto = 0 - force_build = False - for commit_upto in range(0, len(job.commits), job.step): - result, request_config = self.RunCommit(commit_upto, brd, - work_dir, do_config, - force_build or self.builder.force_build, - self.builder.force_build_failures) - failed = result.return_code or result.stderr - did_config = do_config - if failed and not do_config: - # If our incremental build failed, try building again - # with a reconfig. - if self.builder.force_config_on_failure: - result, request_config = self.RunCommit(commit_upto, - brd, work_dir, True, True, False) - did_config = True - if not self.builder.force_reconfig: - do_config = request_config - - # If we built that commit, then config is done. But if we got - # an warning, reconfig next time to force it to build the same - # files that created warnings this time. Otherwise an - # incremental build may not build the same file, and we will - # think that the warning has gone away. - # We could avoid this by using -Werror everywhere... - # For errors, the problem doesn't happen, since presumably - # the build stopped and didn't generate output, so will retry - # that file next time. So we could detect warnings and deal - # with them specially here. For now, we just reconfigure if - # anything goes work. - # Of course this is substantially slower if there are build - # errors/warnings (e.g. 2-3x slower even if only 10% of builds - # have problems). - if (failed and not result.already_done and not did_config and - self.builder.force_config_on_failure): - # If this build failed, try the next one with a - # reconfigure. - # Sometimes if the board_config.h file changes it can mess - # with dependencies, and we get: - # make: *** No rule to make target `include/autoconf.mk', - # needed by `depend'. - do_config = True - force_build = True - else: - force_build = False - if self.builder.force_config_on_failure: - if failed: - do_config = True - result.commit_upto = commit_upto - if result.return_code < 0: - raise ValueError('Interrupt') - - # We have the build results, so output the result - self._WriteResult(result, job.keep_outputs) - self.builder.out_queue.put(result) - else: - # Just build the currently checked-out build - result, request_config = self.RunCommit(None, brd, work_dir, True, - True, self.builder.force_build_failures) - result.commit_upto = 0 - self._WriteResult(result, job.keep_outputs) - self.builder.out_queue.put(result) - - def run(self): - """Our thread's run function - - This thread picks a job from the queue, runs it, and then goes to the - next job. - """ - alive = True - while True: - job = self.builder.queue.get() - if self.builder.active and alive: - self.RunJob(job) - ''' - try: - if self.builder.active and alive: - self.RunJob(job) - except Exception as err: - alive = False - print err - ''' - self.builder.queue.task_done() - - class Builder: """Class for building U-Boot for a particular commit.
@@ -639,13 +216,13 @@ class Builder: self.queue = Queue.Queue() self.out_queue = Queue.Queue() for i in range(self.num_threads): - t = BuilderThread(self, i) + t = builderthread.BuilderThread(self, i) t.setDaemon(True) t.start() self.threads.append(t) self.last_line_len = 0 - t = ResultThread(self) + t = builderthread.ResultThread(self) t.setDaemon(True) t.start() self.threads.append(t)
@@ -1385,7 +962,7 @@ class Builder: for self.commit_upto in range(self.commit_count): self.SelectCommit(commits[self.commit_upto]) self.SelectOutputDir() - Mkdir(self.output_dir) + builderthread.Mkdir(self.output_dir) self.BuildBoardsForCommit(board_selected, keep_outputs) board_dict, err_lines = self.GetResultSummary()
@@ -1413,7 +990,7 @@ class Builder: setup_git: True to set up a git repo clone """ thread_dir = self.GetThreadDir(thread_num) - Mkdir(thread_dir) + builderthread.Mkdir(thread_dir) git_dir = os.path.join(thread_dir, '.git') # Clone the repo if it doesn't already exist
@@ -1436,7 +1013,7 @@ class Builder: max_threads: Maximum number of threads we expect to need. setup_git: True to set up a git repo clone """ - Mkdir(self._working_dir) + builderthread.Mkdir(self._working_dir) for thread in range(max_threads): self._PrepareThread(thread, setup_git)
@@ -1469,7 +1046,7 @@ class Builder: self.commits = commits self.ResetResultSummary(board_selected) - Mkdir(self.base_dir) + builderthread.Mkdir(self.base_dir) self._PrepareWorkingSpace(min(self.num_threads, len(board_selected)), commits is not None) self._PrepareOutputSpace()
@@ -1478,7 +1055,7 @@ class Builder: # Create jobs to build all commits for each board for brd in board_selected.itervalues(): - job = BuilderJob() + job = builderthread.BuilderJob() job.board = brd job.commits = commits job.keep_outputs = keep_outputs
tools/buildman/builderthread.py new file +434 -0
434 lines not on this page · raw message
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 5/13 buildman: Sort command line options
In reply to Simon GlassThese options have got slightly out of order. Fix them.
---
Changes in v5: None
Changes in v4:
- Fix typo in commit message
Changes in v3:
- Add new patch to sort command line options
Changes in v2: None
tools/buildman/buildman.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
tools/buildman/buildman.py +6 -6
@@ -70,6 +70,9 @@ parser.add_option('-c', '--count', dest='count', type='int', parser.add_option('-C', '--force-reconfig', dest='force_reconfig', action='store_true', default=False, help='Reconfigure for every commit (disable incremental build)') +parser.add_option('-d', '--detail', dest='show_detail', + action='store_true', default=False, + help='Show detailed information for each board in summary') parser.add_option('-e', '--show_errors', action='store_true', default=False, help='Show errors and warnings') parser.add_option('-f', '--force-build', dest='force_build',
@@ -78,9 +81,6 @@ parser.add_option('-f', '--force-build', dest='force_build', parser.add_option('-F', '--force-build-failures', dest='force_build_failures', action='store_true', default=False, help='Force build of previously-failed build') -parser.add_option('-d', '--detail', dest='show_detail', - action='store_true', default=False, - help='Show detailed information for each board in summary') parser.add_option('-g', '--git', type='string', help='Git repo containing branch to build', default='.') parser.add_option('-H', '--full-help', action='store_true', dest='full_help',
@@ -96,6 +96,9 @@ parser.add_option('--list-tool-chains', action='store_true', default=False, help='List available tool chains') parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run', default=False, help="Do a try run (describe actions, but no nothing)") +parser.add_option('-o', '--output-dir', type='string', + dest='output_dir', default='..', + help='Directory where all builds happen and buildman has its workspace (default is ../)') parser.add_option('-Q', '--quick', action='store_true', default=False, help='Do a rough build, with limited warning resolution') parser.add_option('-s', '--summary', action='store_true',
@@ -110,9 +113,6 @@ parser.add_option('-T', '--threads', type='int', default=None, help='Number of builder threads to use') parser.add_option('-u', '--show_unknown', action='store_true', default=False, help='Show boards with unknown build result') -parser.add_option('-o', '--output-dir', type='string', - dest='output_dir', default='..', - help='Directory where all builds happen and buildman has its workspace (default is ../)') parser.usage = """buildman -b <branch> [options]
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 6/13 buildman: Refactor output options
In reply to Simon GlassWe need the output options to be available in several places. It's a pain to pass them into each function. Make them properties of the builder and add a single function to set them up. At the same time, add a function which produces summary output using these options.
---
Changes in v5:
- Update test for output options
Changes in v4: None
Changes in v3:
- Add new patch to refactor output options
Changes in v2: None
tools/buildman/builder.py | 46 +++++++++++++++++++++++++++------------------- tools/buildman/control.py | 8 ++++---- tools/buildman/test.py | 6 +++--- 3 files changed, 34 insertions(+), 26 deletions(-)
tools/buildman/builder.py +27 -19
@@ -235,6 +235,20 @@ class Builder: for t in self.threads: del t + def SetDisplayOptions(self, show_errors=False, show_sizes=False, + show_detail=False, show_bloat=False): + """Setup display options for the builder. + + show_errors: True to show summarised error/warning info + show_sizes: Show size deltas + show_detail: Show detail for each board + show_bloat: Show detail for each function + """ + self._show_errors = show_errors + self._show_sizes = show_sizes + self._show_detail = show_detail + self._show_bloat = show_bloat + def _AddTimestamp(self): """Add a new timestamp to the list and record the build period.
@@ -891,9 +905,18 @@ class Builder: print "Boards not built (%d): %s" % (len(not_built), ', '.join(not_built)) + def ProduceResultSummary(self, commit_upto, commits, board_selected): + board_dict, err_lines = self.GetResultSummary(board_selected, + commit_upto, read_func_sizes=self._show_bloat) + if commits: + msg = '%02d: %s' % (commit_upto + 1, + commits[commit_upto].subject) + print self.col.Color(self.col.BLUE, msg) + self.PrintResultSummary(board_selected, board_dict, + err_lines if self._show_errors else [], + self._show_sizes, self._show_detail, self._show_bloat) - def ShowSummary(self, commits, board_selected, show_errors, show_sizes, - show_detail, show_bloat): + def ShowSummary(self, commits, board_selected): """Show a build summary for U-Boot for a given board list. Reset the result summary, then repeatedly call GetResultSummary on
@@ -902,27 +925,13 @@ class Builder: Args: commit: Commit objects to summarise board_selected: Dict containing boards to summarise - show_errors: Show errors that occured - show_sizes: Show size deltas - show_detail: Show detail for each board - show_bloat: Show detail for each function """ self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected) for commit_upto in range(0, self.commit_count, self._step): - board_dict, err_lines = self.GetResultSummary(board_selected, - commit_upto, read_func_sizes=show_bloat) - if commits: - msg = '%02d: %s' % (commit_upto + 1, - commits[commit_upto].subject) - else: - msg = 'current' - print self.col.Color(self.col.BLUE, msg) - self.PrintResultSummary(board_selected, board_dict, - err_lines if show_errors else [], show_sizes, show_detail, - show_bloat) + self.ProduceResultSummary(commit_upto, commits, board_selected) def SetupBuild(self, board_selected, commits):
@@ -1032,14 +1041,13 @@ class Builder: if dirname not in dir_list: shutil.rmtree(dirname) - def BuildBoards(self, commits, board_selected, show_errors, keep_outputs): + def BuildBoards(self, commits, board_selected, keep_outputs): """Build all commits for a list of boards Args: commits: List of commits to be build, each a Commit object boards_selected: Dict of selected boards, key is target name, value is Board object - show_errors: True to show summarised error/warning info keep_outputs: True to save build output files """ self.commit_count = len(commits) if commits else 1
tools/buildman/control.py +4 -4
@@ -207,13 +207,13 @@ def DoBuildman(options, args): print GetActionSummary(options.summary, commits, board_selected, options) + builder.SetDisplayOptions(options.show_errors, options.show_sizes, + options.show_detail, options.show_bloat) if options.summary: # We can't show function sizes without board details at present if options.show_bloat: options.show_detail = True - builder.ShowSummary(commits, board_selected, - options.show_errors, options.show_sizes, - options.show_detail, options.show_bloat) + builder.ShowSummary(commits, board_selected) else: builder.BuildBoards(commits, board_selected, - options.show_errors, options.keep_outputs) + options.keep_outputs)
tools/buildman/test.py +3 -3
@@ -137,9 +137,9 @@ class TestBuild(unittest.TestCase): board_selected = self.boards.GetSelectedDict() #build.BuildCommits(self.commits, board_selected, False) - build.BuildBoards(self.commits, board_selected, False, False) - build.ShowSummary(self.commits, board_selected, True, False, - False, False) + build.BuildBoards(self.commits, board_selected, False) + build.SetDisplayOptions(show_errors=True); + build.ShowSummary(self.commits, board_selected) def _testGit(self): """Test basic builder operation by building a branch"""
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 7/13 buildman: Add verbose option to display errors as they happen
In reply to Simon GlassNormally buildman operates in two passes - one to do the build and another to summarise the errors. Add a verbose option (-v) to display build problems as they happen. With -e also given, this will display errors too.
When building the current source tree (rather than a list of commits in a branch), both -v and -e are enabled automatically.
---
Changes in v5:
- Update test for verbose option
Changes in v4: None
Changes in v3:
- Add new patch to add a verbose option
Changes in v2: None
tools/buildman/README | 46 ++++++++++++++++++++++++++++++++++++---------- tools/buildman/builder.py | 16 +++++++++++++--- tools/buildman/buildman.py | 2 ++ tools/buildman/control.py | 4 +++- tools/buildman/test.py | 3 ++- 5 files changed, 56 insertions(+), 15 deletions(-)
tools/buildman/README +36 -10
@@ -41,9 +41,10 @@ Theory of Operation Buildman is a builder. It is not make, although it runs make. It does not produce any useful output on the terminal while building, except for -progress information. All the output (errors, warnings and binaries if you -are ask for them) is stored in output directories, which you can look at -while the build is progressing, or when it is finished. +progress information (except with -v, see below). All the output (errors, +warnings and binaries if you are ask for them) is stored in output +directories, which you can look at while the build is progressing, or when +it is finished. Buildman produces a concise summary of which boards succeeded and failed. It shows which commit introduced which board failure using a simple
@@ -77,12 +78,17 @@ Buildman automatically selects the correct tool chain for each board. You must supply suitable tool chains, but buildman takes care of selecting the right one. -Buildman always builds a branch, and always builds the upstream commit as -well, for comparison. It cannot build individual commits at present, unless -(maybe) you point it at an empty branch. Put all your commits in a branch, -set the branch's upstream to a valid value, and all will be well. Otherwise -buildman will perform random actions. Use -n to check what the random -actions might be. +Buildman generally builds a branch (with the -b flag), and in this case +builds the upstream commit as well, for comparison. It cannot build +individual commits at present, unless (maybe) you point it at an empty +branch. Put all your commits in a branch, set the branch's upstream to a +valid value, and all will be well. Otherwise buildman will perform random +actions. Use -n to check what the random actions might be. + +If you just want to build the current source tree, leave off the -b flag. +This will display results and errors as they happen. You can still look +at them later using -s. Note that buildman will assume that the source +has changed, and will build all specified boards in this case. Buildman is optimised for building many commits at once, for many boards. On multi-core machines, Buildman is fast because it uses most of the
@@ -659,6 +665,15 @@ It is expected that any variables added are dealt with in U-Boot's config.mk file and documented in the README. +Quick Sanity Check +================== + +If you have made changes and want to do a quick sanity check of the +currently-checked-out source, run buildman without the -b flag. This will +build the selected boards and display build status and errors as it runs +(i.e. -v amd -e are enabled automatically). + + Other options =============
@@ -685,7 +700,15 @@ First you need to set up your tool chains - see the 'Setting up' section for details. Once you have your required toolchain(s) detected then you are ready to go. -Buildman works on entire branches, so the normal use is: +To build the current source tree, run buildman without a -b flag: + + ./tools/buildman/buildman <list of things to build> + +This will build the current source tree for the given boards and display +the results and errors. + +However buildman usually works on entire branches, and for that you must +specify a board flag: ./tools/buildman/buildman -b <branch_name> <list of things to build>
@@ -698,6 +721,9 @@ buildman just shows a summary, with red indicating that a commit introduced an error and green indicating that a commit fixed an error. Use the -e flag to see the full errors. +If you really want to see build results as they happen, use -v when doing a +build (and -e if you want to see errors as well). + You don't need to stick around on that branch while buildman is running. It checks out its own copy of the source code, so you can change branches, add commits, etc. without affecting the build in progress.
tools/buildman/builder.py +13 -3
@@ -321,7 +321,8 @@ class Builder: """Process the result of a build, showing progress information Args: - result: A CommandResult object + result: A CommandResult object, which indicates the result for + a single build """ col = terminal.Color() if result:
@@ -339,6 +340,13 @@ class Builder: self.warned += 1 if result.already_done: self.already_done += 1 + if self._verbose: + print '\r', + self.ClearLine(0) + boards_selected = {target : result.brd} + self.ResetResultSummary(boards_selected) + self.ProduceResultSummary(result.commit_upto, self.commits, + boards_selected) else: target = '(starting)'
@@ -362,7 +370,7 @@ class Builder: name += target print line + name, - length = 13 + len(name) + length = 14 + len(name) self.ClearLine(length) def _GetOutputDir(self, commit_upto):
@@ -1041,7 +1049,7 @@ class Builder: if dirname not in dir_list: shutil.rmtree(dirname) - def BuildBoards(self, commits, board_selected, keep_outputs): + def BuildBoards(self, commits, board_selected, keep_outputs, verbose): """Build all commits for a list of boards Args:
@@ -1049,9 +1057,11 @@ class Builder: boards_selected: Dict of selected boards, key is target name, value is Board object keep_outputs: True to save build output files + verbose: Display build results as they are completed """ self.commit_count = len(commits) if commits else 1 self.commits = commits + self._verbose = verbose self.ResetResultSummary(board_selected) builderthread.Mkdir(self.base_dir)
tools/buildman/buildman.py +2 -0
@@ -113,6 +113,8 @@ parser.add_option('-T', '--threads', type='int', default=None, help='Number of builder threads to use') parser.add_option('-u', '--show_unknown', action='store_true', default=False, help='Show boards with unknown build result') +parser.add_option('-v', '--verbose', action='store_true', + default=False, help='Show build results while the build progresses') parser.usage = """buildman -b <branch> [options]
tools/buildman/control.py +3 -1
@@ -158,6 +158,8 @@ def DoBuildman(options, args): series) else: series = None + options.verbose = True + options.show_errors = True # By default we have one thread per CPU. But if there are not enough jobs # we can have fewer threads and use a high '-j' value for make.
@@ -216,4 +218,4 @@ def DoBuildman(options, args): builder.ShowSummary(commits, board_selected) else: builder.BuildBoards(commits, board_selected, - options.keep_outputs) + options.keep_outputs, options.verbose)
tools/buildman/test.py +2 -1
@@ -137,7 +137,8 @@ class TestBuild(unittest.TestCase): board_selected = self.boards.GetSelectedDict() #build.BuildCommits(self.commits, board_selected, False) - build.BuildBoards(self.commits, board_selected, False) + build.BuildBoards(self.commits, board_selected, keep_outputs=False, + verbose=False) build.SetDisplayOptions(show_errors=True); build.ShowSummary(self.commits, board_selected)
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 8/13 buildman: Remove unused non-incremental build method code
In reply to Simon GlassThe non-incremental build method is no longer used, so remove it.
---
Changes in v5: None
Changes in v4:
- Add new patch to remove unused non-incremental build method code
Changes in v3: None
Changes in v2: None
tools/buildman/builder.py | 34 ---------------------------------- tools/buildman/test.py | 1 - 2 files changed, 35 deletions(-)
tools/buildman/builder.py +0 -34
@@ -955,40 +955,6 @@ class Builder: self.upto = self.warned = self.fail = 0 self._timestamps = collections.deque() - def BuildBoardsForCommit(self, board_selected, keep_outputs): - """Build all boards for a single commit""" - self.SetupBuild(board_selected) - self.count = len(board_selected) - for brd in board_selected.itervalues(): - job = BuilderJob() - job.board = brd - job.commits = None - job.keep_outputs = keep_outputs - self.queue.put(brd) - - self.queue.join() - self.out_queue.join() - print - self.ClearLine(0) - - def BuildCommits(self, commits, board_selected, show_errors, keep_outputs): - """Build all boards for all commits (non-incremental)""" - self.commit_count = len(commits) - - self.ResetResultSummary(board_selected) - for self.commit_upto in range(self.commit_count): - self.SelectCommit(commits[self.commit_upto]) - self.SelectOutputDir() - builderthread.Mkdir(self.output_dir) - - self.BuildBoardsForCommit(board_selected, keep_outputs) - board_dict, err_lines = self.GetResultSummary() - self.PrintResultSummary(board_selected, board_dict, - err_lines if show_errors else []) - - if self.already_done: - print '%d builds already done' % self.already_done - def GetThreadDir(self, thread_num): """Get the directory path to the working dir for a thread.
tools/buildman/test.py +0 -1
@@ -136,7 +136,6 @@ class TestBuild(unittest.TestCase): build.do_make = self.Make board_selected = self.boards.GetSelectedDict() - #build.BuildCommits(self.commits, board_selected, False) build.BuildBoards(self.commits, board_selected, keep_outputs=False, verbose=False) build.SetDisplayOptions(show_errors=True);
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 9/13 buildman: Add an option to specify the buildman config file
In reply to Simon GlassAdd a new --config-file option (-G) to specify a different configuration file from the default ~/.buildman.
---
Changes in v5: None
Changes in v4:
- Add new patch to allow the config file to be specified
Changes in v3: None
Changes in v2: None
tools/buildman/buildman.py | 2 ++ tools/buildman/control.py | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
tools/buildman/buildman.py +2 -0
@@ -83,6 +83,8 @@ parser.add_option('-F', '--force-build-failures', dest='force_build_failures', help='Force build of previously-failed build') parser.add_option('-g', '--git', type='string', help='Git repo containing branch to build', default='.') +parser.add_option('-G', '--config-file', type='string', + help='Path to buildman config file', default='') parser.add_option('-H', '--full-help', action='store_true', dest='full_help', default=False, help='Display the README file') parser.add_option('-i', '--in-tree', dest='in_tree',
tools/buildman/control.py +1 -1
@@ -86,7 +86,7 @@ def DoBuildman(options, args): """ gitutil.Setup() - bsettings.Setup() + bsettings.Setup(options.config_file) options.git_dir = os.path.join(options.git, '.git') toolchains = toolchain.Toolchains()
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 10/13 buildman: Add a message indicating there are no errors
In reply to Simon GlassIf buildman finds no problems it prints nothing. This can be a bit confusing, so add a message that all is well.
---
Changes in v5: None
Changes in v4:
- Add new patch to show a message when there are no errors
Changes in v3: None
Changes in v2: None
tools/buildman/builder.py | 6 ++++++ 1 file changed, 6 insertions(+)
tools/buildman/builder.py +6 -0
@@ -891,10 +891,13 @@ class Builder: self.col.MAGENTA) for arch, target_list in arch_list.iteritems(): print '%10s: %s' % (arch, target_list) + self._error_lines += 1 if better_err: print self.col.Color(self.col.GREEN, '\n'.join(better_err)) + self._error_lines += 1 if worse_err: print self.col.Color(self.col.RED, '\n'.join(worse_err)) + self._error_lines += 1 if show_sizes: self.PrintSizeSummary(board_selected, board_dict, show_detail,
@@ -937,9 +940,12 @@ class Builder: self.commit_count = len(commits) if commits else 1 self.commits = commits self.ResetResultSummary(board_selected) + self._error_lines = 0 for commit_upto in range(0, self.commit_count, self._step): self.ProduceResultSummary(commit_upto, commits, board_selected) + if not self._error_lines: + print self.col.Color(self.col.GREEN, '(no errors to report)') def SetupBuild(self, board_selected, commits):
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 11/13 buildman: Add a few more toolchain examples to the README
In reply to Simon GlassThe current README is a bit sparse in this area, so add a few more examples.
---
Changes in v5: None
Changes in v4:
- Add new path to add a few more toolchain examples to the README
Changes in v3: None
Changes in v2: None
tools/buildman/README | 2 ++ 1 file changed, 2 insertions(+)
tools/buildman/README +2 -0
@@ -134,6 +134,8 @@ example: root: / rest: /toolchains/* eldk: /opt/eldk-4.2 +arm: /opt/linaro/gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux +aarch64: /opt/linaro/gcc-linaro-aarch64-none-elf-4.8-2013.10_linux [toolchain-alias] x86: i386
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 12/13 buildman: Introduce an 'and' operator for board selection
In reply to Simon GlassCurrently buildman allows a list of boards to build to be specified on the command line. The list can include specific board names, architecture, SOC and so on.
At present the list of boards is dealt with in an 'OR' fashion, and there is no way to specify something like 'arm & freescale', meaning boards with ARM architecture but only those made by Freescale. This would exclude the PowerPC boards made by Freescale.
Support an '&' operator on the command line to permit this. Ensure that arguments can be specified in a single string to permit easy shell quoting.
---
Changes in v5:
- Add new patch to add an 'and' operator for board selection
Changes in v4: None
Changes in v3: None
Changes in v2: None
tools/buildman/README | 9 +++ tools/buildman/board.py | 144 ++++++++++++++++++++++++++++++++++++++++++------ tools/buildman/test.py | 48 ++++++++++++++++ 3 files changed, 184 insertions(+), 17 deletions(-)
tools/buildman/README +9 -0
@@ -108,6 +108,15 @@ behaviour is a superset of exact or substring matching. Examples are: * '^tegra[23]0$' All boards with either Tegra20 or Tegra30 SoC * 'powerpc' All PowerPC boards +While the default is to OR the terms together, you can also make use of +the '&' operator to limit the selection: + +* 'freescale & arm sandbox' All Freescale boards with ARM architecture, + plus sandbox + +It is convenient to use the -n option to see whaat will be built based on +the subset given. + Buildman does not store intermediate object files. It optionally copies the binary output into a directory when a build is successful. Size information is always recorded. It needs a fair bit of disk space to work,
tools/buildman/board.py +127 -17
165 lines not on this page · raw message
tools/buildman/test.py +48 -0
53 lines not on this page · raw message
Signature · 1 line
2.0.0.526.g5318336
PATCH v5 12/13 buildman: Introduce an 'and' operator for board selection
Quoted 38 lines · Simon Glass
Currently buildman allows a list of boards to build to be specified on the command line. The list can include specific board names, architecture, SOC and so on.
At present the list of boards is dealt with in an 'OR' fashion, and there is no way to specify something like 'arm & freescale', meaning boards with ARM architecture but only those made by Freescale. This would exclude the PowerPC boards made by Freescale.
Support an '&' operator on the command line to permit this. Ensure that arguments can be specified in a single string to permit easy shell quoting.
Suggested-by York Sun <[email protected]>Signed-off-by Simon Glass <[email protected]>---
Changes in v5:
- Add new patch to add an 'and' operator for board selectionChanges in v4: None
Changes in v3: None
Changes in v2: Nonetools/buildman/README | 9 +++ tools/buildman/board.py | 144 ++++++++++++++++++++++++++++++++++++++++++------ tools/buildman/test.py | 48 ++++++++++++++++ 3 files changed, 184 insertions(+), 17 deletions(-)tools/buildman/README +2
-0@@ -108,6 +108,15 @@ behaviour is a superset of exact or substring matching. Examples are: * '^tegra[23]0$' All boards with either Tegra20 or Tegra30 SoC * 'powerpc' All PowerPC boards +While the default is to OR the terms together, you can also make use of +the '&' operator to limit the selection:
tools/buildman/README behaviour is a superset of exact or substring matching. Examples are: · line 122
Quoted 10 lines
tools/buildman/README +7
-0+ +* 'freescale & arm sandbox' All Freescale boards with ARM architecture, + plus sandbox + +It is convenient to use the -n option to see whaat will be built based on +the subset given. + Buildman does not store intermediate object files. It optionally copies the binary output into a directory when a build is successful. Size information is always recorded. It needs a fair bit of disk space to work,
Simon,
It is good to see this working. It would be helpful to point out when using & logic, users should put them in quotes (single or double).
I can't get the --count=1 work though. See my example
$ tools/buildman/buildman -c 1 -b working_85xx t104x -n No section: 'make-flags' Dry run, so not doing much. But I would do this:
Building 16 commits for 9 boards (4 threads, 1 job per thread) Build directory: ../working_upstream_85xx
e76b933e Prepare v2014.10-rc1
89c8bc86 fsl/diu: ch7301 encoder split off from t1040qds/diu.c 57d28806 powerpc/t1042RDB: Add Video - HDMI support dc9ed535 buildman: Fix a few typos
04344da3 buildman: Add some notes about moving from MAKEALL 4aa63fae buildman: Allow building of current source tree 1e5d6ab5 buildman: Move BuilderThread code to its own file fc5f1107 buildman: Sort command line options
eacd9496 buildman: Refactor output options
7793b089 buildman: Add verbose option to display errors as they happen 1e0d4a12 buildman: Remove unused non-incremental build method code deaa8cfd buildman: Add an option to specify the buildman config file 8bacf282 buildman: Add a message indicating there are no errors e292dd4c buildman: Add a few more toolchain examples to the README 384e98f1 buildman: Introduce an 'and' operator for board selection 1eddd296 RFC: Deprecate MAKEALL
t104x : 9 boards
Total boards to build for each commit: 9
I expect only the last commit to build. Am I wrong? I don't see --count in the README file.
York
PATCH v5 12/13 buildman: Introduce an 'and' operator for board selection
Hi York,
Quoted 2 lines · York Sun
Quoted 1 line · Simon Glass
Currently buildman allows a list of boards to build to be specified on
the
Quoted 1 line
Quoted 1 line
command line. The list can include specific board names, architecture,
SOC
Quoted 4 lines
Quoted 4 lines
and so on.
At present the list of boards is dealt with in an 'OR' fashion, and there is no way to specify something like 'arm & freescale', meaning boards
with
Quoted 5 lines
Quoted 5 lines
ARM architecture but only those made by Freescale. This would exclude the PowerPC boards made by Freescale.
Support an '&' operator on the command line to permit this. Ensure that arguments can be specified in a single string to permit easy shell
quoting.
Quoted 8 lines
Quoted 8 lines
Suggested-by York Sun <[email protected]>Signed-off-by Simon Glass <[email protected]>---
Changes in v5:
- Add new patch to add an 'and' operator for board selection
Quoted 6 lines
Quoted 6 lines
Changes in v4: None
Changes in v3: None
Changes in v2: None
tools/buildman/README| 9 +++tools/buildman/board.py| 144
++++++++++++++++++++++++++++++++++++++++++------
Quoted 8 lines
tools/buildman/README behaviour is a superset of exact or substring
Quoted 8 lines
tools/buildman/test.py| 48 ++++++++++++++++3 files changed, 184 insertions(+), 17 deletions(-)tools/buildman/README +0
-0@@ -108,6 +108,15 @@ behaviour is a superset of exact or substring
matching. Examples are:
Quoted 10 lines
tools/buildman/README behaviour is a superset of exact or substring matching. Examples are:
Quoted 10 lines
tools/buildman/README +7
-0* '^tegra[23]0$' All boards with either Tegra20 or Tegra30 SoC * 'powerpc' All PowerPC boards +While the default is to OR the terms together, you can also make use of +the '&' operator to limit the selection: + +* 'freescale & arm sandbox' All Freescale boards with ARM architecture, + plus sandbox + +It is convenient to use the -n option to see whaat will be built based
on
Quoted 5 lines
Quoted 5 lines
tools/buildman/README +2
-0+the subset given. +Buildman does not store intermediate object files. It optionally copies the binary output into a directory when a build is successful. Size information is always recorded. It needs a fair bit of disk space to
work,
Quoted 4 lines
Simon,
It is good to see this working. It would be helpful to point out when
using &
Quoted 27 lines
logic, users should put them in quotes (single or double).
I can't get the --count=1 work though. See my example
$ tools/buildman/buildman -c 1 -b working_85xx t104x -n No section: 'make-flags' Dry run, so not doing much. But I would do this:Building 16 commits for 9 boards (4 threads, 1 job per thread) Build directory: ../working_upstream_85xx
e76b933ePrepare v2014.10-rc189c8bc86fsl/diu: ch7301 encoder split off fromt1040qds/diu.c57d28806powerpc/t1042RDB: Add Video - HDMI supportdc9ed535buildman: Fix a few typos04344da3buildman: Add some notes about moving from MAKEALL4aa63faebuildman: Allow building of current source tree1e5d6ab5buildman: Move BuilderThread code to its own filefc5f1107buildman: Sort command line optionseacd9496buildman: Refactor output options7793b089buildman: Add verbose option to display errors as they happen1e0d4a12buildman: Remove unused non-incremental build method codedeaa8cfdbuildman: Add an option to specify the buildman config file8bacf282buildman: Add a message indicating there are no errorse292dd4cbuildman: Add a few more toolchain examples to the README384e98f1buildman: Introduce an 'and' operator for board selection1eddd296RFC: Deprecate MAKEALL
Quoted 4 lines
t104x : 9 boards
Total boards to build for each commit: 9I expect only the last commit to build. Am I wrong? I don't see --count
in the
Quoted 1 line
README file.
Yes it is not implemented. I will add this and send a new series.
Regards,
Simon
PATCH v5 13/13 RFC: Deprecate MAKEALL
In reply to Simon GlassSince buildman now includes most of the features of MAKEALL it is probably time to talk about deprecating MAKEALL.
Comments welcome.
---
Changes in v5:
- Drop patch to search for *cc instead of *gcc for the compiler
Changes in v4: None
Changes in v3: None
Changes in v2: None
MAKEALL | 4 ++++ 1 file changed, 4 insertions(+)
MAKEALL +4 -0
@@ -60,6 +60,10 @@ usage() exit ${ret} } +echo "** Note: MAKEALL is deprecated - please use buildman instead" +echo "** See tools/buildman/README for details" +echo + SHORT_OPTS="ha:c:v:s:b:lmMCnr" LONG_OPTS="help,arch:,cpu:,vendor:,soc:,board:,list,maintainers,mails,check,continue,rebuild-errors"
Signature · 1 line
2.0.0.526.g5318336
v5 cover Add some missing buildman features and deprecate MAKEALL
In reply to Simon GlassQuoted 2 lines · Simon Glass
Buildman has been around for a little over a year and is used by a fair number of U-Boot developers. However quite a few people still use MAKEALL.
Not a blocker for the series, but [toolchain-alias] is a required section today. Otherwise with arm, armv7a and powerpc found and doing a build for only powerpc I see tons of "No section: toolchain-alias" messages. I didn't plan on populating the aliases since I'm only doing arm/powerpc builds in this instance.
Signature · 8 lines
Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20140808/26df34bc/attachment.pgp>