Skip to contentSkip to conversation

scripts/autobuild-run: add --no-toolchains-csv option and drop docopt

PATCH In review James Hilliard 2 participants 3 messages Apr 13, 2022 – Jun 7, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 scripts/autobuild-run: add --no-toolchains-csv option and drop docopt

Since docopt is unmaintained and less flexible than argparse lets drop it and migrate to argparse.

---
Changes v2 -> v3:
- suppress --no-toolchains-csv default output from help
Changes v1 -> v2:
- fix configparser

---
 scripts/autobuild-run      | 202 ++++++-------
 scripts/docopt.LICENSE-MIT |  23 --
 scripts/docopt.py          | 581 -------------------------------------
 3 files changed, 93 insertions(+), 713 deletions(-)
 delete mode 100644 scripts/docopt.LICENSE-MIT
 delete mode 100644 scripts/docopt.py
scripts/autobuild-run +93 -109
index 9219134..efe9858 100755
@@ -57,56 +57,7 @@
 
 from __future__ import print_function
 
-# Don't tell docopt about the defaults, as it would not allow the following
-# priority hierarchy for arguments: command-line > config file > defaults
-defaults = {
-    '--ninstances': '1',
-    '--njobs': '1',
-    '--submitter': 'N/A',
-    '--make-opts': '',
-    '--nice': 0,
-    '--pid-file': '/tmp/buildroot-autobuild.pid',
-    '--http-url': 'http://autobuild.buildroot.org/',
-    '--toolchains-csv': 'support/config-fragments/autobuild/toolchain-configs.csv',
-    '--repo': 'https://github.com/buildroot/buildroot.git',
-}
-
-doc = """autobuild-run - run Buildroot autobuilder
-
-Usage: autobuild-run [options]
-
-Options:
-  -h, --help                     show this help message and exit
-  -V, --version                  show version
-  -n, --ninstances NINSTANCES    number of parallel instances
-                                 Defaults to %(--ninstances)s.
-  -j, --njobs NJOBS              number of parallel jobs
-                                 Defaults to %(--njobs)s.
-  --nice N                       Niceness, positive number
-                                 Defaults to %(--nice)s.
-  -s, --submitter SUBMITTER      name/machine of submitter
-                                 Defaults to %(--submitter)s.
-  --http-url URL                 URL of resource to submit your results.
-                                 Defaults to %(--http-url)s.
-  --http-login LOGIN             username to send results with
-                                 Not set by default.
-  --http-password PASSWORD       password to send results with (for security
-                                 reasons it is recommended to define this in the
-                                 config file instead, with user-read permissions
-                                 only)
-                                 Not set by default.
-  --make-opts OPTSTRING          string of extra options to pass to Buildroot
-                                 make, such as specific command wrappers
-                                 Empty by default.
-  --pid-file PATH                path to a file where to store the PID
-                                 Defaults to %(--pid-file)s.
-  -c, --config CONFIG            path to configuration file
-                                 Not set by default.
-  -d, --debug                    Send log output to stdout instead of log file
-  --toolchains-csv CSVFILE       Toolchain configuration file
-  -r, --repo URL                 URL of Buildroot repository to clone
-                                 Defaults to %(--repo)s
-
+epilog = """
 Format of the configuration file:
 
   All arguments can also be specified in the configuration file specified with
@@ -120,20 +71,18 @@ Format of the configuration file:
    http-login = <value>
    http-password = <value>
    submitter = <value>
-
-
-""" % defaults
-
-__doc__ = doc
+   no-toolchains-csv
+"""
 
 import contextlib
 import csv
-import docopt
+import argparse
 import errno
 import hashlib
 import mmap
 import multiprocessing
 import os
+import pathlib
 from random import randint
 import re
 import shutil
@@ -423,6 +372,8 @@ class Builder:
             if not os.path.isabs(toolchains_csv):
                 toolchains_csv = os.path.join(self.srcdir, toolchains_csv)
             args.extend(["--toolchains-csv", toolchains_csv])
+        else:
+            args.extend(["--no-toolchains-csv"])
 
         ret = subprocess.call(args, stdout=devnull, stderr=self.log)
         return ret
@@ -797,35 +748,20 @@ class Builder:
             except URLError as e:
                 sleep(30)
 
-# args / config file merging inspired by:
-# https://github.com/docopt/docopt/blob/master/examples/config_file_example.py
-
-def load_ini_config(configfile):
-    """Load configuration from file, returning a docopt-like dictionary"""
-
-    if not os.path.exists(configfile):
-        print("ERROR: configuration file %s does not exist" % configfile)
-        sys.exit(1)
-
-    config = configparser.RawConfigParser()
-    if not config.read(configfile):
-        print("ERROR: cannot parse configuration file %s" % configfile)
-        sys.exit(1)
-
-    # Prepend '--' to options specified in the config file, so they can be
-    # merged with those given on the command-line
-    return dict(('--%s' % key, value) for key, value in config.items('main'))
-
-
-def merge(dict_1, dict_2):
-    """Merge two dictionaries.
-
-    Values that evaluate to true take priority over falsy values.
-    `dict_1` takes priority over `dict_2`.
-
-    """
-    return dict((str(key), dict_1.get(key) or dict_2.get(key))
-                for key in set(dict_2) | set(dict_1))
+class Formatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
+    pass
+
+class LoadConfigFile(argparse.Action):
+    def __call__(self, parser, namespace, values, option_string=None):
+        config = configparser.RawConfigParser(allow_no_value=True)
+        config.read_file(values)
+        for k, v in config.items('main'):
+            key = k.replace("-", "_")
+            value = v
+            if key.startswith("no_") and value is None:
+                key = key[3:]
+                value = False
+            setattr(namespace, key, value)
 
 def main():
 
@@ -835,24 +771,72 @@ def main():
 
     sysinfo = SystemInfo()
 
-    args = docopt.docopt(doc, version=VERSION)
-
-    if args['--config']:
-        ini_config = load_ini_config(args['--config'])
-        # merge config/args, priority given to args
-        args = merge(args, ini_config)
-
-    # load in defaults at lowest priority
-    args = merge(args, defaults)
+    parser = argparse.ArgumentParser(description="Buildroot autobuilder",
+                                     epilog=epilog,
+                                     formatter_class=Formatter)
+    parser.add_argument("--ninstances", "-n",
+                        help="number of parallel instances",
+                        type=int, default=1)
+    parser.add_argument("--njobs", "-j",
+                        help="number of parallel jobs",
+                        type=int, default=1)
+    parser.add_argument("--nice",
+                        help="Niceness, positive number",
+                        type=int, default=0)
+    parser.add_argument("--submitter", "-s",
+                        help="name/machine of submitter",
+                        type=str, default='N/A')
+    parser.add_argument("--http-url",
+                        help="URL of resource to submit your results.",
+                        type=str, default='http://autobuild.buildroot.org/')
+    parser.add_argument("--http-login",
+                        help="username to send results with",
+                        type=str)
+    parser.add_argument("--http-password",
+                        help="password to send results with (for security "
+                             "reasons it is recommended to define this in the "
+                             "config file instead, with user-read permissions "
+                             "only)",
+                        type=str)
+    parser.add_argument("--make-opts",
+                        help="string of extra options to pass to Buildroot "
+                             "make, such as specific command wrappers",
+                        type=str)
+    parser.add_argument("--pid-file",
+                        help="path to a file where to store the PID",
+                        type=pathlib.Path, default=pathlib.Path("/tmp/buildroot-autobuild.pid"))
+    parser.add_argument("--config", "-c",
+                        help="path to configuration file",
+                        type=open, action=LoadConfigFile)
+    parser.add_argument("--debug", "-d",
+                        help="Send log output to stdout instead of log file",
+                        type=str)
+    toolchains_csv = parser.add_mutually_exclusive_group(required=False)
+    toolchains_csv.add_argument("--toolchains-csv",
+                                dest="toolchains_csv",
+                                help="Toolchain configuration file",
+                                type=pathlib.Path,
+                                default=pathlib.Path("support/config-fragments/autobuild/toolchain-configs.csv"))
+    toolchains_csv.add_argument("--no-toolchains-csv",
+                                dest="toolchains_csv",
+                                help="Generate random toolchain configuration",
+                                action='store_false',
+                                default=argparse.SUPPRESS)
+    parser.add_argument("--repo", "-r",
+                        help="URL of Buildroot repository to clone",
+                        type=str,
+                        default="https://github.com/buildroot/buildroot.git")
+
+    args = parser.parse_args()
 
     # Save our PID very early, so we can be stopped
-    with open(args['--pid-file'], "w+") as pidf:
+    with args.pid_file.open("w+") as pidf:
         pidf.write("%d" % os.getpid())
 
     # http_login/password could theoretically be allowed as empty, so check
     # explicitly on None.
-    upload = (args['--http-login'] is not None) \
-             and (args['--http-password'] is not None)
+    upload = (args.http_login is not None) \
+             and (args.http_password is not None)
     if upload:
         sysinfo.needed_progs.append("curl")
     else:
@@ -894,24 +878,24 @@ def main():
 
         sys.exit(1)
 
-    buildpid = multiprocessing.Array('i', int(args['--ninstances']))
+    buildpid = multiprocessing.Array('i', int(args.ninstances))
     processes = []
-    for i in range(0, int(args['--ninstances'])):
+    for i in range(0, int(args.ninstances)):
         builder = Builder(
             instance = i,
-            njobs = args['--njobs'],
+            njobs = args.njobs,
             sysinfo = sysinfo,
-            http_url = args['--http-url'],
-            http_login = args['--http-login'],
-            http_password = args['--http-password'],
-            submitter = args['--submitter'],
-            make_opts = (args['--make-opts'] or ''),
-            nice = (args['--nice'] or 0),
-            toolchains_csv = args['--toolchains-csv'],
-            repo = args['--repo'],
+            http_url = args.http_url,
+            http_login = args.http_login,
+            http_password = args.http_password,
+            submitter = args.submitter,
+            make_opts = (args.make_opts or ''),
+            nice = (args.nice or 0),
+            toolchains_csv = args.toolchains_csv,
+            repo = args.repo,
             upload = upload,
             buildpid = buildpid,
-            debug = args['--debug'])
+            debug = args.debug)
         p = multiprocessing.Process(target=builder.run_instance)
         p.start()
         processes.append(p)
scripts/docopt.LICENSE-MIT deleted +0 -23
deleted file mode 100644
index 58ff1bc..0000000
@@ -1,23 +0,0 @@
-Copyright (c) 2012 Vladimir Keleshev, <[email protected]>
-
-Permission is hereby granted, free of charge, to any person
-obtaining a copy of this software and associated
-documentation files (the "Software"), to deal in the Software
-without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense,
-and/or sell copies of the Software, and to permit persons to
-whom the Software is furnished to do so, subject to the
-following conditions:
-
-The above copyright notice and this permission notice shall
-be included in all copies or substantial portions of the
-Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
-KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
-WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
-PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
-OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
scripts/docopt.py deleted +0 -581

581 lines not on this page · raw message

Signature · 6 lines
2.25.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Hello James,

Quoted 9 lines · On Wed, 13 Apr 2022 15:27:45 -0600 James Hilliard <[email protected]>…

Since docopt is unmaintained and less flexible than argparse lets drop it and migrate to argparse.

---
Changes v2 -> v3:
- suppress --no-toolchains-csv default output from help
Changes v1 -> v2:
- fix configparser

Quoted 7 lines
---
 scripts/autobuild-run      | 202 ++++++-------
 scripts/docopt.LICENSE-MIT |  23 --
 scripts/docopt.py          | 581 -------------------------------------
 3 files changed, 93 insertions(+), 713 deletions(-)
 delete mode 100644 scripts/docopt.LICENSE-MIT
 delete mode 100644 scripts/docopt.py

I have finally applied this patch, but after splitting it into two commits:

* One implementing the docopt -> argparse migration

* One implementing the --no-toolchains-csv option

Also, the --no-toolchains-csv option of utils/genrandconfig did not exist back in the 2022.02.x branch, so when autobuild-run is executed with --no-toolchains-csv, it never built any configuration on 2022.02.x, which is problematic.

I worked this around by doing a special case:

https://git.buildroot.org/buildroot-test/commit/?id=cae22243b75f7766095cbef196016b3b3143b677

I have now deployed the new script on two of the autobuilder instances that I manage (gcc159 and gcc160).

Could you on your side also update to use the latest version of autobuild-run, so that we are aligned?

Thanks a lot for this work, it has been extremely useful to find more problematic configurations!

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw
Quoted 26 lines · Thomas Petazzoni

Hello James,

Quoted 9 lines · On Wed, 13 Apr 2022 15:27:45 -0600 James Hilliard <[email protected]>…

Since docopt is unmaintained and less flexible than argparse lets drop it and migrate to argparse.

---
Changes v2 -> v3:
- suppress --no-toolchains-csv default output from help
Changes v1 -> v2:
- fix configparser

Quoted 7 lines
---
 scripts/autobuild-run      | 202 ++++++-------
 scripts/docopt.LICENSE-MIT |  23 --
 scripts/docopt.py          | 581 -------------------------------------
 3 files changed, 93 insertions(+), 713 deletions(-)
 delete mode 100644 scripts/docopt.LICENSE-MIT
 delete mode 100644 scripts/docopt.py

I have finally applied this patch, but after splitting it into two commits:

Quoted 8 lines

* One implementing the docopt -> argparse migration

* One implementing the --no-toolchains-csv option

Also, the --no-toolchains-csv option of utils/genrandconfig did not exist back in the 2022.02.x branch, so when autobuild-run is executed with --no-toolchains-csv, it never built any configuration on 2022.02.x, which is problematic.

I've been running autobuilders restricted to master at the moment since the other branches were rather broken anyways with full package randomization.

Quoted 10 lines

I worked this around by doing a special case:

https://git.buildroot.org/buildroot-test/commit/?id=cae22243b75f7766095cbef196016b3b3143b677

I have now deployed the new script on two of the autobuilder instances that I manage (gcc159 and gcc160).

Could you on your side also update to use the latest version of autobuild-run, so that we are aligned?

Yeah, I'll give that a try. May keep the master restriction in place for a little
since the full randomization triggered a lot of errors that weren't regressions so much as things that just didn't have good coverage.

Quoted 3 lines

Thanks a lot for this work, it has been extremely useful to find more problematic configurations!

I think this patch should make it easier to find problematic configurations as well, right now there seems to be a heavy bias towards selecting packages with very few dependencies:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

Quoted 8 lines

Best regards,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot