devtool improvements

When you need to set EXTRA_OECONF for a recipe, you need to know what options the configure script actually supports; the configure script however is only accessible from within a devshell and (at least in the case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

There is basic support for autotools, cmake and bare configure scripts. The cmake support is a little hacky since cmake doesn't really have a concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at the moment.

Signed-off-by Paul Eggleton <[email protected]>

---
scripts/lib/devtool/__init__.py | 9 ++-
scripts/lib/devtool/utilcmds.py | 170 ++++++++++++++++++++++++++++++++++++++++

2 files changed, 175 insertions(+), 4 deletions(-)

scripts/lib/devtool/__init__.py +5 -4

index 0405d22..ff97dfc 100644
@@ -129,7 +129,7 @@ def get_recipe_file(cooker, pn):
            logger.error("Unable to find any recipe file matching %s" % pn)
    return recipefile
 
-def parse_recipe(config, tinfoil, pn, appends):
+def parse_recipe(config, tinfoil, pn, appends, filter_workspace=True):
    """Parse recipe of a package"""
    import oe.recipeutils
    recipefile = get_recipe_file(tinfoil.cooker, pn)

@@ -138,9 +138,10 @@ def parse_recipe(config, tinfoil, pn, appends):
        return None
    if appends:
        append_files = tinfoil.cooker.collection.get_file_appends(recipefile)
-        # Filter out appends from the workspace
-        append_files = [path for path in append_files if
-                        not path.startswith(config.workspace_path)]
+        if filter_workspace:
+            # Filter out appends from the workspace
+            append_files = [path for path in append_files if
+                            not path.startswith(config.workspace_path)]
    else:
        append_files = None
    return oe.recipeutils.parse_recipe(recipefile, append_files,

scripts/lib/devtool/utilcmds.py +159 -0

index 375d7a3..a8f5e97 100644
@@ -61,6 +61,165 @@ def edit_recipe(args, config, basepath, workspace):
    return 0
 
 
+def configure_help(args, config, basepath, workspace):
+    """Entry point for the devtool 'configure-help' subcommand"""
+    import oe.utils
+
+    check_workspace_recipe(workspace, args.recipename)
+    tinfoil = setup_tinfoil(config_only=False, basepath=basepath)
+    try:
+        rd = parse_recipe(config, tinfoil, args.recipename, appends=True, filter_workspace=False)
+        if not rd:
+            return 1
+        b = rd.getVar('B', True)
+        s = rd.getVar('S', True)
+        configurescript = os.path.join(s, 'configure')
+        confdisabled = 'noexec' in rd.getVarFlags('do_configure') or 'do_configure' not in (rd.getVar('__BBTASKS', False) or [])
+        configureopts = oe.utils.squashspaces(rd.getVar('CONFIGUREOPTS', True) or '')
+        extra_oeconf = oe.utils.squashspaces(rd.getVar('EXTRA_OECONF', True) or '')
+        extra_oecmake = oe.utils.squashspaces(rd.getVar('EXTRA_OECMAKE', True) or '')
+        do_configure = rd.getVar('do_configure', True) or ''
+        do_configure_noexpand = rd.getVar('do_configure', False) or ''
+        packageconfig = rd.getVarFlags('PACKAGECONFIG') or []
+        autotools = bb.data.inherits_class('autotools', rd) and ('oe_runconf' in do_configure or 'autotools_do_configure' in do_configure)
+        cmake = bb.data.inherits_class('cmake', rd) and ('cmake_do_configure' in do_configure)
+        cmake_do_configure = rd.getVar('cmake_do_configure', True)
+        pn = rd.getVar('PN', True)
+    finally:
+        tinfoil.shutdown()
+
+    if 'doc' in packageconfig:
+        del packageconfig['doc']
+
+    if autotools and not os.path.exists(configurescript):
+        logger.info('Running do_configure to generate configure script')
+        try:
+            stdout, _ = exec_build_env_command(config.init_path, basepath,
+                                               'bitbake -c configure %s' % msg, args.recipename,
+                                               stderr=subprocess.STDOUT)
+        except bb.process.ExecutionError:
+            pass
+
+    if confdisabled or do_configure.strip() in ('', ':'):
+        raise DevtoolError("do_configure task has been disabled for this recipe")
+    elif args.no_pager and not os.path.exists(configurescript):
+        raise DevtoolError("No configure script found and no other information to display")
+    else:
+        configopttext = ''
+        if autotools and configureopts:
+            configopttext = '''
+Arguments currently passed to the configure script:
+
+%s
+
+Some of those are fixed.''' % (configureopts + ' ' + extra_oeconf)
+            if extra_oeconf:
+                configopttext += ''' The ones that are specified through EXTRA_OECONF (which you can change or add to easily):
+
+%s''' % extra_oeconf
+
+        elif cmake:
+            in_cmake = False
+            cmake_cmd = ''
+            for line in cmake_do_configure.splitlines():
+                if in_cmake:
+                    cmake_cmd = cmake_cmd + ' ' + line.strip().rstrip('\\')
+                    if not line.endswith('\\'):
+                        break
+                if line.lstrip().startswith('cmake '):
+                    cmake_cmd = line.strip().rstrip('\\')
+                    if line.endswith('\\'):
+                        in_cmake = True
+                    else:
+                        break
+            if cmake_cmd:
+                configopttext = '''
+The current cmake command line:
+
+%s
+
+Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
+
+%s''' % (oe.utils.squashspaces(cmake_cmd), extra_oecmake)
+            else:
+                configopttext = '''
+The current implementation of cmake_do_configure:
+
+cmake_do_configure() {
+%s
+}
+
+Arguments specified through EXTRA_OECMAKE (which you can change or add to easily)
+
+%s''' % (cmake_do_configure.rstrip(), extra_oecmake)
+
+        elif do_configure:
+            configopttext = '''
+The current implementation of do_configure:
+
+do_configure() {
+%s
+}''' % do_configure.rstrip()
+            if '${EXTRA_OECONF}' in do_configure_noexpand:
+                configopttext += '''
+
+Arguments specified through EXTRA_OECONF (which you can change or add to easily):
+
+%s''' % extra_oeconf
+
+        if packageconfig:
+            configopttext += '''
+
+Some of these options may be controlled through PACKAGECONFIG; for more details please see the recipe.'''
+
+        if args.arg:
+            helpargs = ' '.join(args.arg)
+        elif cmake:
+            helpargs = '-LH'
+        else:
+            helpargs = '--help'
+
+        msg = '''configure information for %s
+------------------------------------------
+%s''' % (pn, configopttext)
+
+        if cmake:
+            msg += '''
+
+The cmake %s output for %s follows. After "-- Cache values" you should see a list of variables you can add to EXTRA_OECMAKE (prefixed with -D and suffixed with = followed by the desired value, without any spaces).
+------------------------------------------''' % (helpargs, pn)
+        elif os.path.exists(configurescript):
+            msg += '''
+
+The ./configure %s output for %s follows.
+------------------------------------------''' % (helpargs, pn)
+
+        olddir = os.getcwd()
+        tmppath = tempfile.mkdtemp()
+        with tempfile.NamedTemporaryFile('w', delete=False) as tf:
+            if not args.no_header:
+                tf.write(msg + '\n')
+            tf.close()
+            try:
+                try:
+                    cmd = 'cat %s' % tf.name
+                    if cmake:
+                        cmd += '; cmake %s %s 2>&1' % (helpargs, s)
+                        os.chdir(b)
+                    elif os.path.exists(configurescript):
+                        cmd += '; %s %s' % (configurescript, helpargs)
+                    if sys.stdout.isatty() and not args.no_pager:
+                        pager = os.environ.get('PAGER', 'less')
+                        cmd = '(%s) | %s' % (cmd, pager)
+                    subprocess.check_call(cmd, shell=True)
+                except subprocess.CalledProcessError as e:
+                    return e.returncode
+            finally:
+                os.chdir(olddir)
+                shutil.rmtree(tmppath)
+                os.remove(tf.name)
+
+

def register_commands(subparsers, context):
"""Register devtool subcommands from this plugin"""
parser_edit_recipe = subparsers.add_parser('edit-recipe', help='Edit a recipe file in your workspace',

scripts/lib/devtool/utilcmds.py +11 -1

@@ -68,3 +227,14 @@ def register_commands(subparsers, context):
    parser_edit_recipe.add_argument('recipename', help='Recipe to edit')
    parser_edit_recipe.add_argument('--any-recipe', '-a', action="store_true", help='Edit any recipe, not just where the recipe file itself is in the workspace')
    parser_edit_recipe.set_defaults(func=edit_recipe)
+
+    # NOTE: Needed to override the usage string here since the default
+    # gets the order wrong - recipename must come before --arg
+    parser_configure_help = subparsers.add_parser('configure-help', help='Get help on configure script options',
+                                         usage='devtool configure-help [options] recipename [--arg ...]',
+                                         description='Displays the help for the configure script for the specified recipe (i.e. runs ./configure --help) prefaced by a header describing the current options being specified. Output is piped through less (or whatever PAGER is set to, if set) for easy browsing.')
+    parser_configure_help.add_argument('recipename', help='Recipe to show configure help for')
+    parser_configure_help.add_argument('-p', '--no-pager', help='Disable paged output', action="store_true")
+    parser_configure_help.add_argument('-n', '--no-header', help='Disable explanatory header text', action="store_true")
+    parser_configure_help.add_argument('--arg', help='Pass remaining arguments to the configure script instead of --help (useful if the script has additional help options)', nargs=argparse.REMAINDER)
+    parser_configure_help.set_defaults(func=configure_help)
--

2.5.0

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Quoted 12 lines

On Jan 25, 2016, at 9:53 PM, Paul Eggleton <[email protected]> wrote:

When you need to set EXTRA_OECONF for a recipe, you need to know what options the configure script actually supports; the configure script however is only accessible from within a devshell and (at least in the case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

Quoted 6 lines

There is basic support for autotools, cmake and bare configure scripts. The cmake support is a little hacky since cmake doesn't really have a concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at the moment.

this option is autotools specific. We need to convey this precisely. may be involve autotools in the option name or something

Hi Khem,

Quoted 20 lines · On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:

Quoted 19 lines

On Jan 25, 2016, at 9:53 PM, Paul Eggleton
<[email protected]>
wrote:

When you need to set EXTRA_OECONF for a recipe, you need to know what options the configure script actually supports; the configure script however is only accessible from within a devshell and (at least in the case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

There is basic support for autotools, cmake and bare configure scripts. The cmake support is a little hacky since cmake doesn't really have a concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at the moment.

Quoted 3 lines

this option is autotools specific. We need to convey this precisely. may be
involve autotools in the option name or something

Whilst not all non-autoconf configure scripts provide a --help option, some do. ffmpeg/libav is one example (and one where you really do need to see the output since there are a lot of options...)

Quoted 27 lines · On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:

Hi Khem,

Quoted 23 lines · On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:

Quoted 13 lines

On Jan 25, 2016, at 9:53 PM, Paul Eggleton
<[email protected]>
wrote:

When you need to set EXTRA_OECONF for a recipe, you need to know what options the configure script actually supports; the configure script however is only accessible from within a devshell and (at least in the case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

Quoted 6 lines

There is basic support for autotools, cmake and bare configure scripts. The cmake support is a little hacky since cmake doesn't really have a concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at the moment.

this option is autotools specific. We need to convey this precisely. may be
involve autotools in the option name or something

Quoted 3 lines

Whilst not all non-autoconf configure scripts provide a --help option, some do. ffmpeg/libav is one example (and one where you really do need to see the output since there are a lot of options...)

I meant to add, even if configure --help doesn't work (or there is no configure script at all) the header this subcommand prints will work with any recipe provided that do_configure is actually implemented and contains some actual commands as opposed to calls to some other function.

Cheers,
Paul

--

Paul Eggleton
Intel Open Source Technology Centre

On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton

Quoted 37 lines · <[email protected]> wrote:

Quoted 35 lines · On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:

Hi Khem,

Quoted 23 lines · On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:

Quoted 15 lines

On Jan 25, 2016, at 9:53 PM, Paul Eggleton
<[email protected]>
wrote:

When you need to set EXTRA_OECONF for a recipe, you need to know what options the configure script actually supports; the configure script however is only accessible from within a devshell and (at least in the
case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe
with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

Quoted 7 lines

There is basic support for autotools, cmake and bare configure scripts.
The cmake support is a little hacky since cmake doesn't really have a concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at the moment.

Quoted 4 lines

this option is autotools specific. We need to convey this precisely. may
be
involve autotools in the option name or something

Whilst not all non-autoconf configure scripts provide a --help option, some
do. ffmpeg/libav is one example (and one where you really do need to see the output since there are a lot of options...)

Quoted 4 lines

I meant to add, even if configure --help doesn't work (or there is no configure script at all) the header this subcommand prints will work with any recipe provided that do_configure is actually implemented and contains some actual commands as opposed to calls to some other function.

my worry is it being associated to do_configure, and we know that only a subset ( although a large one) is only using autotools but not all. So think about avoiding that confusion.

Quoted 53 lines · On Mon, 25 Jan 2016 22:27:27 Khem Raj wrote:

On Mon, Jan 25, 2016 at 10:21 PM, Paul Eggleton

Quoted 43 lines · <[email protected]> wrote:

Quoted 36 lines · On Tue, 26 Jan 2016 16:17:39 Paul Eggleton wrote:

Hi Khem,

Quoted 28 lines · On Mon, 25 Jan 2016 22:14:09 Khem Raj wrote:

Quoted 18 lines

On Jan 25, 2016, at 9:53 PM, Paul Eggleton
<[email protected]>
wrote:

When you need to set EXTRA_OECONF for a recipe, you need to know what
options the configure script actually supports; the configure script
however is only accessible from within a devshell and (at least in the
case of autotooled software fetched from an SCM repository) may not actually exist until do_configure has run. Thus, provide a "devtool configure-help" subcommand that runs the configure script for a recipe
with --help and shows you the output through a pager (e.g. less), prefaced by a header describing the current options being specified.

Quoted 9 lines

There is basic support for autotools, cmake and bare configure scripts.
The cmake support is a little hacky since cmake doesn't really have a
concise help option that lists user-defined knobs (without actually running through the configure process), however that being a design feature of cmake there's not much I can think of to do about that at
the moment.

Quoted 4 lines

this option is autotools specific. We need to convey this precisely. may
be
involve autotools in the option name or something

Quoted 5 lines

Whilst not all non-autoconf configure scripts provide a --help option, some
do. ffmpeg/libav is one example (and one where you really do need to see
the output since there are a lot of options...)

Quoted 6 lines

I meant to add, even if configure --help doesn't work (or there is no configure script at all) the header this subcommand prints will work with
any recipe provided that do_configure is actually implemented and contains some actual commands as opposed to calls to some other function.

Quoted 3 lines

my worry is it being associated to do_configure, and we know that only a subset ( although a large one) is only using autotools but not all. So think about avoiding that confusion.

I'm not terribly convinced there could be much confusion. FWIW, I just looked through the other recipes we have in OE-Core that have a call to a configure script that don't use autoconf (man, zlib, x264, v86d, ed, libpostproc); the configure scripts for all of these at least produce some useful output when called with --help. I then took a look at meta-oe (the layer)

, and checked the following recipes:

* concurrencykit - works
* ddrescue - works
* fio - works
* libvpx - works
* lzip - works
* nodejs - works
* mg - works (no options, but message is sensible)
* terminus-font - configure script is supplied without the execute bit set but does support --help
* iscsi-initiator-utils - configure script is in a subdir so isn't found, but does support --help
* netkit-rsh - works
* netkit-rpc - works
* netkit-telnet - works
* netkit-tftp - works
* netkit-rusers - works
* netkit-rwho - works
* netmap - configure script is in a subdir so isn't found, but does support --help
* ntimed - configure script is dumb and takes no options at all
* tvheadend - works
* mozjs - seems to break devtool modify, but configure script does support --help

So there are some minor bugs to fix, but of those recipes I tested there is a grand total of 1 script that doesn't support --help. Honestly I think if a configure script exists it's reasonable to assume that
--help is going to print something useful; even if it doesn't there shouldn't be any major harm and you're still going to get a printout of the do_configure value in which you'll be able to see what options are currently passed.

Cheers,
Paul