Skip to contentSkip to conversation

devtool fixes

PATCH New Paul Eggleton 1 participant 5 messages Aug 17, 2015

lore
Share & tools
lore
#0 lore raw

cover devtool fixes

Three bugfixes for devtool plus one refactoring patch to improve a related function in the recipeutils module.

The following changes since commit 059db140885bad379534e6ec713f3ceb4e18faea:

adt-installer: use DEPLOY_DIR in ANT_DEPLOY expansion (2015-08-16 17:28:14 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/devtool-fixes5
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/devtool-fixes5

Paul Eggleton (4):
devtool: extract: prevent externalsrc from interfering with extraction
devtool: extract: remove patches subdirectory when S == WORKDIR
devtool: fix handling of BBCLASSEXTENDed recipes lib/oe/recipeutils: avoid parsing in get_var_files()

 meta/lib/oe/recipeutils.py        |  3 +--
 meta/lib/oeqa/selftest/devtool.py | 30 ++++++++++++++++++++++++++++++
 scripts/devtool                   | 12 ++++++------
 scripts/lib/devtool/standard.py   | 29 +++++++++++++++++------------
 4 files changed, 54 insertions(+), 20 deletions(-)
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/4 devtool: extract: prevent externalsrc from interfering with extraction

In case the user has set up externalsrc outside of devtool, force EXTERNALSRC to blank for the recipe when extracting so that the original source URI is still in SRC_URI and we're still able to extract it. (This isn't a problem with devtool itself because the bbappends within the workspace layer that apply externalsrc are explicitly filtered out when devtool parses a recipe).

---
 scripts/lib/devtool/standard.py | 2 ++
 1 file changed, 2 insertions(+)
scripts/lib/devtool/standard.py +2 -0
index ea21877..e4ee7f7 100644
@@ -294,6 +294,8 @@ def _extract_source(srctree, keep_temp, devbranch, d):
 
         task_executor = BbTaskExecutor(crd)
 
+        crd.setVar('EXTERNALSRC_forcevariable', '')
+
         logger.info('Fetching %s...' % pn)
         task_executor.exec_func('do_fetch', False)
         logger.info('Unpacking...')
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/4 devtool: extract: remove patches subdirectory when S == WORKDIR

In reply to Paul Eggleton

Ensure that the "patches" subdirectory is removed from the right location when S == WORKDIR (e.g. devtool extract makedevs).

---
 scripts/lib/devtool/standard.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
scripts/lib/devtool/standard.py +1 -2
index e4ee7f7..658076c 100644
@@ -301,7 +301,6 @@ def _extract_source(srctree, keep_temp, devbranch, d):
         logger.info('Unpacking...')
         task_executor.exec_func('do_unpack', False)
         srcsubdir = crd.getVar('S', True)
-        patchsubdir = srcsubdir
         if srcsubdir == workdir:
             # Find non-patch sources that were "unpacked" to srctree directory
             recipe_patches = [os.path.basename(patch) for patch in
@@ -322,7 +321,7 @@ def _extract_source(srctree, keep_temp, devbranch, d):
 
         scriptutils.git_convert_standalone_clone(srcsubdir)
 
-        patchdir = os.path.join(patchsubdir, 'patches')
+        patchdir = os.path.join(srcsubdir, 'patches')
         haspatches = False
         if os.path.exists(patchdir):
             if os.listdir(patchdir):
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/4 devtool: fix handling of BBCLASSEXTENDed recipes

In reply to Paul Eggleton

If a recipe is BBCLASSEXTENDed (e.g. to -native), its PN value and the name of the bbappend will be different; we were assuming them to be the same when reading in the workspace, leading to us seeing the base recipe name everywhere afterwards.

Also add a test so we ensure this doesn't regress in future.

Fixes [YOCTO #8157].

---
 meta/lib/oeqa/selftest/devtool.py | 30 ++++++++++++++++++++++++++++++
 scripts/devtool                   | 12 ++++++------
 scripts/lib/devtool/standard.py   | 20 ++++++++++++--------
 3 files changed, 48 insertions(+), 14 deletions(-)
meta/lib/oeqa/selftest/devtool.py +30 -0
index b59db15..947d8ee 100644
@@ -367,6 +367,36 @@ class DevtoolTests(DevtoolBase):
             self.assertNotEqual(result.status, 0, 'devtool modify on %s should have failed. devtool output: %s' %  (testrecipe, result.output))
             self.assertIn('ERROR: ', result.output, 'devtool modify on %s should have given an ERROR' % testrecipe)
 
+    def test_devtool_modify_native(self):
+        # Check preconditions
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        # Try modifying some recipes
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+
+        bbclassextended = False
+        inheritnative = False
+        testrecipes = 'mtools-native apt-native desktop-file-utils-native'.split()
+        for testrecipe in testrecipes:
+            checkextend = 'native' in (get_bb_var('BBCLASSEXTEND', testrecipe) or '').split()
+            if not bbclassextended:
+                bbclassextended = checkextend
+            if not inheritnative:
+                inheritnative = not checkextend
+            result = runCmd('devtool modify %s -x %s' % (testrecipe, os.path.join(tempdir, testrecipe)))
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool modify output: %s' % result.output)
+            result = runCmd('devtool build %s' % testrecipe)
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool build output: %s' % result.output)
+            result = runCmd('devtool reset %s' % testrecipe)
+            self.assertNotIn('ERROR: ', result.output, 'ERROR in devtool reset output: %s' % result.output)
+
+        self.assertTrue(bbclassextended, 'None of these recipes are BBCLASSEXTENDed to native - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
+        self.assertTrue(inheritnative, 'None of these recipes do "inherit native" - need to adjust testrecipes list: %s' % ', '.join(testrecipes))
+
+
     @testcase(1165)
     def test_devtool_modify_git(self):
         # Check preconditions
scripts/devtool +6 -6
index 1c22438..b9d3bb9 100755
@@ -104,15 +104,15 @@ def read_workspace():
         _enable_workspace_layer(config.workspace_path, config, basepath)
 
     logger.debug('Reading workspace in %s' % config.workspace_path)
-    externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-[^ =]+)? =.*$')
+    externalsrc_re = re.compile(r'^EXTERNALSRC(_pn-([^ =]+))? *= *"([^"]*)"$')
     for fn in glob.glob(os.path.join(config.workspace_path, 'appends', '*.bbappend')):
-        pn = os.path.splitext(os.path.basename(fn))[0].split('_')[0]
         with open(fn, 'r') as f:
             for line in f:
-                if externalsrc_re.match(line.rstrip()):
-                    splitval = line.split('=', 2)
-                    workspace[pn] = splitval[1].strip('" \n\r\t')
-                    break
+                res = externalsrc_re.match(line.rstrip())
+                if res:
+                    pn = res.group(2) or os.path.splitext(os.path.basename(fn))[0].split('_')[0]
+                    workspace[pn] = {'srctree': res.group(3),
+                                     'bbappend': fn}
 
 def create_workspace(args, config, basepath, workspace):
     if args.layerpath:
scripts/lib/devtool/standard.py +12 -8
index 658076c..e85e1ad 100644
@@ -430,6 +430,16 @@ def modify(args, config, basepath, workspace):
     if not rd:
         return 1
     recipefile = rd.getVar('FILE', True)
+    appendname = os.path.splitext(os.path.basename(recipefile))[0]
+    if args.wildcard:
+        appendname = re.sub(r'_.*', '_%', appendname)
+    appendpath = os.path.join(config.workspace_path, 'appends')
+    appendfile = os.path.join(appendpath, appendname + '.bbappend')
+    if os.path.exists(appendfile):
+        raise DevtoolError("Another variant of recipe %s is already in your "
+                           "workspace (only one variant of a recipe can "
+                           "currently be worked on at once)"
+                           % args.recipename)
 
     _check_compatible_recipe(args.recipename, rd)
 
@@ -467,14 +477,8 @@ def modify(args, config, basepath, workspace):
         srcsubdir = os.path.relpath(s, workdir).split(os.sep, 1)[1]
         srctree = os.path.join(srctree, srcsubdir)
 
-    appendpath = os.path.join(config.workspace_path, 'appends')
     if not os.path.exists(appendpath):
         os.makedirs(appendpath)
-
-    appendname = os.path.splitext(os.path.basename(recipefile))[0]
-    if args.wildcard:
-        appendname = re.sub(r'_.*', '_%', appendname)
-    appendfile = os.path.join(appendpath, appendname + '.bbappend')
     with open(appendfile, 'w') as f:
         f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
         f.write('inherit externalsrc\n')
@@ -777,7 +781,7 @@ def update_recipe(args, config, basepath, workspace):
     else:
         mode = args.mode
 
-    srctree = workspace[args.recipename]
+    srctree = workspace[args.recipename]['srctree']
 
     if mode == 'srcrev':
         _update_recipe_srcrev(args, srctree, rd, tinfoil.config_data)
@@ -793,7 +797,7 @@ def status(args, config, basepath, workspace):
     """Entry point for the devtool 'status' subcommand"""
     if workspace:
         for recipe, value in workspace.iteritems():
-            print("%s: %s" % (recipe, value))
+            print("%s: %s" % (recipe, value['srctree']))
     else:
         logger.info('No recipes currently in your workspace - you can use "devtool modify" to work on an existing recipe or "devtool add" to add a new one')
     return 0
Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/4 lib/oe/recipeutils: avoid parsing in get_var_files()

In reply to Paul Eggleton

Let's have the caller do this and then the function is a bit more flexible (e.g. we can choose to parse with bbappends or not); fix up calls to this function appropriately (of which there are only two, both within devtool).

---
 meta/lib/oe/recipeutils.py      | 3 +--
 scripts/lib/devtool/standard.py | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)
meta/lib/oe/recipeutils.py +1 -2
index cd74213..d4fa726 100644
@@ -79,10 +79,9 @@ def get_var_files(fn, varlist, d):
     """Find the file in which each of a list of variables is set.
     Note: requires variable history to be enabled when parsing.
     """
-    envdata = parse_recipe(fn, [], d)
     varfiles = {}
     for v in varlist:
-        history = envdata.varhistory.variable(v)
+        history = d.varhistory.variable(v)
         files = []
         for event in history:
             if 'file' in event and not 'flag' in event:
scripts/lib/devtool/standard.py +2 -2
index e85e1ad..e1c5584 100644
@@ -630,7 +630,7 @@ def _update_recipe_srcrev(args, srctree, rd, config_data):
                 rd, args.append, None, wildcardver=args.wildcard_version,
                 extralines=patchfields)
     else:
-        oe.recipeutils.patch_recipe(config_data, recipefile, patchfields)
+        oe.recipeutils.patch_recipe(rd, recipefile, patchfields)
 
     if not 'git://' in orig_src_uri:
         logger.info('You will need to update SRC_URI within the recipe to '
@@ -742,7 +742,7 @@ def _update_recipe_patch(args, config, srctree, rd, config_data):
                     updaterecipe = True
             if updaterecipe:
                 logger.info('Updating recipe %s' % os.path.basename(recipefile))
-                oe.recipeutils.patch_recipe(config_data, recipefile,
+                oe.recipeutils.patch_recipe(rd, recipefile,
                                             {'SRC_URI': ' '.join(srcuri)})
             elif not updatepatches:
                 # Neither patches nor recipe were updated
Signature · 1 line
2.1.0