Skip to contentSkip to conversation

devtool: improve handling of local source files

PATCH Superseded Markus Lehtonen 2 participants 10 messages Jun 3, 2015 – Jun 4, 2015

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover devtool: improve handling of local source files

Second version of my patchset aiming to improve handling of local source files. The only significant change since the first version is the addition of unit tests.

The following changes since commit 968973d55d4b33e1a929ed4cdf9387fcaba2d93f:

qt4: unconditionally disable gstreamer 0.10 support in qt webkit (2015-05-30 22:25:12 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib marquiz/devtool/localfiles
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=marquiz/devtool/localfiles

Markus Lehtonen (7):
recipeutils: implement get_recipe_local_files() oe.patch.GitApplyTree: add paths argument to extractPatches
oe-selftest: devtool: add method for checking workspace dir
oe-selftest: devtool: add method for checking srctree repo
devtool: update-recipe: update local files directly
devtool: extract: always import local files to srctree
devtool: modify: make bitbake use local files from srctree

 meta/lib/oe/patch.py              |   5 +-
 meta/lib/oe/recipeutils.py        |   8 ++
 meta/lib/oeqa/selftest/devtool.py | 204 +++++++++++++++++++++++++-------------
 scripts/lib/devtool/standard.py   |  93 +++++++++++++++--
 4 files changed, 231 insertions(+), 79 deletions(-)
Signature · 1 line
2.1.4
#1 lore raw

PATCH v2 1/7 recipeutils: implement get_recipe_local_files()

---
 meta/lib/oe/recipeutils.py | 8 ++++++++
 1 file changed, 8 insertions(+)
meta/lib/oe/recipeutils.py +8 -0
index f05b6c0..4bd5ce7 100644
@@ -278,6 +278,14 @@ def copy_recipe_files(d, tgt_dir, whole_dir=False, download=True):
     return remotes
 
 
+def get_recipe_local_files(d):
+    """Get a list of local files in SRC_URI within a recipe."""
+    uris = (d.getVar('SRC_URI', True) or "").split()
+    fetch = bb.fetch2.Fetch(uris, d)
+    return dict([(fetch.ud[uri].basepath, fetch.localpath(uri)) for uri in uris
+                 if uri.startswith('file://')])
+
+
 def get_recipe_patches(d):
     """Get a list of the patches included in SRC_URI within a recipe."""
     patchfiles = []
Signature · 1 line
2.1.4
#2 lore raw

PATCH v2 2/7 oe.patch.GitApplyTree: add paths argument to extractPatches

In reply to Markus Lehtonen

Makes it possible to define which paths are included in the patches.

---
 meta/lib/oe/patch.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oe/patch.py +4 -1
index afb0013..6da2511 100644
@@ -337,12 +337,15 @@ class GitApplyTree(PatchTree):
         return (tmpfile, cmd)
 
     @staticmethod
-    def extractPatches(tree, startcommit, outdir):
+    def extractPatches(tree, startcommit, outdir, paths=None):
         import tempfile
         import shutil
         tempdir = tempfile.mkdtemp(prefix='oepatch')
         try:
             shellcmd = ["git", "format-patch", startcommit, "-o", tempdir]
+            if paths:
+                shellcmd.append('--')
+                shellcmd.extend(paths)
             out = runcmd(["sh", "-c", " ".join(shellcmd)], tree)
             if out:
                 for srcfile in out.split():
Signature · 1 line
2.1.4
#3 lore raw

PATCH v2 3/7 oe-selftest: devtool: add method for checking workspace dir

In reply to Markus Lehtonen

In order to remove some code duplication.

---
 meta/lib/oeqa/selftest/devtool.py | 64 +++++++++++++++------------------------
 1 file changed, 24 insertions(+), 40 deletions(-)
meta/lib/oeqa/selftest/devtool.py +24 -40
index 4e22e1d..c01e361 100644
@@ -63,10 +63,17 @@ class DevtoolBase(oeSelfTest):
 
 class DevtoolTests(DevtoolBase):
 
+    def _get_workspace_dir(self):
+        """Get workspace directory"""
+        workspacedir = os.path.join(self.builddir, 'workspace')
+        self.assertTrue(not os.path.exists(workspacedir),
+                        'This test cannot be run with a workspace directory '
+                        'under the build directory')
+        return workspacedir
+
     def test_create_workspace(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         result = runCmd('bitbake-layers show-layers')
         self.assertTrue('/workspace' not in result.output, 'This test cannot be run with a workspace layer in bblayers.conf')
         # Try creating a workspace layer with a specific path
@@ -86,9 +93,7 @@ class DevtoolTests(DevtoolBase):
         self.assertIn(workspacedir, result.output)
 
     def test_devtool_add(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
@@ -120,9 +125,7 @@ class DevtoolTests(DevtoolBase):
         self.assertTrue(os.path.isfile(os.path.join(installdir, bindir, 'pv')), 'pv binary not found in D')
 
     def test_devtool_add_library(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # We don't have the ability to pick up this dependency automatically yet...
         bitbake('libusb1')
         # Fetch source
@@ -160,9 +163,7 @@ class DevtoolTests(DevtoolBase):
         self.assertFalse(os.path.isfile(os.path.join(staging_libdir, 'libftdi1.so.2.1.0')), 'libftdi binary still found in STAGING_LIBDIR after cleaning')
 
     def test_devtool_add_fetch(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
@@ -206,9 +207,7 @@ class DevtoolTests(DevtoolBase):
         self._test_recipe_contents(recipefile, checkvars, [])
 
     def test_devtool_add_fetch_git(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # Fetch source
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
@@ -257,9 +256,7 @@ class DevtoolTests(DevtoolBase):
         self._test_recipe_contents(recipefile, checkvars, [])
 
     def test_devtool_modify(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # Clean up anything in the workdir/sysroot/sstate cache
         bitbake('mdadm -c cleansstate')
         # Try modifying a recipe
@@ -308,9 +305,7 @@ class DevtoolTests(DevtoolBase):
         self.assertFalse(matches, 'Stamp files exist for recipe mdadm that should have been cleaned')
 
     def test_devtool_modify_invalid(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         # Try modifying some recipes
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
@@ -341,8 +336,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_modify_git(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'mkelfimage'
         src_uri = get_bb_var('SRC_URI', testrecipe)
         self.assertIn('git://', src_uri, 'This test expects the %s recipe to be a git recipe' % testrecipe)
@@ -374,8 +368,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_modify_localfiles(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'lighttpd'
         src_uri = (get_bb_var('SRC_URI', testrecipe) or '').split()
         foundlocal = False
@@ -406,8 +399,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_update_recipe(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'minicom'
         recipefile = get_bb_var('FILE', testrecipe)
         src_uri = get_bb_var('SRC_URI', testrecipe)
@@ -452,8 +444,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_update_recipe_git(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'mtd-utils'
         recipefile = get_bb_var('FILE', testrecipe)
         src_uri = get_bb_var('SRC_URI', testrecipe)
@@ -526,8 +517,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_update_recipe_append(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'mdadm'
         recipefile = get_bb_var('FILE', testrecipe)
         src_uri = get_bb_var('SRC_URI', testrecipe)
@@ -601,8 +591,7 @@ class DevtoolTests(DevtoolBase):
 
     def test_devtool_update_recipe_append_git(self):
         # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         testrecipe = 'mtd-utils'
         recipefile = get_bb_var('FILE', testrecipe)
         src_uri = get_bb_var('SRC_URI', testrecipe)
@@ -696,9 +685,7 @@ class DevtoolTests(DevtoolBase):
         # Deleting isn't expected to work under these circumstances
 
     def test_devtool_extract(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         # Try devtool extract
         self.track_for_cleanup(tempdir)
@@ -709,9 +696,7 @@ class DevtoolTests(DevtoolBase):
         self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
 
     def test_devtool_reset_all(self):
-        # Check preconditions
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
         self.track_for_cleanup(tempdir)
         self.track_for_cleanup(workspacedir)
@@ -760,8 +745,7 @@ class DevtoolTests(DevtoolBase):
                 break
         else:
             self.skipTest('No tap devices found - you must set up tap devices with scripts/runqemu-gen-tapdevs before running this test')
-        workspacedir = os.path.join(self.builddir, 'workspace')
-        self.assertTrue(not os.path.exists(workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        workspacedir = self._get_workspace_dir()
         import pexpect
         # Definitions
         testrecipe = 'mdadm'
Signature · 1 line
2.1.4
#4 lore raw

PATCH v2 4/7 oe-selftest: devtool: add method for checking srctree repo

In reply to Markus Lehtonen

Removes some code duplication.

---
 meta/lib/oeqa/selftest/devtool.py | 50 +++++++++++++++------------------------
 1 file changed, 19 insertions(+), 31 deletions(-)
meta/lib/oeqa/selftest/devtool.py +19 -31
index c01e361..08ed2eb 100644
@@ -71,6 +71,18 @@ class DevtoolTests(DevtoolBase):
                         'under the build directory')
         return workspacedir
 
+    def _check_src_repo(self, repo_dir):
+        """Check srctree git repository"""
+        self.assertTrue(os.path.isdir(os.path.join(repo_dir, '.git')),
+                        'git repository for external source tree not found')
+        result = runCmd('git status --porcelain', cwd=repo_dir)
+        self.assertEqual(result.output.strip(), "",
+                         'Created git repo is not clean')
+        result = runCmd('git symbolic-ref HEAD', cwd=repo_dir)
+        self.assertEqual(result.output.strip(), "refs/heads/devtool",
+                         'Wrong branch in git repo')
+
+
     def test_create_workspace(self):
         # Check preconditions
         workspacedir = self._get_workspace_dir()
@@ -267,7 +279,6 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake -c clean mdadm')
         result = runCmd('devtool modify mdadm -x %s' % tempdir)
         self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile')), 'Extracted source could not be found')
-        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
         self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
         matches = glob.glob(os.path.join(workspacedir, 'appends', 'mdadm_*.bbappend'))
         self.assertTrue(matches, 'bbappend not created')
@@ -276,10 +287,7 @@ class DevtoolTests(DevtoolBase):
         self.assertIn('mdadm', result.output)
         self.assertIn(tempdir, result.output)
         # Check git repo
-        result = runCmd('git status --porcelain', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Try building
         bitbake('mdadm')
         # Try making (minor) modifications to the source
@@ -350,7 +358,6 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake -c clean %s' % testrecipe)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
         self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile')), 'Extracted source could not be found')
-        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
         self.assertTrue(os.path.exists(os.path.join(workspacedir, 'conf', 'layer.conf')), 'Workspace directory not created')
         matches = glob.glob(os.path.join(workspacedir, 'appends', 'mkelfimage_*.bbappend'))
         self.assertTrue(matches, 'bbappend not created')
@@ -359,10 +366,7 @@ class DevtoolTests(DevtoolBase):
         self.assertIn(testrecipe, result.output)
         self.assertIn(tempdir, result.output)
         # Check git repo
-        result = runCmd('git status --porcelain', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Try building
         bitbake(testrecipe)
 
@@ -414,11 +418,7 @@ class DevtoolTests(DevtoolBase):
         # (don't bother with cleaning the recipe on teardown, we won't be building it)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
         # Check git repo
-        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
-        result = runCmd('git status --porcelain', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Add a couple of commits
         # FIXME: this only tests adding, need to also test update and remove
         result = runCmd('echo "Additional line" >> README', cwd=tempdir)
@@ -464,11 +464,7 @@ class DevtoolTests(DevtoolBase):
         # (don't bother with cleaning the recipe on teardown, we won't be building it)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
         # Check git repo
-        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
-        result = runCmd('git status --porcelain', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Add a couple of commits
         # FIXME: this only tests adding, need to also test update and remove
         result = runCmd('echo "# Additional line" >> Makefile', cwd=tempdir)
@@ -534,11 +530,7 @@ class DevtoolTests(DevtoolBase):
         # (don't bother with cleaning the recipe on teardown, we won't be building it)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempsrcdir))
         # Check git repo
-        self.assertTrue(os.path.isdir(os.path.join(tempsrcdir, '.git')), 'git repository for external source tree not found')
-        result = runCmd('git status --porcelain', cwd=tempsrcdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempsrcdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Add a commit
         result = runCmd("sed 's!\\(#define VERSION\\W*\"[^\"]*\\)\"!\\1-custom\"!' -i ReadMe.c", cwd=tempsrcdir)
         result = runCmd('git commit -a -m "Add our custom version"', cwd=tempsrcdir)
@@ -612,11 +604,7 @@ class DevtoolTests(DevtoolBase):
         # (don't bother with cleaning the recipe on teardown, we won't be building it)
         result = runCmd('devtool modify %s -x %s' % (testrecipe, tempsrcdir))
         # Check git repo
-        self.assertTrue(os.path.isdir(os.path.join(tempsrcdir, '.git')), 'git repository for external source tree not found')
-        result = runCmd('git status --porcelain', cwd=tempsrcdir)
-        self.assertEqual(result.output.strip(), "", 'Created git repo is not clean')
-        result = runCmd('git symbolic-ref HEAD', cwd=tempsrcdir)
-        self.assertEqual(result.output.strip(), "refs/heads/devtool", 'Wrong branch in git repo')
+        self._check_src_repo(tempdir)
         # Add a commit
         result = runCmd('echo "# Additional line" >> Makefile', cwd=tempsrcdir)
         result = runCmd('git commit -a -m "Change the Makefile"', cwd=tempsrcdir)
@@ -693,7 +681,7 @@ class DevtoolTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         result = runCmd('devtool extract remake %s' % tempdir)
         self.assertTrue(os.path.exists(os.path.join(tempdir, 'Makefile.am')), 'Extracted source could not be found')
-        self.assertTrue(os.path.isdir(os.path.join(tempdir, '.git')), 'git repository for external source tree not found')
+        self._check_src_repo(tempdir)
 
     def test_devtool_reset_all(self):
         workspacedir = self._get_workspace_dir()
Signature · 1 line
2.1.4
#5 lore raw

PATCH v2 5/7 devtool: update-recipe: update local files directly

In reply to Markus Lehtonen

Non-remote source files (i.e. SRC_URI files that use the file:// URI prefix and thus reside in the "recipe space" in the local copy of the metadata) are imported into srctree (with devtool extract) in the case S=WORKDIR. If these files are local (i.e. they reside in the "recipe space" and not behind a remote URL) we don't want create a patch against them, but, rather copy our modified version over the original source.

NOTE: if new files are created, they are represented as patches, rather than copied over the orignal source.

[YOCTO #7602]

---
 meta/lib/oeqa/selftest/devtool.py | 44 +++++++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   | 48 ++++++++++++++++++++++++++++++++++++---
 2 files changed, 89 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/devtool.py +44 -0
index 08ed2eb..6aaf5a5 100644
@@ -672,6 +672,50 @@ class DevtoolTests(DevtoolBase):
             self.assertEqual(expectedlines, f.readlines())
         # Deleting isn't expected to work under these circumstances
 
+    def test_devtool_update_recipe_local_files(self):
+        """Check that local source files are copied over instead of patched"""
+        workspacedir = self._get_workspace_dir()
+        testrecipe = 'makedevs'
+        recipefile = get_bb_var('FILE', testrecipe)
+        # Setup srctree for modifying the recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be
+        # building it)
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        # Check git repo
+        self._check_src_repo(tempdir)
+        # Edit / commit local source
+        runCmd('echo "/* New comment */" >> makedevs.c', cwd=tempdir)
+        runCmd('git commit -am "My change"', cwd=tempdir)
+        runCmd('echo "Foo" > new-file', cwd=tempdir)
+        runCmd('git add new-file', cwd=tempdir)
+        runCmd('git commit -m "Add new file"', cwd=tempdir)
+        self.add_command_to_tearDown('cd %s; git clean -fd .; git checkout .' %
+                                     os.path.dirname(recipefile))
+        runCmd('devtool update-recipe %s' % testrecipe)
+        result = runCmd('git status . --porcelain',
+                        cwd=os.path.dirname(recipefile))
+        status = result.output.splitlines()
+        self.assertEqual(len(status), 3,
+                         'Less/more files modified than expected. '
+                         'Entire status:\n%s' % result.output)
+        for line in status:
+            if line.endswith('makedevs.c'):
+                self.assertEqual(line[:3], ' M ',
+                                 'Unexpected status in line: %s' % line)
+            elif line.endswith('0001-Add-new-file.patch'):
+                self.assertEqual(line[:3], '?? ',
+                                 'Unexpected status in line: %s' % line)
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
+                self.assertEqual(line[:3], ' M ',
+                                 'Unexpected status in line: %s' % line)
+            else:
+                raise AssertionError('Unexpected modified file in status: %s' %
+                                     line)
+
     def test_devtool_extract(self):
         workspacedir = self._get_workspace_dir()
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
scripts/lib/devtool/standard.py +45 -3

78 lines not on this page · raw message

Signature · 1 line
2.1.4
#6 lore raw

PATCH v2 6/7 devtool: extract: always import local files to srctree

In reply to Markus Lehtonen

Import all non-compressed/non-arcived non-remote source files (i.e. local files from the SRC_URI) - excluding patches - to the srctree repository. The files will be placed in a subdirectory called 'local-files'. However, in case S=WORKDIR, the files are imported into root ot srctree (and not under 'local-files'), just like before.

[YOCTO #7602]

---
 meta/lib/oeqa/selftest/devtool.py | 46 +++++++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py   | 58 ++++++++++++++++++++++++++++++---------
 2 files changed, 91 insertions(+), 13 deletions(-)
meta/lib/oeqa/selftest/devtool.py +46 -0
index 6aaf5a5..c1623c2 100644
@@ -716,6 +716,52 @@ class DevtoolTests(DevtoolBase):
                 raise AssertionError('Unexpected modified file in status: %s' %
                                      line)
 
+    def test_devtool_update_recipe_local_files_2(self):
+        """Check handling of local source files for recipes where
+        S != WORKDIR"""
+        workspacedir = self._get_workspace_dir()
+        testrecipe = 'lzo'
+        recipefile = get_bb_var('FILE', testrecipe)
+        # Setup srctree for modifying the recipe
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool modify %s -x %s' % (testrecipe, tempdir))
+        # Check git repo
+        self._check_src_repo(tempdir)
+        # Edit / commit local source
+        runCmd('echo "# New comment" >> local-files/acinclude.m4', cwd=tempdir)
+        runCmd('git commit -am "Edit existing file"', cwd=tempdir)
+        runCmd('echo "Foo" > local-files/new-local', cwd=tempdir)
+        runCmd('git add local-files/new-local', cwd=tempdir)
+        runCmd('git commit -m "Add new local file"', cwd=tempdir)
+        runCmd('echo "Foo" > new-file', cwd=tempdir)
+        runCmd('git add new-file', cwd=tempdir)
+        runCmd('git commit -m "Add new file"', cwd=tempdir)
+        self.add_command_to_tearDown('cd %s; git clean -fd .; git checkout .' %
+                                     os.path.dirname(recipefile))
+        runCmd('devtool update-recipe %s' % testrecipe)
+        result = runCmd('git status . --porcelain',
+                        cwd=os.path.dirname(recipefile))
+        status = result.output.splitlines()
+        self.assertEqual(len(status), 3,
+                         'Less/more files modified than expected. '
+                         'Entire status:\n%s' % result.output)
+        for line in status:
+            if line.endswith('acinclude.m4'):
+                self.assertEqual(line[:3], ' M ',
+                                 'Unexpected status in line: %s' % line)
+            elif line.endswith('Add-new-file.patch'):
+                self.assertEqual(line[:3], '?? ',
+                                 'Unexpected status in line: %s' % line)
+            elif re.search('%s_[^_]*.bb$' % testrecipe, line):
+                self.assertEqual(line[:3], ' M ',
+                                 'Unexpected status in line: %s' % line)
+            else:
+                raise AssertionError('Unexpected modified file in status: %s' %
+                                     line)
+
     def test_devtool_extract(self):
         workspacedir = self._get_workspace_dir()
         tempdir = tempfile.mkdtemp(prefix='devtoolqa')
scripts/lib/devtool/standard.py +45 -13

78 lines not on this page · raw message

Signature · 1 line
2.1.4
#7 lore raw

PATCH v2 7/7 devtool: modify: make bitbake use local files from srctree

In reply to Markus Lehtonen

This change makes it possible to have local files (non-remote SRC_URI files, i.e. files that are located in the "recipe space") under the srctree even if S!=WORKDIR. The files must be placed under the 'local-files' subdirectory.

Complements the previous patch that imports local files into srctree.

[YOCTO #7602]

---
 scripts/lib/devtool/standard.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +7 -2
index 44948fd..5d2a264 100644
@@ -501,8 +501,13 @@ def modify(args, config, basepath, workspace):
         appendname = re.sub(r'_.*', '_%', appendname)
     appendfile = os.path.join(appendpath, appendname + '.bbappend')
     with open(appendfile, 'w') as f:
-        f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n\n')
-        f.write('inherit externalsrc\n')
+        f.write('FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"\n')
+        # Local files can be modified/tracked in separate subdir under srctree
+        # Mostly useful for packages with S != WORKDIR
+        f.write('FILESPATH_prepend := "%s:"\n' %
+                os.path.join(srctree, 'local-files'))
+
+        f.write('\ninherit externalsrc\n')
         f.write('# NOTE: We use pn- overrides here to avoid affecting multiple variants in the case where the recipe uses BBCLASSEXTEND\n')
         f.write('EXTERNALSRC_pn-%s = "%s"\n' % (args.recipename, srctree))
 
Signature · 1 line
2.1.4
#8 lore raw
In reply to Markus Lehtonen

Hi Markus,

Quoted 3 lines · On Wednesday 03 June 2015 17:50:25 Markus Lehtonen wrote:

Second version of my patchset aiming to improve handling of local source files. The only significant change since the first version is the addition of unit tests.

Did you see the comments I made on v1 patchset?

https://www.mail-archive.com/[email protected]/msg64396.html

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#9 lore raw

Hi,

Quoted 10 lines · Paul Eggleton

Hi Markus,

Quoted 3 lines · On Wednesday 03 June 2015 17:50:25 Markus Lehtonen wrote:

Second version of my patchset aiming to improve handling of local source files. The only significant change since the first version is the addition of unit tests.

Did you see the comments I made on v1 patchset?

https://www.mail-archive.com/[email protected]/msg64396.html

Hmm, I had somehow managed to miss that one. I now responded to that.

Cheers,
Markus