Skip to contentSkip to conversation

devtool: use -f and don't use --exclude-standard when adding files to workspace

PATCH In review Martin Jansa 2 participants 7 messages May 29, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/6 devtool: use -f and don't use --exclude-standard when adding files to workspace

* I see a case where a tarball contains .gitignore and bunch of files which are normally ignored in git, but still included in the tarball (e.g. configure script next to configure.ac)
* when devtool is creating a git repo in workspace it won't include these files from tarball in the initial devtool-base commit, because git ls-files won't list them
* but then the first .patch file (without git headers) when applied with GitApplyTree._applypatch() will add all these still ignored files to a commit which used to only modify some files, because it's using -f:

      # Add all files
      shellcmd = ["git", "add", "-f", "-A", "."]
      output += runcmd(["sh", "-c", " ".join(shellcmd)], self.dir)

at least in this case it would be better to add all ignored files in the initial devtool-base commit and then --force-patch-refresh will just include the small modification as before instead of adding unrelated files, just because they were initially ignored - this behavior will also match with the do_patch task in the actual build where the .gitignore is ignored when unpacking some tarball
* my use-case is fixed in setup_git_repo, but similar function is in devtool upgrade, I've changed it there as well

---
 scripts/lib/devtool/__init__.py | 2 +-
 scripts/lib/devtool/upgrade.py  | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)
scripts/lib/devtool/__init__.py +1 -1
index d39c474fbd..6ebe368a9e 100644
@@ -196,7 +196,7 @@ def setup_git_repo(repodir, version, devbranch, basetag='devtool-base', d=None):
     if not os.path.exists(os.path.join(repodir, '.git')):
         bb.process.run('git init', cwd=repodir)
         bb.process.run('git config --local gc.autodetach 0', cwd=repodir)
-        bb.process.run('git add .', cwd=repodir)
+        bb.process.run('git add -f -A .', cwd=repodir)
         commit_cmd = ['git']
         oe.patch.GitApplyTree.gitCommandUserOptions(commit_cmd, d=d)
         commit_cmd += ['commit', '-q']
scripts/lib/devtool/upgrade.py +2 -2
index f962a71e41..ebe72282bb 100644
@@ -235,14 +235,14 @@ def _extract_new_source(newpv, srctree, no_patch, srcrev, srcbranch, branch, kee
         # Copy in new ones
         _copy_source_code(tmpsrctree, srctree)
 
-        (stdout,_) = __run('git ls-files --modified --others --exclude-standard')
+        (stdout,_) = __run('git ls-files --modified --others')
         filelist = stdout.splitlines()
         pbar = bb.ui.knotty.BBProgress('Adding changed files', len(filelist))
         pbar.start()
         batchsize = 100
         for i in range(0, len(filelist), batchsize):
             batch = filelist[i:i+batchsize]
-            __run('git add -A %s' % ' '.join(['"%s"' % item for item in batch]))
+            __run('git add -f -A %s' % ' '.join(['"%s"' % item for item in batch]))
             pbar.update(i)
         pbar.finish()
 
Signature · 1 line
2.25.1
#1 lore raw

PATCH 2/6 meta-selftest: add test of .gitignore in tarball

---
 .../devtool/devtool-test-ignored.bb           |   9 ++++++
 .../devtool-test-ignored.patch                |   7 +++++
 .../devtool-test-ignored.patch.expected       |  16 +++++++++++
 .../devtool-test-ignored.tar.gz               | Bin 0 -> 205 bytes
 meta/lib/oeqa/selftest/cases/devtool.py       |  26 ++++++++++++++++++
 5 files changed, 58 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-ignored.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch.expected
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.tar.gz
meta-selftest/recipes-test/devtool/devtool-test-ignored.bb new file +9 -0
new file mode 100644
index 0000000000..6a3d58c884
@@ -0,0 +1,9 @@
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "file://${BPN}.tar.gz \
+           file://${BPN}.patch"
+
+S = "${WORKDIR}/${BPN}"
+
+EXCLUDE_FROM_WORLD = "1"
meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch new file +7 -0
new file mode 100644
index 0000000000..96ea0eb4e3
@@ -0,0 +1,7 @@
+diff --git a/ignored b/ignored
+index a579759..e3d7b43 100644
+--- a/ignored
++++ b/ignored
+@@ -1 +1 @@
+-I'm so ignored
++# I'm so ignored
meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch.expected new file +16 -0
new file mode 100644
index 0000000000..68ec6d9875
@@ -0,0 +1,16 @@
+From 3a286343cc5cadd83f41d524ee3606ae51df9ee7 Mon Sep 17 00:00:00 2001
+From: Martin Jansa <[email protected]>
+Date: Thu, 28 May 2020 01:32:31 +0200
+Subject: [PATCH] meta-selftest: add test of .gitignore in tarball
+
+---
+ ignored | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/ignored b/ignored
+index a579759..e3d7b43 100644
+--- a/ignored
++++ b/ignored
+@@ -1 +1 @@
+-I'm so ignored
++# I'm so ignored
meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.tar.gz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..b2e9935eb9d3fdbeb24345435b11df2438dfd6eb
GIT binary patch
meta/lib/oeqa/selftest/cases/devtool.py +26 -0
index 5886862d6c..0218f0821c 100644
@@ -1108,6 +1108,32 @@ class DevtoolUpdateTests(DevtoolBase):
                            ('??', '.*/0001-Add-new-file.patch$')]
         self._check_repo_status(os.path.dirname(recipefile), expected_status)
 
+    def test_devtool_update_recipe_with_gitignore(self):
+        # First, modify the recipe
+        testrecipe = 'devtool-test-ignored'
+        bb_vars = get_bb_vars(['FILE'], testrecipe)
+        recipefile = bb_vars['FILE']
+        patchfile = os.path.join(os.path.dirname(recipefile), testrecipe, testrecipe + '.patch')
+        newpatchfile = os.path.join(os.path.dirname(recipefile), testrecipe, testrecipe + '.patch.expected')
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool finish --force-patch-refresh %s meta-selftest' % testrecipe)
+        # Check recipe got changed as expected
+        with open(newpatchfile, 'r') as f:
+            desiredlines = f.readlines()
+        with open(patchfile, 'r') as f:
+            newlines = f.readlines()
+        # Ignore the initial lines, because oe-selftest creates own meta-selftest repo
+        # which changes the metadata subject which is added into the patch, but keep
+        # .patch.expected as it is in case someone runs devtool finish --force-patch-refresh
+        # devtool-test-ignored manually, then it should generate exactly the same .patch file
+        self.assertEqual(desiredlines[5:], newlines[5:])
+
     def test_devtool_update_recipe_local_files_3(self):
         # First, modify the recipe
         testrecipe = 'devtool-test-localonly'
Signature · 1 line
2.25.1
#2 lore raw

PATCH 3/6 lib/oe/patch: prevent applying patches without any subject

In reply to Martin Jansa

* this was discovered with

  $ devtool finish --force-patch-refresh
  where it was removing some patches and replacing them with
  patch in filename called "patch:"

e.g. this .patch file:
https://github.com/OSSystems/meta-browser/blob/311067d2d8a50cee5c836892606444f63f2bb3ab/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch
confuses devtool which results to create new .patch file called "patch:"

  $ devtool finish --force-patch-refresh firefox meta-browser
  NOTE: Starting bitbake server...
  WARNING: Host distribution "ubuntu-20.04" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
  Loading cache: 100% |###################################################################################################################################################################################################################################| Time: 0:00:00
  Loaded 2480 entries from dependency cache.
  Parsing recipes: 100% |#################################################################################################################################################################################################################################| Time: 0:00:00
  Parsing of 1718 .bb files complete (1717 cached, 1 parsed). 2480 targets, 68 skipped, 0 masked, 0 errors.

Summary: There was 1 WARNING message shown.
INFO: Updating patch 0001-Bug-1554949-Fix-WebRTC-build-failure-with-newer-linu.patch
...
INFO: Updating patch pre-generated-old-configure.patch
INFO: Adding new patch patch:
INFO: Updating recipe firefox_68.0esr.bb
INFO: Removing file /OE/build/test-oe-build-time/poky/meta-browser/dynamic-layers/rust-layer/recipes-browser/firefox/firefox/fixes/fix-camera-permission-dialg-doesnot-close.patch
INFO: Cleaning sysroot for recipe firefox...
INFO: Leaving source tree /OE/build/test-oe-build-time/poky/build/workspace/sources/firefox as-is; if you no longer need it then please delete it manually

this looked like incorrect parsing of the git format-patch files exported from workspace/sources (the git format-patch version of fix-camera-permission-dialg-doesnot-close.patch starts like this:

  $ head 0008-original-patch-fix-camera-permission-dialg-doesnot-c.patch
  From 37dfa11961b48024bedcfb9336f49107c9535638 Mon Sep 17 00:00:00 2001
  From: Takuro Ashie <[email protected]>
  Date: Mon, 20 Aug 2018 10:16:20 +0900
  Subject: [PATCH 08/34] %% original patch:
   fix-camera-permission-dialg-doesnot-close.patch

so first I've modified GitApplyTree.extractPatches() to be able to parse the original patch name correctly even in this case where subject is wrapped, but then it still wasn't right, because we ended with correctly named .patch file, but all we could use for Subject line was the name of the original .patch file (instead of the Subject from metadata commit which introduced this .patch files as some other .patch files get when refreshed with devtool.

In the end the issue happens even sooner in GitApplyTree.prepareCommit() where it correctly found the Subject from metadata commit, but then didn't apply it when there weren't any other outlines from patch headers.

---
 meta/lib/oe/patch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/patch.py +1 -1
index 2b1eee1003..fa92abe248 100644
@@ -416,7 +416,7 @@ class GitApplyTree(PatchTree):
                     date = newdate
                 if not subject:
                     subject = newsubject
-        if subject and outlines and not outlines[0].strip() == subject:
+        if subject and not (outlines and outlines[0].strip() == subject):
             outlines.insert(0, '%s\n\n' % subject.strip())
 
         # Write out commit message to a file
Signature · 1 line
2.25.1
#3 lore raw

PATCH 4/6 lib/oe/patch: GitApplyTree: save 1 echo in commit-msg hook

In reply to Martin Jansa

* also remove the extra blank lines which is often added to patches when refreshed with devtool (GitApplyTree.patch_line_prefix lines are ignored when refreshing .patch files, but newly added blank lines aren't - the leading blank line wasneeded for patches with just the subject line (to prevent the GitApplyTree.patch_line_prefix line ending appended to the commit summary), but we can add it in prepareCommit instead

---
 meta/lib/oe/patch.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
meta/lib/oe/patch.py +1 -2
index fa92abe248..bb1c40aa1e 100644
@@ -508,8 +508,7 @@ class GitApplyTree(PatchTree):
         with open(commithook, 'w') as f:
             # NOTE: the formatting here is significant; if you change it you'll also need to
             # change other places which read it back
-            f.write('echo >> $1\n')
-            f.write('echo "%s: $PATCHFILE" >> $1\n' % GitApplyTree.patch_line_prefix)
+            f.write('echo "\n%s: $PATCHFILE" >> $1' % GitApplyTree.patch_line_prefix)
         os.chmod(commithook, 0o755)
         shutil.copy2(commithook, applyhook)
         try:
Signature · 1 line
2.25.1
#4 lore raw

PATCH 5/6 Revert "lib/oe/patch: fix handling of patches with no header"

In reply to Martin Jansa

* This reverts commit d9971f5dc8eb7de551fd6f5e058fd24770ef5d78.

* With the missing Subject line fixed in GitApplyTree.prepareCommit() we should be able to revert, the fix which was trying to help it by parsing GitApplyTree.patch_line_prefix ("%% original patch:") also from Subject line, now GitApplyTree.patch_line_prefix should always end on separate line which is then skipped when copying the lines to resulting patch, see original commit message from Paul:

lib/oe/patch: fix handling of patches with no header

If a patch applied by a recipe has no header and we turn the recipe's source into a git tree (when PATCHTOOL = "git" or when using devtool extract / modify / upgrade), the commit message ends up consisting only of the original filename marker ("%% original patch: filename.patch"). When we come to do turn the commits back into a set of patches in extractPatches(), this first line ends up in the "Subject: " part of the file, but we were ignoring it because the line didn't start with the marker text. The end result was we weren't able to get the original patch name. Strip off any "Subject [PATCH x/y]" part before looking for the marker text to fix.

This caused "devtool modify openssl" followed by "devtool update-recipe openssl" (without any changes in-between) to remove version-script.patch because that patch has no header and we weren't able to determine the original filename.

---
 meta/lib/oe/patch.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)
meta/lib/oe/patch.py +2 -6
index bb1c40aa1e..7ca2e28b1f 100644
@@ -439,7 +439,6 @@ class GitApplyTree(PatchTree):
     def extractPatches(tree, startcommit, outdir, paths=None):
         import tempfile
         import shutil
-        import re
         tempdir = tempfile.mkdtemp(prefix='oepatch')
         try:
             shellcmd = ["git", "format-patch", "--no-signature", "--no-numbered", startcommit, "-o", tempdir]
@@ -455,13 +454,10 @@ class GitApplyTree(PatchTree):
                         try:
                             with open(srcfile, 'r', encoding=encoding) as f:
                                 for line in f:
-                                    checkline = line
-                                    if checkline.startswith('Subject: '):
-                                        checkline = re.sub(r'\[.+?\]\s*', '', checkline[9:])
-                                    if checkline.startswith(GitApplyTree.patch_line_prefix):
+                                    if line.startswith(GitApplyTree.patch_line_prefix):
                                         outfile = line.split()[-1].strip()
                                         continue
-                                    if checkline.startswith(GitApplyTree.ignore_commit_prefix):
+                                    if line.startswith(GitApplyTree.ignore_commit_prefix):
                                         continue
                                     patchlines.append(line)
                         except UnicodeDecodeError:
Signature · 1 line
2.25.1
#5 lore raw

PATCH 6/6 meta-selftest: add test for .patch file with long filename and without subject

In reply to Martin Jansa
---
 .../devtool/devtool-test-long-filename.bb     |   9 ++++++
 ...nly-if-devtool-lets-me-to-do-it-corr.patch |   7 +++++
 ...vtool-lets-me-to-do-it-corr.patch.expected |  16 +++++++++++
 .../devtool-test-long-filename.tar.gz         | Bin 0 -> 180 bytes
 meta/lib/oeqa/selftest/cases/devtool.py       |  27 ++++++++++++++++++
 5 files changed, 59 insertions(+)
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-long-filename.bb
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-long-filename/0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-long-filename/0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch.expected
 create mode 100644 meta-selftest/recipes-test/devtool/devtool-test-long-filename/devtool-test-long-filename.tar.gz
meta-selftest/recipes-test/devtool/devtool-test-long-filename.bb new file +9 -0
new file mode 100644
index 0000000000..3ec22cae7f
@@ -0,0 +1,9 @@
+LICENSE = "CLOSED"
+INHIBIT_DEFAULT_DEPS = "1"
+
+SRC_URI = "file://${BPN}.tar.gz \
+           file://0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch"
+
+S = "${WORKDIR}/${BPN}"
+
+EXCLUDE_FROM_WORLD = "1"
meta-selftest/recipes-test/devtool/devtool-test-long-filename/0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch new file +7 -0
new file mode 100644
index 0000000000..6aaf409ebc
@@ -0,0 +1,7 @@
+diff --git a/patch-me b/patch-me
+index a20b29a..5e35d1b 100644
+--- a/patch-me
++++ b/patch-me
+@@ -1 +1 @@
+-please
++NO
meta-selftest/recipes-test/devtool/devtool-test-long-filename/0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch.expected new file +16 -0
new file mode 100644
index 0000000000..1bf25a61d0
@@ -0,0 +1,16 @@
+From 45ba3d107ea60777a6b6e134fd00fe5009749177 Mon Sep 17 00:00:00 2001
+From: Martin Jansa <[email protected]>
+Date: Thu, 28 May 2020 02:03:39 +0200
+Subject: [PATCH] meta-selftest: add test for .patch file with long filename
+
+---
+ patch-me | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/patch-me b/patch-me
+index a20b29a..5e35d1b 100644
+--- a/patch-me
++++ b/patch-me
+@@ -1 +1 @@
+-please
++NO
meta-selftest/recipes-test/devtool/devtool-test-long-filename/devtool-test-long-filename.tar.gz new file binary +0 -0
new file mode 100644
index 0000000000000000000000000000000000000000..ab6242aae78dfc2257d44e5caef94f89a269f668
GIT binary patch
meta/lib/oeqa/selftest/cases/devtool.py +27 -0
index 0218f0821c..49cee6d6a8 100644
@@ -1134,6 +1134,33 @@ class DevtoolUpdateTests(DevtoolBase):
         # devtool-test-ignored manually, then it should generate exactly the same .patch file
         self.assertEqual(desiredlines[5:], newlines[5:])
 
+    def test_devtool_update_recipe_long_filename(self):
+        # First, modify the recipe
+        testrecipe = 'devtool-test-long-filename'
+        bb_vars = get_bb_vars(['FILE'], testrecipe)
+        recipefile = bb_vars['FILE']
+        patchfilename = '0001-I-ll-patch-you-only-if-devtool-lets-me-to-do-it-corr.patch'
+        patchfile = os.path.join(os.path.dirname(recipefile), testrecipe, patchfilename)
+        newpatchfile = os.path.join(os.path.dirname(recipefile), testrecipe, patchfilename + '.expected')
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        # (don't bother with cleaning the recipe on teardown, we won't be building it)
+        result = runCmd('devtool modify %s' % testrecipe)
+        self.add_command_to_tearDown('cd %s; rm %s/*; git checkout %s %s' % (os.path.dirname(recipefile), testrecipe, testrecipe, os.path.basename(recipefile)))
+        result = runCmd('devtool finish --force-patch-refresh %s meta-selftest' % testrecipe)
+        # Check recipe got changed as expected
+        with open(newpatchfile, 'r') as f:
+            desiredlines = f.readlines()
+        with open(patchfile, 'r') as f:
+            newlines = f.readlines()
+        # Ignore the initial lines, because oe-selftest creates own meta-selftest repo
+        # which changes the metadata subject which is added into the patch, but keep
+        # .patch.expected as it is in case someone runs devtool finish --force-patch-refresh
+        # devtool-test-ignored manually, then it should generate exactly the same .patch file
+        self.assertEqual(desiredlines[5:], newlines[5:])
+
     def test_devtool_update_recipe_local_files_3(self):
         # First, modify the recipe
         testrecipe = 'devtool-test-localonly'
Signature · 1 line
2.25.1
#6 lore raw

✗ patchtest: failure for "devtool: use -f and don't use ..." and 5 more

In reply to Martin Jansa

== Series Details ==

Series: "devtool: use -f and don't use ..." and 5 more
Revision: 1
URL : https://patchwork.openembedded.org/series/24387/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue A patch file has been added, but does not have a Signed-off-by tag [test_signed_off_by_presence]
Suggested fix Sign off the added patch file (meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch)

* Issue Added patch file is missing Upstream-Status in the header [test_upstream_status_presence_format]

  Suggested fix    Add Upstream-Status: <Valid status> to the header of meta-selftest/recipes-test/devtool/devtool-test-ignored/devtool-test-ignored.patch
  Standard format  Upstream-Status: <Valid status>
  Valid status     Pending, Accepted, Backport, Denied, Inappropriate [reason], Submitted [where]

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe