Skip to contentSkip to conversation

devtool: fix finish for recipes using AUTOREV

PATCH Superseded Babanpreet Singh 2 participants 8 messages Jul 13 – Jul 14

lore
Share & tools
lore
#0 lore raw

cover devtool: fix finish for recipes using AUTOREV

devtool finish (and devtool update-recipe in srcrev mode) crashes with "fatal: bad revision 'AUTOINC'" on any recipe with SRCREV = "${AUTOREV}", e.g. one created with devtool add --autorev, because the unresolved AUTOREV placeholder is passed to git format-patch as the base revision for exporting patches.

Patch 1 uses the initial revision(s) recorded in the workspace bbappend instead (as patch mode already does), and stops replacing the AUTOREV with a pinned revision on finish. Patch 2 adds the oe-selftest case Randy asked for in the bug; it fails with the exact reported error before patch 1 and passes after.

While investigating I verified this is not a recent regression: the same crash reproduces on wrynose, whinlatter, scarthgap and kirkstone, so it most likely dates back to the introduction of --autorev.

Tested on oe-core master in a qemux86-64 build environment:
- devtool add --autorev + devtool finish: fails before, passes after, finished recipe keeps SRCREV = "${AUTOREV}"
- devtool add (pinned) + devtool finish: unchanged
- oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_git_autorev_finish (new): PASSED
- oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_update_recipe_git devtool.DevtoolUpdateTests.test_devtool_update_recipe_append_git: PASSED

[YOCTO #16354]

Babanpreet Singh (2):
devtool: standard: fix finish/update-recipe for recipes using AUTOREV oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV

 meta/lib/oeqa/selftest/cases/devtool.py | 32 +++++++++++++++++++++++++
 scripts/lib/devtool/standard.py         | 16 +++++++++++--
 2 files changed, 46 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH 1/2 devtool: standard: fix finish/update-recipe for recipes using AUTOREV

For a recipe with SRCREV = "${AUTOREV}" (e.g. created by devtool add --autorev), devtool finish and devtool update-recipe in srcrev mode crash:

INFO: Updating SRCREV in recipe minit_git.bb
...
oe.patch.CmdError: Command Error: 'git format-patch --no-signature --no-numbered AUTOINC -o /tmp/oepatchxu0ig0v0 -- .' exited with 128
Output: stderr: fatal: bad revision 'AUTOINC'

_update_recipe_srcrev() uses the recipe's SRCREV as the base revision for exporting patches from the source tree, but for AUTOREV recipes getVar('SRCREV') expands to the literal placeholder "AUTOINC", which is not a valid git revision. Use the initial revision(s) recorded in the workspace bbappend instead, as patch mode already does via _get_patchset_revs().

Do not replace SRCREV with the current source tree HEAD for such recipes either: pinning the revision on finish would silently drop the floating revision the user explicitly asked for.

This is not a recent regression: the same crash reproduces at least as far back as kirkstone, so it most likely dates back to the introduction of devtool add --autorev.

[YOCTO #16354]

AI-Generated: Uses Claude (claude-sonnet-5)

---
 scripts/lib/devtool/standard.py | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +14 -2
index 2a3a62d081..386e628d88 100644
@@ -1530,7 +1530,18 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
     old_srcrev = rd.getVar('SRCREV') or ''
     if old_srcrev == "INVALID":
             raise DevtoolError('Update mode srcrev is only valid for recipe fetched from an SCM repository')
-    old_srcrev = {'.': old_srcrev}
+    autorev = old_srcrev == 'AUTOINC'
+    if autorev:
+        # SRCREV is set to "${AUTOREV}" so there is no fixed revision to
+        # use as the base for exporting patches; use the initial revision(s)
+        # recorded when the source tree was set up, as patch mode does
+        append = workspace[recipename]['bbappend']
+        old_srcrev, _, _, _ = _get_patchset_revs(srctree, append)
+        if not old_srcrev:
+            raise DevtoolError('Unable to find the initial revision of the '
+                               'source tree for %s in the workspace' % recipename)
+    else:
+        old_srcrev = {'.': old_srcrev}
 
     # Get HEAD revision
     try:
@@ -1545,7 +1556,8 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
     destpath = None
     remove_files = []
     patchfields = {}
-    patchfields['SRCREV'] = srcrev
+    if not autorev:
+        patchfields['SRCREV'] = srcrev
     orig_src_uri = rd.getVar('SRC_URI', False) or ''
     srcuri = orig_src_uri.split()
     tempdir = tempfile.mkdtemp(prefix='devtool')
Signature · 1 line
2.43.0
#2 lore raw

PATCH 2/2 oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV

In reply to Babanpreet Singh

Add a test checking that a recipe created with devtool add --autorev can be finished into a layer, that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated.

Before the previous commit, devtool finish failed on such recipes with:

fatal: bad revision 'AUTOINC'

Test requested by Randy MacLeod in the bug report.

[YOCTO #16354]

AI-Generated: Uses Claude (claude-sonnet-5)

---
 meta/lib/oeqa/selftest/cases/devtool.py | 32 +++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +32 -0
index b2cd13e9a5..a3c7c293c5 100644
@@ -469,6 +469,38 @@ class DevtoolAddTests(DevtoolBase):
         checkvars['DEPENDS'] = set(['dbus'])
         self._test_recipe_contents(recipefile, checkvars, [])
 
+    def test_devtool_add_git_autorev_finish(self):
+        # Test that a recipe created by devtool add --autorev can be finished
+        # into a layer; this used to fail with "fatal: bad revision 'AUTOINC'"
+        # because the unresolved AUTOREV placeholder was used as the git
+        # revision to export patches from [YOCTO #16354]
+        pn = 'dbus-wait'
+        url = 'git://git.yoctoproject.org/dbus-wait;protocol=https;branch=master'
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd("devtool add --autorev %s '%s'" % (pn, url))
+        recipefile = os.path.join(self.workspacedir, 'recipes', pn, '%s_git.bb' % pn)
+        self.assertExists(recipefile, 'Recipe not created by devtool add')
+        checkvars = {}
+        checkvars['SRCREV'] = '${AUTOREV}'
+        self._test_recipe_contents(recipefile, checkvars, [])
+        # Finish the recipe into a temporary layer
+        templayerdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(templayerdir)
+        self._create_temp_layer(templayerdir, False, 'selftestautorev')
+        result = runCmd('devtool finish %s %s' % (pn, templayerdir))
+        result = runCmd('devtool status')
+        self.assertNotIn(pn, result.output, 'Recipe should have been reset by finish but wasn\'t')
+        newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', pn, '%s_git.bb' % pn))
+        self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish')
+        # The recipe should still use a floating revision and no patch
+        # files should have been created
+        checkvars = {}
+        checkvars['SRCREV'] = '${AUTOREV}'
+        self._test_recipe_contents(newrecipefiles[0], checkvars, [])
+        patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', pn, '*', '*.patch'))
+        self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish')
+
     def test_devtool_add_git_style1(self):
         version = 'v3.1.0'
         pn = 'mbedtls'
Signature · 1 line
2.43.0
#3 lore raw

Re: [OE-core] [PATCH 2/2] oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolAddTests(DevtoolBase): · line 476

Quoted 6 lines · Baban via lists.openembedded.org

AI-Generated: Uses Claude (claude-sonnet-5)

meta/lib/oeqa/selftest/cases/devtool.py +5 -0
+    def test_devtool_add_git_autorev_finish(self):
+        # Test that a recipe created by devtool add --autorev can be finished
+        # into a layer; this used to fail with "fatal: bad revision 'AUTOINC'"
+        # because the unresolved AUTOREV placeholder was used as the git
+        # revision to export patches from [YOCTO #16354]

There is no need to add a whole new test that largely copy-pastes other tests. There is already test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.

Alex

#4 lore raw

Re: [OE-core] [PATCH 2/2] oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV

Quoted 4 lines · Alexander Kanavin

There is no need to add a whole new test that largely copy-pastes other tests. There is already test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.

Fair point - the add/check portion did duplicate what that test already does. I'll send a v2 that drops the new test and instead adjust test_devtool_add_fetch_git().

Thanks for the review,
Baban

Quoted 18 lines · Alexander Kanavin

Review

Quoted 3 lines · Baban via lists.openembedded.org

AI-Generated: Uses Claude (claude-sonnet-5)

(quoted diff) +2 -0
+    def test_devtool_add_git_autorev_finish(self):
+        # Test that a recipe created by devtool add --autorev can be

finished

Quoted 1 line

+ # into a layer; this used to fail with "fatal: bad revision

'AUTOINC'"

meta/lib/oeqa/selftest/cases/devtool.py class DevtoolAddTests(DevtoolBase): · line 476

Quoted 2 lines
meta/lib/oeqa/selftest/cases/devtool.py +2 -0
+        # because the unresolved AUTOREV placeholder was used as the git
+        # revision to export patches from [YOCTO #16354]

There is no need to add a whole new test that largely copy-pastes other tests. There is already test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.

Alex

#5 lore raw

PATCH v2 1/2 devtool: standard: fix finish/update-recipe for recipes using AUTOREV

In reply to Babanpreet Singh

For a recipe with SRCREV = "${AUTOREV}" (e.g. created by devtool add --autorev), devtool finish and devtool update-recipe in srcrev mode crash:

INFO: Updating SRCREV in recipe minit_git.bb
...
oe.patch.CmdError: Command Error: 'git format-patch --no-signature --no-numbered AUTOINC -o /tmp/oepatchxu0ig0v0 -- .' exited with 128
Output: stderr: fatal: bad revision 'AUTOINC'

_update_recipe_srcrev() uses the recipe's SRCREV as the base revision for exporting patches from the source tree, but for AUTOREV recipes getVar('SRCREV') expands to the literal placeholder "AUTOINC", which is not a valid git revision. Use the initial revision(s) recorded in the workspace bbappend instead, as patch mode already does via _get_patchset_revs().

Do not replace SRCREV with the current source tree HEAD for such recipes either: pinning the revision on finish would silently drop the floating revision the user explicitly asked for.

This is not a recent regression: the same crash reproduces at least as far back as kirkstone, so it most likely dates back to the introduction of devtool add --autorev.

[YOCTO #16354]

AI-Generated: Uses Claude (claude-sonnet-5)

---
changes in v2:
- unchanged, rebased onto current master

 scripts/lib/devtool/standard.py | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +14 -2
index 2a3a62d081..386e628d88 100644
@@ -1530,7 +1530,18 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
     old_srcrev = rd.getVar('SRCREV') or ''
     if old_srcrev == "INVALID":
             raise DevtoolError('Update mode srcrev is only valid for recipe fetched from an SCM repository')
-    old_srcrev = {'.': old_srcrev}
+    autorev = old_srcrev == 'AUTOINC'
+    if autorev:
+        # SRCREV is set to "${AUTOREV}" so there is no fixed revision to
+        # use as the base for exporting patches; use the initial revision(s)
+        # recorded when the source tree was set up, as patch mode does
+        append = workspace[recipename]['bbappend']
+        old_srcrev, _, _, _ = _get_patchset_revs(srctree, append)
+        if not old_srcrev:
+            raise DevtoolError('Unable to find the initial revision of the '
+                               'source tree for %s in the workspace' % recipename)
+    else:
+        old_srcrev = {'.': old_srcrev}
 
     # Get HEAD revision
     try:
@@ -1545,7 +1556,8 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
     destpath = None
     remove_files = []
     patchfields = {}
-    patchfields['SRCREV'] = srcrev
+    if not autorev:
+        patchfields['SRCREV'] = srcrev
     orig_src_uri = rd.getVar('SRC_URI', False) or ''
     srcuri = orig_src_uri.split()
     tempdir = tempfile.mkdtemp(prefix='devtool')
Signature · 1 line
2.43.0
#6 lore raw

PATCH v2 2/2 oeqa/selftest/devtool: exercise devtool finish on an AUTOREV recipe

Adjust test_devtool_add_fetch_git() so that the recipe it creates with SRCREV = "${AUTOREV}" is finished into a temporary layer instead of just being reset: check that devtool finish succeeds (before the previous commit srcrev mode failed with "fatal: bad revision 'AUTOINC'"), that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated. The reset it replaces is covered by many other devtool tests.

srcrev mode is forced with --mode srcrev: update-mode guessing only considers git:// URLs, so for this gitsm:// recipe auto mode would select patch mode and never reach the code path being tested.

[YOCTO #16354]

Cc: Alexander Kanavin <[email protected]>
AI-Generated: Uses Claude (claude-sonnet-5)

---
changes in v2:
- extend the existing test_devtool_add_fetch_git() instead of adding a new test (Alexander Kanavin)
- force --mode srcrev in the test: update-mode guessing never selects srcrev for the gitsm:// URL used there (verified the adjusted test passes with the fix reverted if the mode is not forced, i.e. it would not catch the bug)

 meta/lib/oeqa/selftest/cases/devtool.py | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/devtool.py +20 -1
index b2cd13e9a5..51f9d6f8b7 100644
@@ -613,8 +613,27 @@ class DevtoolAddTests(DevtoolBase):
         checkvars['SRC_URI'] = url_branch
         checkvars['SRCREV'] = '${AUTOREV}'
         self._test_recipe_contents(recipefile, checkvars, [])
+        # Finish into a temporary layer instead of resetting; srcrev mode
+        # used to fail with "fatal: bad revision 'AUTOINC'" because the
+        # unresolved AUTOREV placeholder was used as the git revision to
+        # export patches from [YOCTO #16354]. srcrev mode has to be forced
+        # because update-mode guessing never selects it for gitsm:// URLs.
+        templayerdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(templayerdir)
+        self._create_temp_layer(templayerdir, False, 'selftestautorev')
+        result = runCmd('devtool finish --mode srcrev %s %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool status')
+        self.assertNotIn(testrecipe, result.output, 'Recipe should have been removed from workspace by devtool finish')
+        newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '%s_git.bb' % testrecipe))
+        self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish')
+        # The finished recipe should still use a floating revision and no
+        # patch files should have been created
+        checkvars = {}
+        checkvars['SRCREV'] = '${AUTOREV}'
+        self._test_recipe_contents(newrecipefiles[0], checkvars, [])
+        patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '*', '*.patch'))
+        self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish')
         # Try with revision and version specified
-        result = runCmd('devtool reset -n %s' % testrecipe)
         shutil.rmtree(srcdir)
         url_rev = '%s;rev=%s' % (url, checkrev)
         result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev))
Signature · 1 line
2.43.0
#7 lore raw

Re: [PATCH v2 2/2] oeqa/selftest/devtool: exercise devtool finish on an AUTOREV recipe

Thanks, I think this is good.

Alex

Quoted 65 lines · Babanpreet Singh

Adjust test_devtool_add_fetch_git() so that the recipe it creates with SRCREV = "${AUTOREV}" is finished into a temporary layer instead of just being reset: check that devtool finish succeeds (before the previous commit srcrev mode failed with "fatal: bad revision 'AUTOINC'"), that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated. The reset it replaces is covered by many other devtool tests.

srcrev mode is forced with --mode srcrev: update-mode guessing only considers git:// URLs, so for this gitsm:// recipe auto mode would select patch mode and never reach the code path being tested.

[YOCTO #16354]

Cc: Alexander Kanavin <[email protected]>
AI-Generated: Uses Claude (claude-sonnet-5)

---
changes in v2:
- extend the existing test_devtool_add_fetch_git() instead of adding a new test (Alexander Kanavin)
- force --mode srcrev in the test: update-mode guessing never selects srcrev for the gitsm:// URL used there (verified the adjusted test passes with the fix reverted if the mode is not forced, i.e. it would not catch the bug)

 meta/lib/oeqa/selftest/cases/devtool.py | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/devtool.py +20 -2
index b2cd13e9a5..51f9d6f8b7 100644
@@ -613,8 +613,27 @@ class DevtoolAddTests(DevtoolBase):
         checkvars['SRC_URI'] = url_branch
         checkvars['SRCREV'] = '${AUTOREV}'
         self._test_recipe_contents(recipefile, checkvars, [])
+        # Finish into a temporary layer instead of resetting; srcrev mode
+        # used to fail with "fatal: bad revision 'AUTOINC'" because the
+        # unresolved AUTOREV placeholder was used as the git revision to
+        # export patches from [YOCTO #16354]. srcrev mode has to be forced
+        # because update-mode guessing never selects it for gitsm:// URLs.
+        templayerdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(templayerdir)
+        self._create_temp_layer(templayerdir, False, 'selftestautorev')
+        result = runCmd('devtool finish --mode srcrev %s %s' % (testrecipe, templayerdir))
+        result = runCmd('devtool status')
+        self.assertNotIn(testrecipe, result.output, 'Recipe should have been removed from workspace by devtool finish')
+        newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '%s_git.bb' % testrecipe))
+        self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish')
+        # The finished recipe should still use a floating revision and no
+        # patch files should have been created
+        checkvars = {}
+        checkvars['SRCREV'] = '${AUTOREV}'
+        self._test_recipe_contents(newrecipefiles[0], checkvars, [])
+        patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '*', '*.patch'))
+        self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish')
         # Try with revision and version specified
-        result = runCmd('devtool reset -n %s' % testrecipe)
         shutil.rmtree(srcdir)
         url_rev = '%s;rev=%s' % (url, checkrev)
         result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev))
--

2.43.0