devtool: fix finish for recipes using AUTOREV
cover devtool: fix finish for recipes using AUTOREV
devtool finish (and devtool update-recipe in srcrev mode) crashes with "fatal: bad revision 'AUTOINC'" on any recipe with SRCREV = "${AUTOREV}", e.g. one created with devtool add --autorev, because the unresolved AUTOREV placeholder is passed to git format-patch as the base revision for exporting patches.
Patch 1 uses the initial revision(s) recorded in the workspace bbappend instead (as patch mode already does), and stops replacing the AUTOREV with a pinned revision on finish. Patch 2 adds the oe-selftest case Randy asked for in the bug; it fails with the exact reported error before patch 1 and passes after.
While investigating I verified this is not a recent regression: the same crash reproduces on wrynose, whinlatter, scarthgap and kirkstone, so it most likely dates back to the introduction of --autorev.
Tested on oe-core master in a qemux86-64 build environment:
- devtool add --autorev + devtool finish: fails before, passes after, finished recipe keeps SRCREV = "${AUTOREV}"
- devtool add (pinned) + devtool finish: unchanged
- oe-selftest -r devtool.DevtoolAddTests.test_devtool_add_git_autorev_finish (new): PASSED
- oe-selftest -r devtool.DevtoolUpdateTests.test_devtool_update_recipe_git devtool.DevtoolUpdateTests.test_devtool_update_recipe_append_git: PASSED
[YOCTO #16354]
Babanpreet Singh (2):
devtool: standard: fix finish/update-recipe for recipes using AUTOREV oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV
meta/lib/oeqa/selftest/cases/devtool.py | 32 +++++++++++++++++++++++++ scripts/lib/devtool/standard.py | 16 +++++++++++-- 2 files changed, 46 insertions(+), 2 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/2 devtool: standard: fix finish/update-recipe for recipes using AUTOREV
For a recipe with SRCREV = "${AUTOREV}" (e.g. created by devtool add --autorev), devtool finish and devtool update-recipe in srcrev mode crash:
INFO: Updating SRCREV in recipe minit_git.bb
...
oe.patch.CmdError: Command Error: 'git format-patch --no-signature --no-numbered AUTOINC -o /tmp/oepatchxu0ig0v0 -- .' exited with 128
Output: stderr: fatal: bad revision 'AUTOINC'
_update_recipe_srcrev() uses the recipe's SRCREV as the base revision for exporting patches from the source tree, but for AUTOREV recipes getVar('SRCREV') expands to the literal placeholder "AUTOINC", which is not a valid git revision. Use the initial revision(s) recorded in the workspace bbappend instead, as patch mode already does via _get_patchset_revs().
Do not replace SRCREV with the current source tree HEAD for such recipes either: pinning the revision on finish would silently drop the floating revision the user explicitly asked for.
This is not a recent regression: the same crash reproduces at least as far back as kirkstone, so it most likely dates back to the introduction of devtool add --autorev.
[YOCTO #16354]
AI-Generated: Uses Claude (claude-sonnet-5)
--- scripts/lib/devtool/standard.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +14 -2
@@ -1530,7 +1530,18 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi old_srcrev = rd.getVar('SRCREV') or '' if old_srcrev == "INVALID": raise DevtoolError('Update mode srcrev is only valid for recipe fetched from an SCM repository') - old_srcrev = {'.': old_srcrev} + autorev = old_srcrev == 'AUTOINC' + if autorev: + # SRCREV is set to "${AUTOREV}" so there is no fixed revision to + # use as the base for exporting patches; use the initial revision(s) + # recorded when the source tree was set up, as patch mode does + append = workspace[recipename]['bbappend'] + old_srcrev, _, _, _ = _get_patchset_revs(srctree, append) + if not old_srcrev: + raise DevtoolError('Unable to find the initial revision of the ' + 'source tree for %s in the workspace' % recipename) + else: + old_srcrev = {'.': old_srcrev} # Get HEAD revision try:
@@ -1545,7 +1556,8 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi destpath = None remove_files = [] patchfields = {} - patchfields['SRCREV'] = srcrev + if not autorev: + patchfields['SRCREV'] = srcrev orig_src_uri = rd.getVar('SRC_URI', False) or '' srcuri = orig_src_uri.split() tempdir = tempfile.mkdtemp(prefix='devtool')
Signature · 1 line
2.43.0
PATCH 2/2 oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV
In reply to Babanpreet SinghAdd a test checking that a recipe created with devtool add --autorev can be finished into a layer, that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated.
Before the previous commit, devtool finish failed on such recipes with:
fatal: bad revision 'AUTOINC'
Test requested by Randy MacLeod in the bug report.
[YOCTO #16354]
AI-Generated: Uses Claude (claude-sonnet-5)
--- meta/lib/oeqa/selftest/cases/devtool.py | 32 +++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +32 -0
@@ -469,6 +469,38 @@ class DevtoolAddTests(DevtoolBase): checkvars['DEPENDS'] = set(['dbus']) self._test_recipe_contents(recipefile, checkvars, []) + def test_devtool_add_git_autorev_finish(self): + # Test that a recipe created by devtool add --autorev can be finished + # into a layer; this used to fail with "fatal: bad revision 'AUTOINC'" + # because the unresolved AUTOREV placeholder was used as the git + # revision to export patches from [YOCTO #16354] + pn = 'dbus-wait' + url = 'git://git.yoctoproject.org/dbus-wait;protocol=https;branch=master' + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd("devtool add --autorev %s '%s'" % (pn, url)) + recipefile = os.path.join(self.workspacedir, 'recipes', pn, '%s_git.bb' % pn) + self.assertExists(recipefile, 'Recipe not created by devtool add') + checkvars = {} + checkvars['SRCREV'] = '${AUTOREV}' + self._test_recipe_contents(recipefile, checkvars, []) + # Finish the recipe into a temporary layer + templayerdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(templayerdir) + self._create_temp_layer(templayerdir, False, 'selftestautorev') + result = runCmd('devtool finish %s %s' % (pn, templayerdir)) + result = runCmd('devtool status') + self.assertNotIn(pn, result.output, 'Recipe should have been reset by finish but wasn\'t') + newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', pn, '%s_git.bb' % pn)) + self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish') + # The recipe should still use a floating revision and no patch + # files should have been created + checkvars = {} + checkvars['SRCREV'] = '${AUTOREV}' + self._test_recipe_contents(newrecipefiles[0], checkvars, []) + patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', pn, '*', '*.patch')) + self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish') + def test_devtool_add_git_style1(self): version = 'v3.1.0' pn = 'mbedtls'
Signature · 1 line
2.43.0
Re: [OE-core] [PATCH 2/2] oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV
meta/lib/oeqa/selftest/cases/devtool.py class DevtoolAddTests(DevtoolBase): · line 476
Quoted 6 lines · Baban via lists.openembedded.org
AI-Generated: Uses Claude (claude-sonnet-5)
meta/lib/oeqa/selftest/cases/devtool.py +5
-0+ def test_devtool_add_git_autorev_finish(self): + # Test that a recipe created by devtool add --autorev can be finished + # into a layer; this used to fail with "fatal: bad revision 'AUTOINC'" + # because the unresolved AUTOREV placeholder was used as the git + # revision to export patches from [YOCTO #16354]
There is no need to add a whole new test that largely copy-pastes other tests. There is already test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.
Alex
Re: [OE-core] [PATCH 2/2] oeqa/selftest/devtool: add test for finishing a recipe using AUTOREV
Quoted 4 lines · Alexander Kanavin
There is no need to add a whole new test that largely copy-pastes other tests. There is already
test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.
Fair point - the add/check portion did duplicate what that test already does. I'll send a v2 that drops the new test and instead adjust test_devtool_add_fetch_git().
Thanks for the review,
Baban
Quoted 18 lines · Alexander Kanavin
Review
Quoted 3 lines · Baban via lists.openembedded.org
AI-Generated: Uses Claude (claude-sonnet-5)
(quoted diff) +2
-0+ def test_devtool_add_git_autorev_finish(self): + # Test that a recipe created by devtool add --autorev can befinished
Quoted 1 line
+ # into a layer; this used to fail with "fatal: bad revision
'AUTOINC'"
meta/lib/oeqa/selftest/cases/devtool.py class DevtoolAddTests(DevtoolBase): · line 476
Quoted 2 lines
meta/lib/oeqa/selftest/cases/devtool.py +2
-0+ # because the unresolved AUTOREV placeholder was used as the git + # revision to export patches from [YOCTO #16354]There is no need to add a whole new test that largely copy-pastes other tests. There is already
test_devtool_add_fetch_git(), which adds a recipe with AUTOREV, but then runs 'reset' instead of 'finish', so that test should instead be adjusted.Alex
PATCH v2 1/2 devtool: standard: fix finish/update-recipe for recipes using AUTOREV
In reply to Babanpreet SinghFor a recipe with SRCREV = "${AUTOREV}" (e.g. created by devtool add --autorev), devtool finish and devtool update-recipe in srcrev mode crash:
INFO: Updating SRCREV in recipe minit_git.bb
...
oe.patch.CmdError: Command Error: 'git format-patch --no-signature --no-numbered AUTOINC -o /tmp/oepatchxu0ig0v0 -- .' exited with 128
Output: stderr: fatal: bad revision 'AUTOINC'
_update_recipe_srcrev() uses the recipe's SRCREV as the base revision for exporting patches from the source tree, but for AUTOREV recipes getVar('SRCREV') expands to the literal placeholder "AUTOINC", which is not a valid git revision. Use the initial revision(s) recorded in the workspace bbappend instead, as patch mode already does via _get_patchset_revs().
Do not replace SRCREV with the current source tree HEAD for such recipes either: pinning the revision on finish would silently drop the floating revision the user explicitly asked for.
This is not a recent regression: the same crash reproduces at least as far back as kirkstone, so it most likely dates back to the introduction of devtool add --autorev.
[YOCTO #16354]
AI-Generated: Uses Claude (claude-sonnet-5)
---
changes in v2:
- unchanged, rebased onto current master
scripts/lib/devtool/standard.py | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +14 -2
@@ -1530,7 +1530,18 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi old_srcrev = rd.getVar('SRCREV') or '' if old_srcrev == "INVALID": raise DevtoolError('Update mode srcrev is only valid for recipe fetched from an SCM repository') - old_srcrev = {'.': old_srcrev} + autorev = old_srcrev == 'AUTOINC' + if autorev: + # SRCREV is set to "${AUTOREV}" so there is no fixed revision to + # use as the base for exporting patches; use the initial revision(s) + # recorded when the source tree was set up, as patch mode does + append = workspace[recipename]['bbappend'] + old_srcrev, _, _, _ = _get_patchset_revs(srctree, append) + if not old_srcrev: + raise DevtoolError('Unable to find the initial revision of the ' + 'source tree for %s in the workspace' % recipename) + else: + old_srcrev = {'.': old_srcrev} # Get HEAD revision try:
@@ -1545,7 +1556,8 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi destpath = None remove_files = [] patchfields = {} - patchfields['SRCREV'] = srcrev + if not autorev: + patchfields['SRCREV'] = srcrev orig_src_uri = rd.getVar('SRC_URI', False) or '' srcuri = orig_src_uri.split() tempdir = tempfile.mkdtemp(prefix='devtool')
Signature · 1 line
2.43.0
PATCH v2 2/2 oeqa/selftest/devtool: exercise devtool finish on an AUTOREV recipe
Adjust test_devtool_add_fetch_git() so that the recipe it creates with SRCREV = "${AUTOREV}" is finished into a temporary layer instead of just being reset: check that devtool finish succeeds (before the previous commit srcrev mode failed with "fatal: bad revision 'AUTOINC'"), that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated. The reset it replaces is covered by many other devtool tests.
srcrev mode is forced with --mode srcrev: update-mode guessing only considers git:// URLs, so for this gitsm:// recipe auto mode would select patch mode and never reach the code path being tested.
[YOCTO #16354]
Cc: Alexander Kanavin <[email protected]>
AI-Generated: Uses Claude (claude-sonnet-5)
---
changes in v2:
- extend the existing test_devtool_add_fetch_git() instead of adding a new test (Alexander Kanavin)
- force --mode srcrev in the test: update-mode guessing never selects srcrev for the gitsm:// URL used there (verified the adjusted test passes with the fix reverted if the mode is not forced, i.e. it would not catch the bug)
meta/lib/oeqa/selftest/cases/devtool.py | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/devtool.py +20 -1
@@ -613,8 +613,27 @@ class DevtoolAddTests(DevtoolBase): checkvars['SRC_URI'] = url_branch checkvars['SRCREV'] = '${AUTOREV}' self._test_recipe_contents(recipefile, checkvars, []) + # Finish into a temporary layer instead of resetting; srcrev mode + # used to fail with "fatal: bad revision 'AUTOINC'" because the + # unresolved AUTOREV placeholder was used as the git revision to + # export patches from [YOCTO #16354]. srcrev mode has to be forced + # because update-mode guessing never selects it for gitsm:// URLs. + templayerdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(templayerdir) + self._create_temp_layer(templayerdir, False, 'selftestautorev') + result = runCmd('devtool finish --mode srcrev %s %s' % (testrecipe, templayerdir)) + result = runCmd('devtool status') + self.assertNotIn(testrecipe, result.output, 'Recipe should have been removed from workspace by devtool finish') + newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '%s_git.bb' % testrecipe)) + self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish') + # The finished recipe should still use a floating revision and no + # patch files should have been created + checkvars = {} + checkvars['SRCREV'] = '${AUTOREV}' + self._test_recipe_contents(newrecipefiles[0], checkvars, []) + patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '*', '*.patch')) + self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish') # Try with revision and version specified - result = runCmd('devtool reset -n %s' % testrecipe) shutil.rmtree(srcdir) url_rev = '%s;rev=%s' % (url, checkrev) result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev))
Signature · 1 line
2.43.0
Re: [PATCH v2 2/2] oeqa/selftest/devtool: exercise devtool finish on an AUTOREV recipe
Thanks, I think this is good.
Alex
Quoted 65 lines · Babanpreet Singh
Adjust
test_devtool_add_fetch_git()so that the recipe it creates with SRCREV = "${AUTOREV}" is finished into a temporary layer instead of just being reset: check that devtool finish succeeds (before the previous commit srcrev mode failed with "fatal: bad revision 'AUTOINC'"), that the finished recipe still uses SRCREV = "${AUTOREV}", and that no spurious patch files are generated. The reset it replaces is covered by many other devtool tests.srcrev mode is forced with --mode srcrev: update-mode guessing only considers
git://URLs, so for this gitsm:// recipe auto mode would select patch mode and never reach the code path being tested.[YOCTO #16354]
Cc: Alexander Kanavin <[email protected]>
AI-Generated: Uses Claude (claude-sonnet-5)Signed-off-by Babanpreet Singh <[email protected]>---
changes in v2:
- extend the existingtest_devtool_add_fetch_git()instead of adding a new test (Alexander Kanavin)
- force --mode srcrev in the test: update-mode guessing never selects srcrev for the gitsm:// URL used there (verified the adjusted test passes with the fix reverted if the mode is not forced, i.e. it would not catch the bug)meta/lib/oeqa/selftest/cases/devtool.py | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)meta/lib/oeqa/selftest/cases/devtool.py +20
-2@@ -613,8 +613,27 @@ class DevtoolAddTests(DevtoolBase): checkvars['SRC_URI'] = url_branch checkvars['SRCREV'] = '${AUTOREV}' self._test_recipe_contents(recipefile, checkvars, []) + # Finish into a temporary layer instead of resetting; srcrev mode + # used to fail with "fatal: bad revision 'AUTOINC'" because the + # unresolved AUTOREV placeholder was used as the git revision to + # export patches from [YOCTO #16354]. srcrev mode has to be forced + # because update-mode guessing never selects it for gitsm:// URLs. + templayerdir = tempfile.mkdtemp(prefix='devtoolqa') + self.track_for_cleanup(templayerdir) + self._create_temp_layer(templayerdir, False, 'selftestautorev') + result = runCmd('devtool finish --mode srcrev %s %s' % (testrecipe, templayerdir)) + result = runCmd('devtool status') + self.assertNotIn(testrecipe, result.output, 'Recipe should have been removed from workspace by devtool finish') + newrecipefiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '%s_git.bb' % testrecipe)) + self.assertEqual(len(newrecipefiles), 1, 'Recipe not moved into destination layer by devtool finish') + # The finished recipe should still use a floating revision and no + # patch files should have been created + checkvars = {} + checkvars['SRCREV'] = '${AUTOREV}' + self._test_recipe_contents(newrecipefiles[0], checkvars, []) + patchfiles = glob.glob(os.path.join(templayerdir, 'recipes-*', testrecipe, '*', '*.patch')) + self.assertEqual(patchfiles, [], 'Unexpected patch files generated by devtool finish') # Try with revision and version specified - result = runCmd('devtool reset -n %s' % testrecipe) shutil.rmtree(srcdir) url_rev = '%s;rev=%s' % (url, checkrev) result = runCmd('devtool add %s %s -f "%s" -V 1.5' % (testrecipe, srcdir, url_rev)) --2.43.0