devtool: standard: Fix file copy on finish --force
v2 cover devtool: standard: Fix file copy on finish --force
Fix "devtool finish --force" command, by allowing to copy folders that were not previously existing but also by preventing it from trying to remove non-existent files. Devtool was particularly confused when using the finish subcommand on a recipe that was just added.
Reproducer:
devtool add --version 2.10 https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
...
devtool build hello
...
devtool finish hello -f ../openembedded-core/meta/
...
Traceback (most recent call last):
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/devtool", line 352, in <module>
ret = main()
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/devtool", line 338, in main
ret = args.func(args, config, basepath, workspace)
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 2214, in finish
updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1903, in _update_recipe
updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir, force_patch_refresh)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1675, in _update_recipe_patch
upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1469, in _export_local_files
shutil.copy2(fullfile, os.path.join(destdir, f))
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/shutil.py", line 468, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/shutil.py", line 262, in copyfile
with open(dst, 'wb') as fdst:
~~~~^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/devtool77jejua2/tmpyuo3c_ay/build-aux/compile'
Using --force is arguably a bad idea here, but as we do provide this possibility, we should make sure the code does not crash.
---
Changes in v2:
- Use RecipetoolTransient instead of Proprietary.
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Mathieu Dubois-Briand (4):
scripts: scriptutils: Use LicenseRef-RecipetoolTransient LICENSE while fetching
devtool: standard: Fix file copy on finish --force
devtool: standard: Remove unused variable
oe-selftest: devtool: Add test for add/finish workflow
meta/files/common-licenses/RecipetoolTransient | 1 + meta/lib/oeqa/selftest/cases/devtool.py | 58 ++++++++++++++++++++++++++ scripts/lib/devtool/standard.py | 9 ++-- scripts/lib/scriptutils.py | 2 +- 4 files changed, 66 insertions(+), 4 deletions(-)
---
base-commit: 07a342aa80c7349dd014f743a695a5e006add8df
change-id: 20260813-mathieu-devtool-9b0b2768098f
Best regards,
Signature · 1 line
Mathieu Dubois-Briand <[email protected]>
PATCH v2 1/4 scripts: scriptutils: Use LicenseRef-RecipetoolTransient LICENSE while fetching
Switching away of the "CLOSED" license, as devtool users would encounter deprecation warnings on "devtool add" command.
--- meta/files/common-licenses/RecipetoolTransient | 1 + scripts/lib/scriptutils.py | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
meta/files/common-licenses/RecipetoolTransient new file +1 -0
@@ -0,0 +1 @@ +Recipetool transient license.
scripts/lib/scriptutils.py +1 -1
@@ -170,7 +170,7 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirr logger.debug('Generating initial recipe %s for fetching' % fetchrecipe) with open(fetchrecipe, 'w') as f: # We don't want to have to specify LIC_FILES_CHKSUM - f.write('LICENSE = "CLOSED"\n') + f.write('LICENSE = "LicenseRef-RecipetoolTransient"\n') # We don't need the cross-compiler f.write('INHIBIT_DEFAULT_DEPS = "1"\n') # We don't have the checksums yet so we can't require them
Signature · 1 line
2.47.3
PATCH v2 2/4 devtool: standard: Fix file copy on finish --force
In reply to Mathieu Dubois-BriandFix "devtool finish --force" command, by allowing to copy folders that were not previously existing but also by preventing it from trying to remove non-existent files. Devtool was particularly confused when using the finish subcommand on a recipe that was just added.
--- scripts/lib/devtool/standard.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +6 -2
@@ -1461,13 +1461,17 @@ def _export_local_files(srctree, rd, destdir, srctreebase): if os.path.exists(os.path.join(fullfile, ".git")): # submodules handled elsewhere continue - if f not in existing_files: + if f not in existing_files and os.path.exists(fullfile): added[f] = {} + parentdir = os.path.normpath(os.path.join(destdir, f, os.pardir)) + if not os.path.isdir(parentdir): + os.makedirs(parentdir) + if os.path.isdir(os.path.join(srctree, f)): shutil.copytree(fullfile, os.path.join(destdir, f)) else: shutil.copy2(fullfile, os.path.join(destdir, f)) - elif not os.path.exists(fullfile): + elif f in existing_files and not os.path.exists(fullfile): removed[f] = existing_files[f] elif f in existing_files: updated[f] = {'path' : existing_files[f]}
Signature · 1 line
2.47.3
PATCH v2 3/4 devtool: standard: Remove unused variable
In reply to Mathieu Dubois-Briand--- scripts/lib/devtool/standard.py | 1 - 1 file changed, 1 deletion(-)
scripts/lib/devtool/standard.py +0 -1
@@ -1437,7 +1437,6 @@ def _export_local_files(srctree, rd, destdir, srctreebase): # recipe space). existing_files = oe.recipeutils.get_recipe_local_files(rd) - new_set = None updated = OrderedDict() added = OrderedDict() removed = OrderedDict()
Signature · 1 line
2.47.3
PATCH v2 4/4 oe-selftest: devtool: Add test for add/finish workflow
In reply to Mathieu Dubois-BriandAdd test for devtool add/build/finish workflow, inclusing a test for finish --force.
--- meta/lib/oeqa/selftest/cases/devtool.py | 58 +++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +58 -0
@@ -713,6 +713,64 @@ class DevtoolAddTests(DevtoolBase): self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url)) + def test_devtool_add_build_finish(self): + url = 'https://ftp.gnu.org/gnu/hello/hello-2.12.3.tar.gz' + pn = 'hello' + recipe = 'hello_2.12.3.bb' + # Test devtool add + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool add %s' % (url, )) + self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), + 'Workspace directory not created') + # Test devtool build + result = runCmd('devtool build %s' % pn) + bb_vars = get_bb_vars(['D', 'bindir'], pn) + installdir = bb_vars['D'] + self.assertTrue(installdir, 'Could not query installdir variable') + # devtool finish + result = runCmd('devtool finish %s meta-selftest' % pn, + ignore_status=True) + self.assertNotEqual(result.status, 0, + 'devtool finish should have failed on a dirty directory. devtool output: %s' % + (result.output)) + # clean directory + sourcedir = os.path.join(self.workspacedir, 'sources', 'hello') + runCmd('git reset --hard; git clean -dxf', cwd=sourcedir) + # devtool finish (again) + self.assertExists(os.path.join(self.workspacedir, 'recipes', pn), + 'Recipe directory should exist before finish') + result = runCmd('devtool finish %s meta-selftest' % pn) + self.assertNotExists(os.path.join(self.workspacedir, 'recipes', pn), + 'Recipe directory should not exist after finish') + # Check recipe got created as expected + self.assertExists(os.path.join(get_test_layer(), f'recipes-{pn}', pn, recipe), + 'Recipe should exist in the layer after finish') + + def test_devtool_add_build_finish_force(self): + url = 'https://ftp.gnu.org/gnu/hello/hello-2.12.3.tar.gz' + pn = 'hello' + recipe = 'hello_2.12.3.bb' + # Test devtool add + self.track_for_cleanup(self.workspacedir) + self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn) + self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') + result = runCmd('devtool add %s' % (url, )) + self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), + 'Workspace directory not created') + # Test devtool build + result = runCmd('devtool build %s' % pn) + bb_vars = get_bb_vars(['D', 'bindir'], pn) + installdir = bb_vars['D'] + self.assertTrue(installdir, 'Could not query installdir variable') + # devtool finish + result = runCmd('devtool finish %s meta-selftest --force' % pn) + # Check recipe got created as expected + self.assertExists(os.path.join(get_test_layer(), f'recipes-{pn}', pn, recipe), + 'Recipe should exist in the layer after finish') + + class DevtoolModifyTests(DevtoolBase): def test_devtool_modify(self):
Signature · 1 line
2.47.3
Patchtest results for [PATCH v2 3/4] devtool: standard: Remove unused variable
In reply to Mathieu Dubois-BriandThank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/v2-3-4-devtool-standard-Remove-unused-variable.patch
FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!