Skip to contentSkip to conversation

devtool: standard: Fix file copy on finish --force

PATCH In review Mathieu Dubois-Briand 2 participants 6 messages Aug 16

lore
Share & tools
lore
#0 lore raw

v2 cover devtool: standard: Fix file copy on finish --force

Fix "devtool finish --force" command, by allowing to copy folders that were not previously existing but also by preventing it from trying to remove non-existent files. Devtool was particularly confused when using the finish subcommand on a recipe that was just added.

Reproducer:

devtool add --version 2.10 https://ftp.gnu.org/gnu/hello/hello-2.10.tar.gz
...
devtool build hello
...
devtool finish hello -f ../openembedded-core/meta/
...
Traceback (most recent call last):
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/devtool", line 352, in <module>
ret = main()
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/devtool", line 338, in main
ret = args.func(args, config, basepath, workspace)
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 2214, in finish
updated, appendfile, removed = _update_recipe(args.recipename, workspace, rd, args.mode, appendlayerdir, wildcard_version=True, no_remove=False, no_report_remove=removing_original, initial_rev=args.initial_rev, dry_run_outdir=dry_run_outdir, no_overrides=args.no_overrides, force_patch_refresh=args.force_patch_refresh)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1903, in _update_recipe
updated, appendf, removed = _update_recipe_patch(recipename, workspace, srctree, crd, appendlayerdir, wildcard_version, no_remove, no_report_remove, initial_rev, dry_run_outdir, force_patch_refresh)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1675, in _update_recipe_patch
upd_f, new_f, del_f = _export_local_files(srctree, rd, local_files_dir, srctreebase)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mathieu/projects/swat/gits/openembedded-core/scripts/lib/devtool/standard.py", line 1469, in _export_local_files
shutil.copy2(fullfile, os.path.join(destdir, f))
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/shutil.py", line 468, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/shutil.py", line 262, in copyfile
with open(dst, 'wb') as fdst:
~~~~^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/devtool77jejua2/tmpyuo3c_ay/build-aux/compile'

Using --force is arguably a bad idea here, but as we do provide this possibility, we should make sure the code does not crash.

---
Changes in v2:
- Use RecipetoolTransient instead of Proprietary.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Mathieu Dubois-Briand (4):
scripts: scriptutils: Use LicenseRef-RecipetoolTransient LICENSE while fetching
devtool: standard: Fix file copy on finish --force
devtool: standard: Remove unused variable
oe-selftest: devtool: Add test for add/finish workflow

 meta/files/common-licenses/RecipetoolTransient |  1 +
 meta/lib/oeqa/selftest/cases/devtool.py        | 58 ++++++++++++++++++++++++++
 scripts/lib/devtool/standard.py                |  9 ++--
 scripts/lib/scriptutils.py                     |  2 +-
 4 files changed, 66 insertions(+), 4 deletions(-)

---
base-commit: 07a342aa80c7349dd014f743a695a5e006add8df
change-id: 20260813-mathieu-devtool-9b0b2768098f

Best regards,

Signature · 1 line
Mathieu Dubois-Briand <[email protected]>
#1 lore raw

PATCH v2 1/4 scripts: scriptutils: Use LicenseRef-RecipetoolTransient LICENSE while fetching

Switching away of the "CLOSED" license, as devtool users would encounter deprecation warnings on "devtool add" command.

---
 meta/files/common-licenses/RecipetoolTransient | 1 +
 scripts/lib/scriptutils.py                     | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
meta/files/common-licenses/RecipetoolTransient new file +1 -0
new file mode 100644
index 000000000000..3c59fbb8a580
@@ -0,0 +1 @@
+Recipetool transient license.
scripts/lib/scriptutils.py +1 -1
index 32e749dbb1a2..33e197a22cc7 100644
@@ -170,7 +170,7 @@ def fetch_url(tinfoil, srcuri, srcrev, destdir, logger, preserve_tmp=False, mirr
             logger.debug('Generating initial recipe %s for fetching' % fetchrecipe)
             with open(fetchrecipe, 'w') as f:
                 # We don't want to have to specify LIC_FILES_CHKSUM
-                f.write('LICENSE = "CLOSED"\n')
+                f.write('LICENSE = "LicenseRef-RecipetoolTransient"\n')
                 # We don't need the cross-compiler
                 f.write('INHIBIT_DEFAULT_DEPS = "1"\n')
                 # We don't have the checksums yet so we can't require them
Signature · 1 line
2.47.3
#2 lore raw

PATCH v2 2/4 devtool: standard: Fix file copy on finish --force

In reply to Mathieu Dubois-Briand

Fix "devtool finish --force" command, by allowing to copy folders that were not previously existing but also by preventing it from trying to remove non-existent files. Devtool was particularly confused when using the finish subcommand on a recipe that was just added.

Fixes ce8190c51905: devtool: Drop oe-local-files and simplify
---
 scripts/lib/devtool/standard.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
scripts/lib/devtool/standard.py +6 -2
index 9b2643604d1d..536411efef6a 100644
@@ -1461,13 +1461,17 @@ def _export_local_files(srctree, rd, destdir, srctreebase):
         if os.path.exists(os.path.join(fullfile, ".git")):
             # submodules handled elsewhere
             continue
-        if f not in existing_files:
+        if f not in existing_files and os.path.exists(fullfile):
             added[f] = {}
+            parentdir = os.path.normpath(os.path.join(destdir, f, os.pardir))
+            if not os.path.isdir(parentdir):
+                os.makedirs(parentdir)
+
             if os.path.isdir(os.path.join(srctree, f)):
                 shutil.copytree(fullfile, os.path.join(destdir, f))
             else:
                 shutil.copy2(fullfile, os.path.join(destdir, f))
-        elif not os.path.exists(fullfile):
+        elif f in existing_files and not os.path.exists(fullfile):
             removed[f] = existing_files[f]
         elif f in existing_files:
             updated[f] = {'path' : existing_files[f]}
Signature · 1 line
2.47.3
#3 lore raw

PATCH v2 3/4 devtool: standard: Remove unused variable

In reply to Mathieu Dubois-Briand
---
 scripts/lib/devtool/standard.py | 1 -
 1 file changed, 1 deletion(-)
scripts/lib/devtool/standard.py +0 -1
index 536411efef6a..ac9e7cd61a58 100644
@@ -1437,7 +1437,6 @@ def _export_local_files(srctree, rd, destdir, srctreebase):
     # recipe space).
     existing_files = oe.recipeutils.get_recipe_local_files(rd)
 
-    new_set = None
     updated = OrderedDict()
     added = OrderedDict()
     removed = OrderedDict()
Signature · 1 line
2.47.3
#4 lore raw

Patchtest results for [PATCH v2 3/4] devtool: standard: Remove unused variable

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/v2-3-4-devtool-standard-Remove-unused-variable.patch

FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#5 lore raw

PATCH v2 4/4 oe-selftest: devtool: Add test for add/finish workflow

In reply to Mathieu Dubois-Briand

Add test for devtool add/build/finish workflow, inclusing a test for finish --force.

---
 meta/lib/oeqa/selftest/cases/devtool.py | 58 +++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
meta/lib/oeqa/selftest/cases/devtool.py +58 -0
index d84a18e8b69b..d549be9f7965 100644
@@ -713,6 +713,64 @@ class DevtoolAddTests(DevtoolBase):
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
         result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url))
 
+    def test_devtool_add_build_finish(self):
+        url = 'https://ftp.gnu.org/gnu/hello/hello-2.12.3.tar.gz'
+        pn = 'hello'
+        recipe = 'hello_2.12.3.bb'
+        # Test devtool add
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool add %s' % (url, ))
+        self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'),
+                          'Workspace directory not created')
+        # Test devtool build
+        result = runCmd('devtool build %s' % pn)
+        bb_vars = get_bb_vars(['D', 'bindir'], pn)
+        installdir = bb_vars['D']
+        self.assertTrue(installdir, 'Could not query installdir variable')
+        # devtool finish
+        result = runCmd('devtool finish %s meta-selftest' % pn,
+                        ignore_status=True)
+        self.assertNotEqual(result.status, 0,
+                            'devtool finish should have failed on a dirty directory. devtool output: %s' %
+                            (result.output))
+        # clean directory
+        sourcedir = os.path.join(self.workspacedir, 'sources', 'hello')
+        runCmd('git reset --hard; git clean -dxf', cwd=sourcedir)
+        # devtool finish (again)
+        self.assertExists(os.path.join(self.workspacedir, 'recipes', pn),
+                          'Recipe directory should exist before finish')
+        result = runCmd('devtool finish %s meta-selftest' % pn)
+        self.assertNotExists(os.path.join(self.workspacedir, 'recipes', pn),
+                             'Recipe directory should not exist after finish')
+        # Check recipe got created as expected
+        self.assertExists(os.path.join(get_test_layer(), f'recipes-{pn}', pn, recipe),
+                          'Recipe should exist in the layer after finish')
+
+    def test_devtool_add_build_finish_force(self):
+        url = 'https://ftp.gnu.org/gnu/hello/hello-2.12.3.tar.gz'
+        pn = 'hello'
+        recipe = 'hello_2.12.3.bb'
+        # Test devtool add
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake -c cleansstate %s' % pn)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        result = runCmd('devtool add %s' % (url, ))
+        self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'),
+                          'Workspace directory not created')
+        # Test devtool build
+        result = runCmd('devtool build %s' % pn)
+        bb_vars = get_bb_vars(['D', 'bindir'], pn)
+        installdir = bb_vars['D']
+        self.assertTrue(installdir, 'Could not query installdir variable')
+        # devtool finish
+        result = runCmd('devtool finish %s meta-selftest --force' % pn)
+        # Check recipe got created as expected
+        self.assertExists(os.path.join(get_test_layer(), f'recipes-{pn}', pn, recipe),
+                          'Recipe should exist in the layer after finish')
+
+
 class DevtoolModifyTests(DevtoolBase):
 
     def test_devtool_modify(self):
Signature · 1 line
2.47.3