Skip to contentSkip to conversation

Several fixes around recipetool appendsrcfile(s) and oe.recipeutils.bbappend_recipe

RFC PATCH In review Julien Stephan 4 participants 14 messages Nov 30, 2023 – Dec 1, 2023

lore
Share & tools
lore
#0 lore raw

cover Several fixes around recipetool appendsrcfile(s) and oe.recipeutils.bbappend_recipe

Hi all,

I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the new src_uri entry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itself

So this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)

There is still one point I would like to discuss:
I skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfile(s) used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the new src_entry (so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch mode

What do you think?

Cheers
Julien

Julien Stephan (7):
recipetool: appendsrcfile(s): add dry-run mode
recipeutils: bbappend_recipe: fix undefined variable
recipetool: appendsrcfile(s): use params instead of extraline
recipeutils: bbappend_recipe: fix file not added to SRC_URI if name is
specified
recipeutils: bbappend_recipe: allow to patch the recipe itself
recipetool: appendsrcfile(s): add a mode to update the recipe itself oeqa/selftest/recipetool: appendsrfile: add test for machine

 meta/lib/oe/recipeutils.py                 | 51 +++++++++++-------
 meta/lib/oeqa/selftest/cases/recipetool.py | 30 +++++++++--
 scripts/lib/recipetool/append.py           | 60 ++++++++++++++++------
 3 files changed, 103 insertions(+), 38 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/8 recipetool: appendsrcfile(s): add dry-run mode

Add dry-run mode for recipetool appendsrcfile and appendsrcfiles, with if necessary, a diff of changes

---
 scripts/lib/recipetool/append.py | 33 +++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
scripts/lib/recipetool/append.py +32 -1
index 9dbb1cc4b5a..314ff4e20fd 100644
@@ -18,6 +18,7 @@ import shutil
 import scriptutils
 import errno
 from collections import defaultdict
+import difflib
 
 logger = logging.getLogger('recipetool')
 
@@ -355,7 +356,35 @@ def appendsrc(args, files, rd, extralines=None):
             extralines.append('SRC_URI += {0}'.format(source_uri))
         copyfiles[newfile] = srcfile
 
-    oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines)
+    dry_run_output = None
+    dry_run_outdir = None
+    if args.dry_run:
+        import tempfile
+        dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
+        dry_run_outdir = dry_run_output.name
+
+    appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, redirect_output=dry_run_outdir)
+    if args.dry_run:
+        output = ''
+        appendfilename = os.path.basename(appendfile)
+        newappendfile = appendfile
+        if appendfile and os.path.exists(appendfile):
+            with open(appendfile, 'r') as f:
+                oldlines = f.readlines()
+        else:
+            appendfile = '/dev/null'
+            oldlines = []
+
+        with open(os.path.join(dry_run_outdir, appendfilename), 'r') as f:
+            newlines = f.readlines()
+        diff = difflib.unified_diff(oldlines, newlines, appendfile, newappendfile)
+        difflines = list(diff)
+        if difflines:
+            output += ''.join(difflines)
+        if output:
+            logger.info('Diff of changed files:\n%s' % output)
+        else:
+            logger.info('No changed files')
     tinfoil.modified_files()
 
 def appendsrcfiles(parser, args):
@@ -436,6 +465,7 @@ def register_commands(subparsers):
                                    help='Create/update a bbappend to add or replace source files',
                                    description='Creates a bbappend (or updates an existing one) to add or replace the specified file in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify multiple files with a destination directory, so cannot specify the destination filename. See the `appendsrcfile` command for the other behavior.')
     parser.add_argument('-D', '--destdir', help='Destination directory (relative to S or WORKDIR, defaults to ".")', default='', type=destination_path)
+    parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true')
     parser.add_argument('files', nargs='+', metavar='FILE', help='File(s) to be added to the recipe sources (WORKDIR or S)', type=existing_path)
     parser.set_defaults(func=lambda a: appendsrcfiles(parser, a), parserecipes=True)
 
@@ -443,6 +473,7 @@ def register_commands(subparsers):
                                    parents=[common_src],
                                    help='Create/update a bbappend to add or replace a source file',
                                    description='Creates a bbappend (or updates an existing one) to add or replace the specified files in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify the destination filename, not just destination directory, but only works for one file. See the `appendsrcfiles` command for the other behavior.')
+    parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true')
     parser.add_argument('file', metavar='FILE', help='File to be added to the recipe sources (WORKDIR or S)', type=existing_path)
     parser.add_argument('destfile', metavar='DESTFILE', nargs='?', help='Destination path (relative to S or WORKDIR, optional)', type=destination_path)
     parser.set_defaults(func=lambda a: appendsrcfile(parser, a), parserecipes=True)
Signature · 1 line
2.42.0
#2 lore raw

PATCH 2/8 recipeutils: bbappend_recipe: fix undefined variable

In reply to Julien Stephan

In the case get_bbappend_path returns None (could not find the layer containing the recipe) the error message tries to print the recipefile, but it is not defined. Fix it.

---
 meta/lib/oe/recipeutils.py | 1 +
 1 file changed, 1 insertion(+)
meta/lib/oe/recipeutils.py +1 -0
index 3336db8ab06..7465a2d0ab4 100644
@@ -702,6 +702,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
     if not removevalues:
         removevalues = {}
 
+    recipefile = rd.getVar('FILE')
     # Determine how the bbappend should be named
     appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver)
     if not appendpath:
Signature · 1 line
2.42.0
#3 lore raw

PATCH 3/8 recipeutils: bbappend_recipe: add a way to specify the name of the file to add

In reply to Julien Stephan

bbappend_recipe can take a dict of source files to add to SRC_URI where the key is the full path to the file to be added and the value is the original file in the recipe space.

Modify the dict value to use a tupple: (originalfile, newname), to allow specifying the name of the file we would like to add

---
 meta/lib/oe/recipeutils.py       | 20 ++++++++++++--------
 scripts/lib/devtool/standard.py  |  6 +++---
 scripts/lib/recipetool/append.py |  4 ++--
 3 files changed, 17 insertions(+), 13 deletions(-)
meta/lib/oe/recipeutils.py +12 -8
index 7465a2d0ab4..9784ae3b031 100644
@@ -671,12 +671,13 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
         rd: data dictionary for the recipe
         destlayerdir: base directory of the layer to place the bbappend in
             (subdirectory path from there will be determined automatically)
-        srcfiles: dict of source files to add to SRC_URI, where the value
-            is the full path to the file to be added, and the value is the
-            original filename as it would appear in SRC_URI or None if it
-            isn't already present. You may pass None for this parameter if
-            you simply want to specify your own content via the extralines
-            parameter.
+        srcfiles: dict of source files to add to SRC_URI, where the key
+            is the full path to the file to be added, and the value is a tuple
+            containing the original filename as it would appear in SRC_URI
+            or None if it isn't already present and the new name of the file or
+            None to use by default basename(original filename).
+            You may pass None for this parameter if you simply want to specify
+            your own content via the extralines parameter.
         install: dict mapping entries in srcfiles to a tuple of two elements:
             install path (*without* ${D} prefix) and permission value (as a
             string, e.g. '0644').
@@ -764,11 +765,14 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
     copyfiles = {}
     if srcfiles:
         instfunclines = []
-        for i, (newfile, origsrcfile) in enumerate(srcfiles.items()):
+        for i, (newfile, (origsrcfile, newname)) in enumerate(srcfiles.items()):
             srcfile = origsrcfile
             srcurientry = None
             if not srcfile:
-                srcfile = os.path.basename(newfile)
+                if newname:
+                    srcfile = newname
+                else:
+                    srcfile = os.path.basename(newfile)
                 srcurientry = 'file://%s' % srcfile
                 if params and params[i]:
                     srcurientry = '%s;%s' % (srcurientry, ';'.join('%s=%s' % (k,v) for k,v in params[i].items()))
scripts/lib/devtool/standard.py +3 -3
index d53fb810071..fa473014f90 100644
@@ -1568,7 +1568,7 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi
                 update_srcuri = True
 
         if appendlayerdir:
-            files = dict((os.path.join(local_files_dir, key), val) for
+            files = dict((os.path.join(local_files_dir, key), (val, None)) for
                           key, val in list(upd_f.items()) + list(new_f.items()))
             removevalues = {}
             if update_srcuri:
@@ -1678,9 +1678,9 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil
         destpath = None
         srcuri = (rd.getVar('SRC_URI', False) or '').split()
         if appendlayerdir:
-            files = OrderedDict((os.path.join(local_files_dir, key), val) for
+            files = OrderedDict((os.path.join(local_files_dir, key), (val, None)) for
                          key, val in list(upd_f.items()) + list(new_f.items()))
-            files.update(OrderedDict((os.path.join(patches_dir, key), val) for
+            files.update(OrderedDict((os.path.join(patches_dir, key), (val, None)) for
                               key, val in list(upd_p.items()) + list(new_p.items())))
             if files or remove_files:
                 removevalues = None
scripts/lib/recipetool/append.py +2 -2
index 314ff4e20fd..09e314481f1 100644
@@ -300,7 +300,7 @@ def appendfile(args):
                 if st.st_mode & stat.S_IXUSR:
                     perms = '0755'
             install = {args.newfile: (args.targetpath, perms)}
-        oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: sourcepath}, install, wildcardver=args.wildcard_version, machine=args.machine)
+        oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: (None, sourcepath)}, install, wildcardver=args.wildcard_version, machine=args.machine)
         tinfoil.modified_files()
         return 0
     else:
@@ -354,7 +354,7 @@ def appendsrc(args, files, rd, extralines=None):
                 logger.warning('{0!r} is already in SRC_URI, not adding'.format(source_uri))
         else:
             extralines.append('SRC_URI += {0}'.format(source_uri))
-        copyfiles[newfile] = srcfile
+        copyfiles[newfile] = (None, srcfile)
 
     dry_run_output = None
     dry_run_outdir = None
Signature · 1 line
2.42.0
#4 lore raw

PATCH 4/8 recipetool: appendsrcfile(s): use params instead of extraline

In reply to Julien Stephan

appendsrc function relies on oe.recipeutils.bbappend_recipe to copy files and add the corresponding entries in SRC_URI.

Currently, appendsrc function build itself the new SRC_URI entry to add the correct subdir param, and gives it using the extralines parameter. This has 2 drawbacks:
- oe.recipeutils.bbappend_recipe can already do this if we specify the correct params, so we have duplicate code
- the duplicated code is not fully functional: for example, it doesn't take into account the -m/--machine parameter

So fix this by not using extralines but give correctly formatted params.

Also remove the check for already existing entries as oe.recipeutils.bbappend_recipe already implement it

The new bbappend file now have the SRC_URI entry after the FILESEXTRAPATHS so fix the selftest.

And skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfiles used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.

---
 meta/lib/oeqa/selftest/cases/recipetool.py | 10 ++++++++--
 scripts/lib/recipetool/append.py           | 23 ++++++++--------------
 2 files changed, 16 insertions(+), 17 deletions(-)
meta/lib/oeqa/selftest/cases/recipetool.py +8 -2
index 55cbba9ca74..b445a40cb10 100644
@@ -1083,12 +1083,13 @@ class RecipetoolAppendsrcBase(RecipetoolBase):
 
         expectedlines = ['FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"\n',
                          '\n']
+
         if has_src_uri:
             uri = 'file://%s' % filename
             if expected_subdir:
                 uri += ';subdir=%s' % expected_subdir
-            expectedlines[0:0] = ['SRC_URI += "%s"\n' % uri,
-                                  '\n']
+            expectedlines.extend(['SRC_URI += "%s"\n' % uri,
+                                  '\n'])
 
         return self._try_recipetool_appendsrcfile(testrecipe, newfile, destpath, options, expectedlines, [filename])
 
@@ -1148,6 +1149,11 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase):
         self._test_appendsrcfile(testrecipe, filepath, has_src_uri=False)
 
     def test_recipetool_appendsrcfile_existing_in_src_uri_diff_params(self):
+        # recipetool appensrcfiles now relies on oe.recipeutils.bbappend_recipe to add src_uri entries,
+        # but oe.recipeutils.bbappend_recipe adds the new entry if the file already exist but have different parameters
+        # while before recipetool appendsrcfiles was rejecting it. So we need to figure out, which one is the correct way
+        # and if necessary patch oe.recipeutils.bbappend_recipe
+        self.skipTest("Skipping this test for now, because recipetool appendsrcfile changed his behaviour. Need to decide weither we should keep the old behaviour or not")
         testrecipe = 'base-files'
         subdir = 'tmp'
         filepath = self._get_first_file_uri(testrecipe)
scripts/lib/recipetool/append.py +8 -15
index 09e314481f1..d7597ca5154 100644
@@ -300,6 +300,8 @@ def appendfile(args):
                 if st.st_mode & stat.S_IXUSR:
                     perms = '0755'
             install = {args.newfile: (args.targetpath, perms)}
+        if sourcepath:
+            sourcepath = os.path.basename(sourcepath)
         oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: (None, sourcepath)}, install, wildcardver=args.wildcard_version, machine=args.machine)
         tinfoil.modified_files()
         return 0
@@ -329,6 +331,7 @@ def appendsrc(args, files, rd, extralines=None):
 
     copyfiles = {}
     extralines = extralines or []
+    params = []
     for newfile, srcfile in files.items():
         src_destdir = os.path.dirname(srcfile)
         if not args.use_workdir:
@@ -339,22 +342,12 @@ def appendsrc(args, files, rd, extralines=None):
             src_destdir = os.path.join(os.path.relpath(srcdir, workdir), src_destdir)
         src_destdir = os.path.normpath(src_destdir)
 
-        source_uri = 'file://{0}'.format(os.path.basename(srcfile))
         if src_destdir and src_destdir != '.':
-            source_uri += ';subdir={0}'.format(src_destdir)
-
-        simple = bb.fetch.URI(source_uri)
-        simple.params = {}
-        simple_str = str(simple)
-        if simple_str in simplified:
-            existing = simplified[simple_str]
-            if source_uri != existing:
-                logger.warning('{0!r} is already in SRC_URI, with different parameters: {1!r}, not adding'.format(source_uri, existing))
-            else:
-                logger.warning('{0!r} is already in SRC_URI, not adding'.format(source_uri))
+            params.append({'subdir': src_destdir})
         else:
-            extralines.append('SRC_URI += {0}'.format(source_uri))
-        copyfiles[newfile] = (None, srcfile)
+            params.append({})
+
+        copyfiles[newfile] = (None, os.path.basename(srcfile))
 
     dry_run_output = None
     dry_run_outdir = None
@@ -363,7 +356,7 @@ def appendsrc(args, files, rd, extralines=None):
         dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
         dry_run_outdir = dry_run_output.name
 
-    appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, redirect_output=dry_run_outdir)
+    appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir)
     if args.dry_run:
         output = ''
         appendfilename = os.path.basename(appendfile)
Signature · 1 line
2.42.0
#5 lore raw

Patchtest results for [PATCH 4/8] recipetool: appendsrcfile(s): use params instead of extraline

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/4-8-recipetool-appendsrcfile-s-use-params-instead-of-extraline.patch

FAIL: test max line length: Patch line too long (current length 214, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#6 lore raw

PATCH 5/8 recipeutils: bbappend_recipe: allow to patch the recipe itself

In reply to Julien Stephan

Add a new parameter update_original_recipe to allow to patch a recipe instead of creating/updating a bbappend

---
 meta/lib/oe/recipeutils.py | 30 +++++++++++++++++++++---------
 1 file changed, 21 insertions(+), 9 deletions(-)
meta/lib/oe/recipeutils.py +21 -9
index 9784ae3b031..c768c5a4582 100644
@@ -664,7 +664,7 @@ def get_bbappend_path(d, destlayerdir, wildcardver=False):
     return (appendpath, pathok)
 
 
-def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None, redirect_output=None, params=None):
+def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None, redirect_output=None, params=None, update_original_recipe=False):
     """
     Writes a bbappend file for a recipe
     Parameters:
@@ -698,19 +698,29 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
         params:
             Parameters to use when adding entries to SRC_URI. If specified,
             should be a list of dicts with the same length as srcfiles.
+        update_original_recipe:
+            Force to update the original recipe instead of creating/updating
+            a bbapend. destlayerdir must contain the original recipe
     """
 
     if not removevalues:
         removevalues = {}
 
     recipefile = rd.getVar('FILE')
-    # Determine how the bbappend should be named
-    appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver)
-    if not appendpath:
-        bb.error('Unable to determine layer directory containing %s' % recipefile)
-        return (None, None)
-    if not pathok:
-        bb.warn('Unable to determine correct subdirectory path for bbappend file - check that what %s adds to BBFILES also matches .bbappend files. Using %s for now, but until you fix this the bbappend will not be applied.' % (os.path.join(destlayerdir, 'conf', 'layer.conf'), os.path.dirname(appendpath)))
+    if update_original_recipe:
+        if destlayerdir not in recipefile:
+            bb.error("destlayerdir %s doesn't contain the original recipe (%s), cannot update it" % (destlayerdir, recipefile))
+            return (None, None)
+
+        appendpath = recipefile
+    else:
+        # Determine how the bbappend should be named
+        appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver)
+        if not appendpath:
+            bb.error('Unable to determine layer directory containing %s' % recipefile)
+            return (None, None)
+        if not pathok:
+            bb.warn('Unable to determine correct subdirectory path for bbappend file - check that what %s adds to BBFILES also matches .bbappend files. Using %s for now, but until you fix this the bbappend will not be applied.' % (os.path.join(destlayerdir, 'conf', 'layer.conf'), os.path.dirname(appendpath)))
 
     appenddir = os.path.dirname(appendpath)
     if not redirect_output:
@@ -755,7 +765,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
             bbappendlines.append((varname, op, value))
 
     destsubdir = rd.getVar('PN')
-    if srcfiles:
+    if not update_original_recipe and srcfiles:
         bbappendlines.append(('FILESEXTRAPATHS:prepend', ':=', '${THISDIR}/${PN}:'))
 
     appendoverride = ''
@@ -803,6 +813,8 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False,
         # multiple times per operation when we're handling overrides)
         if os.path.exists(appendpath) and not os.path.exists(outfile):
             shutil.copy2(appendpath, outfile)
+    elif update_original_recipe:
+        outfile = recipefile
     else:
         bb.note('Writing append file %s' % appendpath)
         outfile = appendpath
Signature · 1 line
2.42.0
#7 lore raw

Patchtest results for [PATCH 5/8] recipeutils: bbappend_recipe: allow to patch the recipe itself

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/5-8-recipeutils-bbappend_recipe-allow-to-patch-the-recipe-itself.patch

FAIL: test max line length: Patch line too long (current length 310, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#8 lore raw

PATCH 6/8 recipetool: appendsrcfile(s): add a mode to update the recipe itself

In reply to Julien Stephan

Add a new option -u/--update-recipe to update the recipe itself instead of creating/updating a bbappend file

---
 scripts/lib/recipetool/append.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/lib/recipetool/append.py +5 -1
index d7597ca5154..6c4c9183f78 100644
@@ -356,7 +356,9 @@ def appendsrc(args, files, rd, extralines=None):
         dry_run_output = tempfile.TemporaryDirectory(prefix='devtool')
         dry_run_outdir = dry_run_output.name
 
-    appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir)
+    appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir, update_original_recipe=args.update_recipe)
+    if not appendfile:
+        return
     if args.dry_run:
         output = ''
         appendfilename = os.path.basename(appendfile)
@@ -458,6 +460,7 @@ def register_commands(subparsers):
                                    help='Create/update a bbappend to add or replace source files',
                                    description='Creates a bbappend (or updates an existing one) to add or replace the specified file in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify multiple files with a destination directory, so cannot specify the destination filename. See the `appendsrcfile` command for the other behavior.')
     parser.add_argument('-D', '--destdir', help='Destination directory (relative to S or WORKDIR, defaults to ".")', default='', type=destination_path)
+    parser.add_argument('-u', '--update-recipe', help='Update recipe instead of creating (or updating) a bbapend file. DESTLAYER must contains the recipe to update', action='store_true')
     parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true')
     parser.add_argument('files', nargs='+', metavar='FILE', help='File(s) to be added to the recipe sources (WORKDIR or S)', type=existing_path)
     parser.set_defaults(func=lambda a: appendsrcfiles(parser, a), parserecipes=True)
@@ -466,6 +469,7 @@ def register_commands(subparsers):
                                    parents=[common_src],
                                    help='Create/update a bbappend to add or replace a source file',
                                    description='Creates a bbappend (or updates an existing one) to add or replace the specified files in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify the destination filename, not just destination directory, but only works for one file. See the `appendsrcfiles` command for the other behavior.')
+    parser.add_argument('-u', '--update-recipe', help='Update recipe instead of creating (or updating) a bbapend file. DESTLAYER must contains the recipe to update', action='store_true')
     parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true')
     parser.add_argument('file', metavar='FILE', help='File to be added to the recipe sources (WORKDIR or S)', type=existing_path)
     parser.add_argument('destfile', metavar='DESTFILE', nargs='?', help='Destination path (relative to S or WORKDIR, optional)', type=destination_path)
Signature · 1 line
2.42.0
#9 lore raw

Patchtest results for [PATCH 6/8] recipetool: appendsrcfile(s): add a mode to update the recipe itself

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/6-8-recipetool-appendsrcfile-s-add-a-mode-to-update-the-recipe-itself.patch

FAIL: test max line length: Patch line too long (current length 257, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#10 lore raw

PATCH 7/8 oeqa/selftest/recipetool: appendsrfile: add test for machine

In reply to Julien Stephan

Add a new test for machine specific bbappend override

---
 meta/lib/oeqa/selftest/cases/recipetool.py | 23 +++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/recipetool.py +20 -3
index b445a40cb10..7d5ac4b47a5 100644
@@ -1056,7 +1056,7 @@ class RecipetoolAppendsrcBase(RecipetoolBase):
             if p.scheme == 'file':
                 return p.netloc + p.path
 
-    def _test_appendsrcfile(self, testrecipe, filename=None, destdir=None, has_src_uri=True, srcdir=None, newfile=None, options=''):
+    def _test_appendsrcfile(self, testrecipe, filename=None, destdir=None, has_src_uri=True, srcdir=None, newfile=None, machine=None, options=''):
         if newfile is None:
             newfile = self.testfile
 
@@ -1084,12 +1084,26 @@ class RecipetoolAppendsrcBase(RecipetoolBase):
         expectedlines = ['FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"\n',
                          '\n']
 
+        override = ""
+        if machine:
+            options += ' -m %s' % machine
+            override = ':append:%s' % machine
+            expectedlines.extend(['PACKAGE_ARCH = "${MACHINE_ARCH}"\n',
+                                  '\n'])
+
         if has_src_uri:
             uri = 'file://%s' % filename
             if expected_subdir:
                 uri += ';subdir=%s' % expected_subdir
-            expectedlines.extend(['SRC_URI += "%s"\n' % uri,
-                                  '\n'])
+            if machine:
+                src_uri_line = 'SRC_URI%s = " %s"\n' % (override, uri)
+            else:
+                src_uri_line = 'SRC_URI += "%s"\n' % uri
+
+            expectedlines.extend([src_uri_line, '\n'])
+
+        if machine:
+            filename = '%s/%s' % (machine, filename)
 
         return self._try_recipetool_appendsrcfile(testrecipe, newfile, destpath, options, expectedlines, [filename])
 
@@ -1162,6 +1176,9 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase):
         output = self._test_appendsrcfile(testrecipe, filepath, subdir, has_src_uri=False)
         self.assertTrue(any('with different parameters' in l for l in output))
 
+    def test_recipetool_appendsrcfile_machine(self):
+        self._test_appendsrcfile('base-files', 'a-file', machine='mymachine')
+
     def test_recipetool_appendsrcfile_replace_file_srcdir(self):
         testrecipe = 'bash'
         filepath = 'Makefile.in'
Signature · 1 line
2.42.0
#11 lore raw

PATCH 8/8 oeqa/selftest/recipetool: appendsrc: add test for update mode

In reply to Julien Stephan

add a basic test for testing the update mode of recipetool appendsrcfile(s)

---
 meta/lib/oeqa/selftest/cases/recipetool.py | 28 ++++++++++++++++++++++
 1 file changed, 28 insertions(+)
meta/lib/oeqa/selftest/cases/recipetool.py +28 -0
index 7d5ac4b47a5..7a114335821 100644
@@ -1179,6 +1179,34 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase):
     def test_recipetool_appendsrcfile_machine(self):
         self._test_appendsrcfile('base-files', 'a-file', machine='mymachine')
 
+    def test_recipetool_appendsrcfile_update_recipe_basic(self):
+        testrecipe = "base-files"
+        recipefile = get_bb_var('FILE', testrecipe)
+        result = runCmd('bitbake-layers show-layers')
+        layerrecipe = None
+        for line in result.output.splitlines()[3:]:
+            with open("/tmp/juju.txt", "a") as file:
+                layer = line.split()[1]
+                print(layer, file=file)
+                if layer in recipefile:
+                    layerrecipe = layer
+                    break
+        self.assertTrue(layerrecipe, 'Unable to find the layer containing %s' % testrecipe)
+        cmd = 'recipetool appendsrcfile -u %s %s %s' % (layerrecipe, testrecipe, self.testfile)
+        result = runCmd(cmd)
+        self.assertNotIn('Traceback', result.output)
+        self.add_command_to_tearDown('cd %s; rm -f %s/%s; git checkout .' % (os.path.dirname(recipefile), testrecipe, os.path.basename(self.testfile)))
+
+        expected_status = [(' M', '.*/%s$' % os.path.basename(recipefile)),
+                           ('??', '.*/%s/%s/%s$' % (testrecipe, testrecipe, os.path.basename(self.testfile)))]
+        self._check_repo_status(os.path.dirname(recipefile), expected_status)
+        result = runCmd('git diff %s' % os.path.basename(recipefile), cwd=os.path.dirname(recipefile))
+        removelines = []
+        addlines = [
+            'file://%s \\\\' % os.path.basename(self.testfile),
+        ]
+        self._check_diff(result.output, addlines, removelines)
+
     def test_recipetool_appendsrcfile_replace_file_srcdir(self):
         testrecipe = 'bash'
         filepath = 'Makefile.in'
Signature · 1 line
2.42.0
#12 lore raw
In reply to Julien Stephan

Hello Julien,

thanks again for making devtool better. I don't think there are any particularly strong opinions about the finer points; at this point you know the code better than anyone else, and so I trust you make the right calls :)

Alex

Quoted 61 lines · Julien Stephan

Hi all,

I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the new src_uri entry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itself

So this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)

There is still one point I would like to discuss:
I skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfile(s) used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the new src_entry (so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch mode

What do you think?

Cheers
Julien

Julien Stephan (7):
recipetool: appendsrcfile(s): add dry-run mode
recipeutils: bbappend_recipe: fix undefined variable
recipetool: appendsrcfile(s): use params instead of extraline
recipeutils: bbappend_recipe: fix file not added to SRC_URI if name is
specified
recipeutils: bbappend_recipe: allow to patch the recipe itself
recipetool: appendsrcfile(s): add a mode to update the recipe itself oeqa/selftest/recipetool: appendsrfile: add test for machine

 meta/lib/oe/recipeutils.py                 | 51 +++++++++++-------
 meta/lib/oeqa/selftest/cases/recipetool.py | 30 +++++++++--
 scripts/lib/recipetool/append.py           | 60 ++++++++++++++++------
 3 files changed, 103 insertions(+), 38 deletions(-)

--
2.42.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191544): https://lists.openembedded.org/g/openembedded-core/message/191544
Mute This Topic: https://lists.openembedded.org/mt/102903929/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#13 lore raw
In reply to Julien Stephan
Quoted 28 lines · Julien Stephan

Hi all,

I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the new src_uri entry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itself

So this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)

There is still one point I would like to discuss:
I skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfile(s) used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the new src_entry (so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch mode

Quoted 1 line

What do you think?

I don't know the code well but I think that adding the src_uri if the parameters are different is the right thing to do. We should probably therefore update the tests accordingly.

Thanks for working through these kinds of details!

Cheers,

Richard