Several fixes around recipetool appendsrcfile(s) and oe.recipeutils.bbappend_recipe
cover Several fixes around recipetool appendsrcfile(s) and oe.recipeutils.bbappend_recipe
Hi all,
I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the new src_uri entry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itself
So this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)
There is still one point I would like to discuss:
I skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfile(s) used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the new src_entry (so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch mode
What do you think?
Cheers
Julien
Julien Stephan (7):
recipetool: appendsrcfile(s): add dry-run mode
recipeutils: bbappend_recipe: fix undefined variable
recipetool: appendsrcfile(s): use params instead of extraline
recipeutils: bbappend_recipe: fix file not added to SRC_URI if name is
specified
recipeutils: bbappend_recipe: allow to patch the recipe itself
recipetool: appendsrcfile(s): add a mode to update the recipe itself oeqa/selftest/recipetool: appendsrfile: add test for machine
meta/lib/oe/recipeutils.py | 51 +++++++++++------- meta/lib/oeqa/selftest/cases/recipetool.py | 30 +++++++++-- scripts/lib/recipetool/append.py | 60 ++++++++++++++++------ 3 files changed, 103 insertions(+), 38 deletions(-)
Signature · 1 line
2.42.0
PATCH 1/8 recipetool: appendsrcfile(s): add dry-run mode
Add dry-run mode for recipetool appendsrcfile and appendsrcfiles, with if necessary, a diff of changes
--- scripts/lib/recipetool/append.py | 33 +++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)
scripts/lib/recipetool/append.py +32 -1
@@ -18,6 +18,7 @@ import shutil import scriptutils import errno from collections import defaultdict +import difflib logger = logging.getLogger('recipetool')
@@ -355,7 +356,35 @@ def appendsrc(args, files, rd, extralines=None): extralines.append('SRC_URI += {0}'.format(source_uri)) copyfiles[newfile] = srcfile - oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines) + dry_run_output = None + dry_run_outdir = None + if args.dry_run: + import tempfile + dry_run_output = tempfile.TemporaryDirectory(prefix='devtool') + dry_run_outdir = dry_run_output.name + + appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, redirect_output=dry_run_outdir) + if args.dry_run: + output = '' + appendfilename = os.path.basename(appendfile) + newappendfile = appendfile + if appendfile and os.path.exists(appendfile): + with open(appendfile, 'r') as f: + oldlines = f.readlines() + else: + appendfile = '/dev/null' + oldlines = [] + + with open(os.path.join(dry_run_outdir, appendfilename), 'r') as f: + newlines = f.readlines() + diff = difflib.unified_diff(oldlines, newlines, appendfile, newappendfile) + difflines = list(diff) + if difflines: + output += ''.join(difflines) + if output: + logger.info('Diff of changed files:\n%s' % output) + else: + logger.info('No changed files') tinfoil.modified_files() def appendsrcfiles(parser, args):
@@ -436,6 +465,7 @@ def register_commands(subparsers): help='Create/update a bbappend to add or replace source files', description='Creates a bbappend (or updates an existing one) to add or replace the specified file in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify multiple files with a destination directory, so cannot specify the destination filename. See the `appendsrcfile` command for the other behavior.') parser.add_argument('-D', '--destdir', help='Destination directory (relative to S or WORKDIR, defaults to ".")', default='', type=destination_path) + parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true') parser.add_argument('files', nargs='+', metavar='FILE', help='File(s) to be added to the recipe sources (WORKDIR or S)', type=existing_path) parser.set_defaults(func=lambda a: appendsrcfiles(parser, a), parserecipes=True)
@@ -443,6 +473,7 @@ def register_commands(subparsers): parents=[common_src], help='Create/update a bbappend to add or replace a source file', description='Creates a bbappend (or updates an existing one) to add or replace the specified files in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify the destination filename, not just destination directory, but only works for one file. See the `appendsrcfiles` command for the other behavior.') + parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true') parser.add_argument('file', metavar='FILE', help='File to be added to the recipe sources (WORKDIR or S)', type=existing_path) parser.add_argument('destfile', metavar='DESTFILE', nargs='?', help='Destination path (relative to S or WORKDIR, optional)', type=destination_path) parser.set_defaults(func=lambda a: appendsrcfile(parser, a), parserecipes=True)
Signature · 1 line
2.42.0
PATCH 2/8 recipeutils: bbappend_recipe: fix undefined variable
In reply to Julien StephanIn the case get_bbappend_path returns None (could not find the layer containing the recipe) the error message tries to print the recipefile, but it is not defined. Fix it.
--- meta/lib/oe/recipeutils.py | 1 + 1 file changed, 1 insertion(+)
meta/lib/oe/recipeutils.py +1 -0
@@ -702,6 +702,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, if not removevalues: removevalues = {} + recipefile = rd.getVar('FILE') # Determine how the bbappend should be named appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver) if not appendpath:
Signature · 1 line
2.42.0
PATCH 3/8 recipeutils: bbappend_recipe: add a way to specify the name of the file to add
In reply to Julien Stephanbbappend_recipe can take a dict of source files to add to SRC_URI where the key is the full path to the file to be added and the value is the original file in the recipe space.
Modify the dict value to use a tupple: (originalfile, newname), to allow specifying the name of the file we would like to add
--- meta/lib/oe/recipeutils.py | 20 ++++++++++++-------- scripts/lib/devtool/standard.py | 6 +++--- scripts/lib/recipetool/append.py | 4 ++-- 3 files changed, 17 insertions(+), 13 deletions(-)
meta/lib/oe/recipeutils.py +12 -8
@@ -671,12 +671,13 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, rd: data dictionary for the recipe destlayerdir: base directory of the layer to place the bbappend in (subdirectory path from there will be determined automatically) - srcfiles: dict of source files to add to SRC_URI, where the value - is the full path to the file to be added, and the value is the - original filename as it would appear in SRC_URI or None if it - isn't already present. You may pass None for this parameter if - you simply want to specify your own content via the extralines - parameter. + srcfiles: dict of source files to add to SRC_URI, where the key + is the full path to the file to be added, and the value is a tuple + containing the original filename as it would appear in SRC_URI + or None if it isn't already present and the new name of the file or + None to use by default basename(original filename). + You may pass None for this parameter if you simply want to specify + your own content via the extralines parameter. install: dict mapping entries in srcfiles to a tuple of two elements: install path (*without* ${D} prefix) and permission value (as a string, e.g. '0644').
@@ -764,11 +765,14 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, copyfiles = {} if srcfiles: instfunclines = [] - for i, (newfile, origsrcfile) in enumerate(srcfiles.items()): + for i, (newfile, (origsrcfile, newname)) in enumerate(srcfiles.items()): srcfile = origsrcfile srcurientry = None if not srcfile: - srcfile = os.path.basename(newfile) + if newname: + srcfile = newname + else: + srcfile = os.path.basename(newfile) srcurientry = 'file://%s' % srcfile if params and params[i]: srcurientry = '%s;%s' % (srcurientry, ';'.join('%s=%s' % (k,v) for k,v in params[i].items()))
scripts/lib/devtool/standard.py +3 -3
@@ -1568,7 +1568,7 @@ def _update_recipe_srcrev(recipename, workspace, srctree, rd, appendlayerdir, wi update_srcuri = True if appendlayerdir: - files = dict((os.path.join(local_files_dir, key), val) for + files = dict((os.path.join(local_files_dir, key), (val, None)) for key, val in list(upd_f.items()) + list(new_f.items())) removevalues = {} if update_srcuri:
@@ -1678,9 +1678,9 @@ def _update_recipe_patch(recipename, workspace, srctree, rd, appendlayerdir, wil destpath = None srcuri = (rd.getVar('SRC_URI', False) or '').split() if appendlayerdir: - files = OrderedDict((os.path.join(local_files_dir, key), val) for + files = OrderedDict((os.path.join(local_files_dir, key), (val, None)) for key, val in list(upd_f.items()) + list(new_f.items())) - files.update(OrderedDict((os.path.join(patches_dir, key), val) for + files.update(OrderedDict((os.path.join(patches_dir, key), (val, None)) for key, val in list(upd_p.items()) + list(new_p.items()))) if files or remove_files: removevalues = None
scripts/lib/recipetool/append.py +2 -2
@@ -300,7 +300,7 @@ def appendfile(args): if st.st_mode & stat.S_IXUSR: perms = '0755' install = {args.newfile: (args.targetpath, perms)} - oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: sourcepath}, install, wildcardver=args.wildcard_version, machine=args.machine) + oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: (None, sourcepath)}, install, wildcardver=args.wildcard_version, machine=args.machine) tinfoil.modified_files() return 0 else:
@@ -354,7 +354,7 @@ def appendsrc(args, files, rd, extralines=None): logger.warning('{0!r} is already in SRC_URI, not adding'.format(source_uri)) else: extralines.append('SRC_URI += {0}'.format(source_uri)) - copyfiles[newfile] = srcfile + copyfiles[newfile] = (None, srcfile) dry_run_output = None dry_run_outdir = None
Signature · 1 line
2.42.0
PATCH 4/8 recipetool: appendsrcfile(s): use params instead of extraline
In reply to Julien Stephanappendsrc function relies on oe.recipeutils.bbappend_recipe to copy files and add the corresponding entries in SRC_URI.
Currently, appendsrc function build itself the new SRC_URI entry to add the correct subdir param, and gives it using the extralines parameter. This has 2 drawbacks:
- oe.recipeutils.bbappend_recipe can already do this if we specify the correct params, so we have duplicate code
- the duplicated code is not fully functional: for example, it doesn't take into account the -m/--machine parameter
So fix this by not using extralines but give correctly formatted params.
Also remove the check for already existing entries as oe.recipeutils.bbappend_recipe already implement it
The new bbappend file now have the SRC_URI entry after the FILESEXTRAPATHS so fix the selftest.
And skip test_recipetool_appendsrcfile_existing_in_src_uri_diff_params test because recipetool appendsrcfiles used to not add new src_uri entry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
--- meta/lib/oeqa/selftest/cases/recipetool.py | 10 ++++++++-- scripts/lib/recipetool/append.py | 23 ++++++++-------------- 2 files changed, 16 insertions(+), 17 deletions(-)
meta/lib/oeqa/selftest/cases/recipetool.py +8 -2
@@ -1083,12 +1083,13 @@ class RecipetoolAppendsrcBase(RecipetoolBase): expectedlines = ['FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"\n', '\n'] + if has_src_uri: uri = 'file://%s' % filename if expected_subdir: uri += ';subdir=%s' % expected_subdir - expectedlines[0:0] = ['SRC_URI += "%s"\n' % uri, - '\n'] + expectedlines.extend(['SRC_URI += "%s"\n' % uri, + '\n']) return self._try_recipetool_appendsrcfile(testrecipe, newfile, destpath, options, expectedlines, [filename])
@@ -1148,6 +1149,11 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase): self._test_appendsrcfile(testrecipe, filepath, has_src_uri=False) def test_recipetool_appendsrcfile_existing_in_src_uri_diff_params(self): + # recipetool appensrcfiles now relies on oe.recipeutils.bbappend_recipe to add src_uri entries, + # but oe.recipeutils.bbappend_recipe adds the new entry if the file already exist but have different parameters + # while before recipetool appendsrcfiles was rejecting it. So we need to figure out, which one is the correct way + # and if necessary patch oe.recipeutils.bbappend_recipe + self.skipTest("Skipping this test for now, because recipetool appendsrcfile changed his behaviour. Need to decide weither we should keep the old behaviour or not") testrecipe = 'base-files' subdir = 'tmp' filepath = self._get_first_file_uri(testrecipe)
scripts/lib/recipetool/append.py +8 -15
@@ -300,6 +300,8 @@ def appendfile(args): if st.st_mode & stat.S_IXUSR: perms = '0755' install = {args.newfile: (args.targetpath, perms)} + if sourcepath: + sourcepath = os.path.basename(sourcepath) oe.recipeutils.bbappend_recipe(rd, args.destlayer, {args.newfile: (None, sourcepath)}, install, wildcardver=args.wildcard_version, machine=args.machine) tinfoil.modified_files() return 0
@@ -329,6 +331,7 @@ def appendsrc(args, files, rd, extralines=None): copyfiles = {} extralines = extralines or [] + params = [] for newfile, srcfile in files.items(): src_destdir = os.path.dirname(srcfile) if not args.use_workdir:
@@ -339,22 +342,12 @@ def appendsrc(args, files, rd, extralines=None): src_destdir = os.path.join(os.path.relpath(srcdir, workdir), src_destdir) src_destdir = os.path.normpath(src_destdir) - source_uri = 'file://{0}'.format(os.path.basename(srcfile)) if src_destdir and src_destdir != '.': - source_uri += ';subdir={0}'.format(src_destdir) - - simple = bb.fetch.URI(source_uri) - simple.params = {} - simple_str = str(simple) - if simple_str in simplified: - existing = simplified[simple_str] - if source_uri != existing: - logger.warning('{0!r} is already in SRC_URI, with different parameters: {1!r}, not adding'.format(source_uri, existing)) - else: - logger.warning('{0!r} is already in SRC_URI, not adding'.format(source_uri)) + params.append({'subdir': src_destdir}) else: - extralines.append('SRC_URI += {0}'.format(source_uri)) - copyfiles[newfile] = (None, srcfile) + params.append({}) + + copyfiles[newfile] = (None, os.path.basename(srcfile)) dry_run_output = None dry_run_outdir = None
@@ -363,7 +356,7 @@ def appendsrc(args, files, rd, extralines=None): dry_run_output = tempfile.TemporaryDirectory(prefix='devtool') dry_run_outdir = dry_run_output.name - appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, redirect_output=dry_run_outdir) + appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir) if args.dry_run: output = '' appendfilename = os.path.basename(appendfile)
Signature · 1 line
2.42.0
Patchtest results for [PATCH 4/8] recipetool: appendsrcfile(s): use params instead of extraline
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/4-8-recipetool-appendsrcfile-s-use-params-instead-of-extraline.patch
FAIL: test max line length: Patch line too long (current length 214, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
PATCH 5/8 recipeutils: bbappend_recipe: allow to patch the recipe itself
In reply to Julien StephanAdd a new parameter update_original_recipe to allow to patch a recipe instead of creating/updating a bbappend
--- meta/lib/oe/recipeutils.py | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-)
meta/lib/oe/recipeutils.py +21 -9
@@ -664,7 +664,7 @@ def get_bbappend_path(d, destlayerdir, wildcardver=False): return (appendpath, pathok) -def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None, redirect_output=None, params=None): +def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, machine=None, extralines=None, removevalues=None, redirect_output=None, params=None, update_original_recipe=False): """ Writes a bbappend file for a recipe Parameters:
@@ -698,19 +698,29 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, params: Parameters to use when adding entries to SRC_URI. If specified, should be a list of dicts with the same length as srcfiles. + update_original_recipe: + Force to update the original recipe instead of creating/updating + a bbapend. destlayerdir must contain the original recipe """ if not removevalues: removevalues = {} recipefile = rd.getVar('FILE') - # Determine how the bbappend should be named - appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver) - if not appendpath: - bb.error('Unable to determine layer directory containing %s' % recipefile) - return (None, None) - if not pathok: - bb.warn('Unable to determine correct subdirectory path for bbappend file - check that what %s adds to BBFILES also matches .bbappend files. Using %s for now, but until you fix this the bbappend will not be applied.' % (os.path.join(destlayerdir, 'conf', 'layer.conf'), os.path.dirname(appendpath))) + if update_original_recipe: + if destlayerdir not in recipefile: + bb.error("destlayerdir %s doesn't contain the original recipe (%s), cannot update it" % (destlayerdir, recipefile)) + return (None, None) + + appendpath = recipefile + else: + # Determine how the bbappend should be named + appendpath, pathok = get_bbappend_path(rd, destlayerdir, wildcardver) + if not appendpath: + bb.error('Unable to determine layer directory containing %s' % recipefile) + return (None, None) + if not pathok: + bb.warn('Unable to determine correct subdirectory path for bbappend file - check that what %s adds to BBFILES also matches .bbappend files. Using %s for now, but until you fix this the bbappend will not be applied.' % (os.path.join(destlayerdir, 'conf', 'layer.conf'), os.path.dirname(appendpath))) appenddir = os.path.dirname(appendpath) if not redirect_output:
@@ -755,7 +765,7 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, bbappendlines.append((varname, op, value)) destsubdir = rd.getVar('PN') - if srcfiles: + if not update_original_recipe and srcfiles: bbappendlines.append(('FILESEXTRAPATHS:prepend', ':=', '${THISDIR}/${PN}:')) appendoverride = ''
@@ -803,6 +813,8 @@ def bbappend_recipe(rd, destlayerdir, srcfiles, install=None, wildcardver=False, # multiple times per operation when we're handling overrides) if os.path.exists(appendpath) and not os.path.exists(outfile): shutil.copy2(appendpath, outfile) + elif update_original_recipe: + outfile = recipefile else: bb.note('Writing append file %s' % appendpath) outfile = appendpath
Signature · 1 line
2.42.0
Patchtest results for [PATCH 5/8] recipeutils: bbappend_recipe: allow to patch the recipe itself
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/5-8-recipeutils-bbappend_recipe-allow-to-patch-the-recipe-itself.patch
FAIL: test max line length: Patch line too long (current length 310, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
PATCH 6/8 recipetool: appendsrcfile(s): add a mode to update the recipe itself
In reply to Julien StephanAdd a new option -u/--update-recipe to update the recipe itself instead of creating/updating a bbappend file
--- scripts/lib/recipetool/append.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/lib/recipetool/append.py +5 -1
@@ -356,7 +356,9 @@ def appendsrc(args, files, rd, extralines=None): dry_run_output = tempfile.TemporaryDirectory(prefix='devtool') dry_run_outdir = dry_run_output.name - appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir) + appendfile, _ = oe.recipeutils.bbappend_recipe(rd, args.destlayer, copyfiles, None, wildcardver=args.wildcard_version, machine=args.machine, extralines=extralines, params=params, redirect_output=dry_run_outdir, update_original_recipe=args.update_recipe) + if not appendfile: + return if args.dry_run: output = '' appendfilename = os.path.basename(appendfile)
@@ -458,6 +460,7 @@ def register_commands(subparsers): help='Create/update a bbappend to add or replace source files', description='Creates a bbappend (or updates an existing one) to add or replace the specified file in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify multiple files with a destination directory, so cannot specify the destination filename. See the `appendsrcfile` command for the other behavior.') parser.add_argument('-D', '--destdir', help='Destination directory (relative to S or WORKDIR, defaults to ".")', default='', type=destination_path) + parser.add_argument('-u', '--update-recipe', help='Update recipe instead of creating (or updating) a bbapend file. DESTLAYER must contains the recipe to update', action='store_true') parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true') parser.add_argument('files', nargs='+', metavar='FILE', help='File(s) to be added to the recipe sources (WORKDIR or S)', type=existing_path) parser.set_defaults(func=lambda a: appendsrcfiles(parser, a), parserecipes=True)
@@ -466,6 +469,7 @@ def register_commands(subparsers): parents=[common_src], help='Create/update a bbappend to add or replace a source file', description='Creates a bbappend (or updates an existing one) to add or replace the specified files in the recipe sources, either those in WORKDIR or those in the source tree. This command lets you specify the destination filename, not just destination directory, but only works for one file. See the `appendsrcfiles` command for the other behavior.') + parser.add_argument('-u', '--update-recipe', help='Update recipe instead of creating (or updating) a bbapend file. DESTLAYER must contains the recipe to update', action='store_true') parser.add_argument('-n', '--dry-run', help='Dry run mode', action='store_true') parser.add_argument('file', metavar='FILE', help='File to be added to the recipe sources (WORKDIR or S)', type=existing_path) parser.add_argument('destfile', metavar='DESTFILE', nargs='?', help='Destination path (relative to S or WORKDIR, optional)', type=destination_path)
Signature · 1 line
2.42.0
Patchtest results for [PATCH 6/8] recipetool: appendsrcfile(s): add a mode to update the recipe itself
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/6-8-recipetool-appendsrcfile-s-add-a-mode-to-update-the-recipe-itself.patch
FAIL: test max line length: Patch line too long (current length 257, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
PATCH 7/8 oeqa/selftest/recipetool: appendsrfile: add test for machine
In reply to Julien StephanAdd a new test for machine specific bbappend override
--- meta/lib/oeqa/selftest/cases/recipetool.py | 23 +++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/recipetool.py +20 -3
@@ -1056,7 +1056,7 @@ class RecipetoolAppendsrcBase(RecipetoolBase): if p.scheme == 'file': return p.netloc + p.path - def _test_appendsrcfile(self, testrecipe, filename=None, destdir=None, has_src_uri=True, srcdir=None, newfile=None, options=''): + def _test_appendsrcfile(self, testrecipe, filename=None, destdir=None, has_src_uri=True, srcdir=None, newfile=None, machine=None, options=''): if newfile is None: newfile = self.testfile
@@ -1084,12 +1084,26 @@ class RecipetoolAppendsrcBase(RecipetoolBase): expectedlines = ['FILESEXTRAPATHS:prepend := "${THISDIR}/${PN}:"\n', '\n'] + override = "" + if machine: + options += ' -m %s' % machine + override = ':append:%s' % machine + expectedlines.extend(['PACKAGE_ARCH = "${MACHINE_ARCH}"\n', + '\n']) + if has_src_uri: uri = 'file://%s' % filename if expected_subdir: uri += ';subdir=%s' % expected_subdir - expectedlines.extend(['SRC_URI += "%s"\n' % uri, - '\n']) + if machine: + src_uri_line = 'SRC_URI%s = " %s"\n' % (override, uri) + else: + src_uri_line = 'SRC_URI += "%s"\n' % uri + + expectedlines.extend([src_uri_line, '\n']) + + if machine: + filename = '%s/%s' % (machine, filename) return self._try_recipetool_appendsrcfile(testrecipe, newfile, destpath, options, expectedlines, [filename])
@@ -1162,6 +1176,9 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase): output = self._test_appendsrcfile(testrecipe, filepath, subdir, has_src_uri=False) self.assertTrue(any('with different parameters' in l for l in output)) + def test_recipetool_appendsrcfile_machine(self): + self._test_appendsrcfile('base-files', 'a-file', machine='mymachine') + def test_recipetool_appendsrcfile_replace_file_srcdir(self): testrecipe = 'bash' filepath = 'Makefile.in'
Signature · 1 line
2.42.0
PATCH 8/8 oeqa/selftest/recipetool: appendsrc: add test for update mode
In reply to Julien Stephanadd a basic test for testing the update mode of recipetool appendsrcfile(s)
--- meta/lib/oeqa/selftest/cases/recipetool.py | 28 ++++++++++++++++++++++ 1 file changed, 28 insertions(+)
meta/lib/oeqa/selftest/cases/recipetool.py +28 -0
@@ -1179,6 +1179,34 @@ class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase): def test_recipetool_appendsrcfile_machine(self): self._test_appendsrcfile('base-files', 'a-file', machine='mymachine') + def test_recipetool_appendsrcfile_update_recipe_basic(self): + testrecipe = "base-files" + recipefile = get_bb_var('FILE', testrecipe) + result = runCmd('bitbake-layers show-layers') + layerrecipe = None + for line in result.output.splitlines()[3:]: + with open("/tmp/juju.txt", "a") as file: + layer = line.split()[1] + print(layer, file=file) + if layer in recipefile: + layerrecipe = layer + break + self.assertTrue(layerrecipe, 'Unable to find the layer containing %s' % testrecipe) + cmd = 'recipetool appendsrcfile -u %s %s %s' % (layerrecipe, testrecipe, self.testfile) + result = runCmd(cmd) + self.assertNotIn('Traceback', result.output) + self.add_command_to_tearDown('cd %s; rm -f %s/%s; git checkout .' % (os.path.dirname(recipefile), testrecipe, os.path.basename(self.testfile))) + + expected_status = [(' M', '.*/%s$' % os.path.basename(recipefile)), + ('??', '.*/%s/%s/%s$' % (testrecipe, testrecipe, os.path.basename(self.testfile)))] + self._check_repo_status(os.path.dirname(recipefile), expected_status) + result = runCmd('git diff %s' % os.path.basename(recipefile), cwd=os.path.dirname(recipefile)) + removelines = [] + addlines = [ + 'file://%s \\\\' % os.path.basename(self.testfile), + ] + self._check_diff(result.output, addlines, removelines) + def test_recipetool_appendsrcfile_replace_file_srcdir(self): testrecipe = 'bash' filepath = 'Makefile.in'
Signature · 1 line
2.42.0
Hello Julien,
thanks again for making devtool better. I don't think there are any particularly strong opinions about the finer points; at this point you know the code better than anyone else, and so I trust you make the right calls :)
Alex
Quoted 61 lines · Julien Stephan
Hi all,
I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the newsrc_urientry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itselfSo this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)
There is still one point I would like to discuss:
I skiptest_recipetool_appendsrcfile_existing_in_src_uri_diff_paramstest because recipetool appendsrcfile(s) used to not add newsrc_urientry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the newsrc_entry(so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch modeWhat do you think?
Cheers
JulienJulien Stephan (7):
recipetool: appendsrcfile(s): add dry-run mode
recipeutils:bbappend_recipe: fix undefined variable
recipetool: appendsrcfile(s): use params instead of extraline
recipeutils:bbappend_recipe: fix file not added toSRC_URIif name is
specified
recipeutils:bbappend_recipe: allow to patch the recipe itself
recipetool: appendsrcfile(s): add a mode to update the recipe itself oeqa/selftest/recipetool: appendsrfile: add test for machinemeta/lib/oe/recipeutils.py | 51 +++++++++++------- meta/lib/oeqa/selftest/cases/recipetool.py | 30 +++++++++-- scripts/lib/recipetool/append.py | 60 ++++++++++++++++------ 3 files changed, 103 insertions(+), 38 deletions(-)--
2.42.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191544): https://lists.openembedded.org/g/openembedded-core/message/191544
Mute This Topic: https://lists.openembedded.org/mt/102903929/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 28 lines · Julien Stephan
Hi all,
I was trying to use recipetool appendsrcfile to add a file to a recipe and I noticed several issues:
* -m is not correctly supported
* recipetool appendsrfile(s) are missing a usefull dry-run mode
* appendsrc function relies on oe.recipeutils.bbappend_recipe but
duplicates some code: it constructs itself the newsrc_urientry
although oe.recipeutils.bbappend_recipe is already doing it
* we are lacking a mode to patch the recipe itselfSo this series tries to fix the issues above, fix the selftest accordingly and add new test for -m and for "patch mode" (update recipe) and also add a way to specify the name of the file to add (in oe.recipeutils.bbappend_recipe)
There is still one point I would like to discuss:
I skiptest_recipetool_appendsrcfile_existing_in_src_uri_diff_paramstest because recipetool appendsrcfile(s) used to not add newsrc_urientry if the entry already exist even with different parameters while oe.recipeutils.bbappend_recipe adds it if parameters are different. So we need to figure out if we want to keep the old behaviour and if we need to patch oe.recipeutils.bbappend_recipe, and update the test or remove it.
I feel like if the parameters are different then we should add the newsrc_entry(so old behaviour was not correct) and remove old entry so either use "SRC_URI:remove" in bbapend file or completely remove the old entry in patch mode
Quoted 1 line
What do you think?
I don't know the code well but I think that adding the src_uri if the parameters are different is the right thing to do. We should probably therefore update the tests accordingly.
Thanks for working through these kinds of details!
Cheers,
Richard