Devtool/Recipetool: adding pypi support
cover Devtool/Recipetool: adding pypi support
Hello all,
This series adds basic support for the pypi class in devtool/recipetool.
The idea is to be able to detect that a package is available on pypi and in that case inherits from pypi class. This helps to produce cleaner recipes for pypi packages.
To do this, I am adding a new optional callback "process_url" that plugin can register to implement some logic on matching url.
By implementing this callback in create_buildsys_python we can match on pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".
To create a recipe taking advantage of the pypi class we can use one of the following new syntax:
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>
or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>
If the URL points to a github URL or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching URL, to keep legacy behaviour.
This series also contains some bug fixes I found during my testing.
Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support
Cheers
Julien
Julien Stephan (7):
bitbake: utils: remove spaces on empty lines
recipetool: create_buildsys_python.py: initialize metadata
recipetool: create: add trailing newlines
recipetool: create: add new optional process_url callback for plugins
recipetool: create_buildsys_python: add pypi support oeqa/selftest/recipetool: remove spaces on empty lines oeqa/selftest/recipetool/devtool: add test for pypi class
bitbake/lib/bb/utils.py | 16 +-- meta/lib/oeqa/selftest/cases/devtool.py | 4 +- meta/lib/oeqa/selftest/cases/recipetool.py | 114 +++++++++++++----- scripts/lib/devtool/standard.py | 3 + scripts/lib/recipetool/create.py | 63 ++++++---- .../lib/recipetool/create_buildsys_python.py | 73 +++++++++++ 6 files changed, 208 insertions(+), 65 deletions(-)
Signature · 1 line
2.42.0
PATCH RESEND 1/7 bitbake: utils: remove spaces on empty lines
--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
bitbake/lib/bb/utils.py +8 -8
@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx
@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]
@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next
@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):
@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default
Signature · 1 line
2.42.0
Patchtest results for [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/RESEND-1-7-bitbake-utils-remove-spaces-on-empty-lines.patch
FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)
FAIL: test target mailing list: Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists (test_mbox.TestMbox.test_target_mailing_list)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
In reply to Julien StephanPlease do not send bitbake patches in the same series as oe-core patches if there is no dependency. Those patches are applied on different repositories.
Quoted 75 lines · Julien Stephan
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)bitbake/lib/bb/utils.py +8
-8@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return defaultSignature · 11 lines
2.42.0 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754 Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
Quoted 4 lines · Alexandre Belloni
Please do not send bitbake patches in the same series as oe-core patches if there is no dependency. Those patches are applied on different repositories.
Hi Alexandre,
Noted! Do you want me to resend the series without the bitbake patch?
Cheers
Julien
Quoted 83 lines · Julien Stephan
Quoted 67 lines
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)bitbake/lib/bb/utils.py +8
-9@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default --2.42.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
Quoted 10 lines · Julien Stephan
Le mar. 5 d�c. 2023 � 08:50, Alexandre Belloni <[email protected]> a �crit :
Quoted 4 lines
Please do not send bitbake patches in the same series as oe-core patches if there is no dependency. Those patches are applied on different repositories.
Hi Alexandre,
Noted! Do you want me to resend the series without the bitbake patch?
No but didn't take the bitbake patch because it doesn't do what the commit message says. (i.e. it touches non empty lines)
Quoted 86 lines
Cheers
JulienQuoted 83 lines · Julien Stephan
Quoted 67 lines
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)bitbake/lib/bb/utils.py +8
-9@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default --2.42.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
Quoted 9 lines · Alexandre Belloni
Quoted 7 lines · Julien Stephan
Quoted 4 lines · Alexandre Belloni
Please do not send bitbake patches in the same series as oe-core patches if there is no dependency. Those patches are applied on different repositories.
Quoted 7 lines
Quoted 3 lines
Hi Alexandre,
Noted! Do you want me to resend the series without the bitbake patch?
No but didn't take the bitbake patch because it doesn't do what the commit message says. (i.e. it touches non empty lines)
oups, I will update the commit message and send it to the appropriate list.
Quoted 91 lines
Quoted 86 lines
Cheers
JulienQuoted 83 lines · Julien Stephan
Quoted 67 lines
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)bitbake/lib/bb/utils.py +8
-9@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default --2.42.0
Quoted 8 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191754): https://lists.openembedded.org/g/openembedded-core/message/191754
Mute This Topic: https://lists.openembedded.org/mt/102972946/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [RESEND][PATCH 1/7] bitbake: utils: remove spaces on empty lines
Quoted 14 lines · Julien Stephan via lists.openembedded.org
Quoted 11 lines · Alexandre Belloni
Quoted 9 lines · Julien Stephan
Quoted 4 lines · Alexandre Belloni
Please do not send bitbake patches in the same series as oe-core patches if there is no dependency. Those patches are applied on different repositories.
Hi Alexandre,
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Noted! Do you want me to resend the series without the bitbake patch?
No but didn't take the bitbake patch because it doesn't do what the commit message says. (i.e. it touches non empty lines)
oups, I will update the commit message and send it to the appropriate list.
It has already been merged with the incorrect commit message :( sorry for that
Cheers
Julien
Quoted 94 lines
Quoted 81 lines
Quoted 80 lines
Cheers
JulienQuoted 77 lines · Julien Stephan
Quoted 63 lines
Signed-off-by Julien Stephan <[email protected]>--- bitbake/lib/bb/utils.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)bitbake/lib/bb/utils.py +8
-9@@ -50,7 +50,7 @@ def clean_context(): def get_context(): return _context - + def set_context(ctx): _context = ctx@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True): inversion = True # This list is based on behavior and supported comparisons from deb, opkg and rpm. # - # Even though =<, <<, ==, !=, =>, and >> may not be supported, - # we list each possibly valid item. + # Even though =<, <<, ==, !=, =>, and >> may not be supported, + # we list each possibly valid item. # The build system is responsible for validation of what it supports. if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')): lastcmp = i[0:2]@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context): exception = traceback.format_exception_only(t, value) error.append('Error executing a python function in %s:\n' % realfile) - # Strip 'us' from the stack (better_exec call) unless that was where the + # Strip 'us' from the stack (better_exec call) unless that was where the # error came from if tb.tb_next is not None: tb = tb.tb_next@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False): # but thats possibly insane and suffixes is probably going to be small # def prune_suffix(var, suffixes, d): - """ + """ See if var ends with any of the suffixes listed and - remove it if found + remove it if found """ for suffix in suffixes: if suffix and var.endswith(suffix):@@ -1001,9 +1001,9 @@ def umask(new_mask): os.umask(current_mask) def to_boolean(string, default=None): - """ + """ Check input string and return boolean value True/False/None - depending upon the checks + depending upon the checks """ if not string: return default --Quoted 6 lines
2.42.0
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.comQuoted 4 lines
--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#191847): https://lists.openembedded.org/g/openembedded-core/message/191847
Mute This Topic: https://lists.openembedded.org/mt/102972946/4356723
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH RESEND 2/7 recipetool: create_buildsys_python.py: initialize metadata
In reply to Julien StephanIn the case pyproject.toml doesn't contains metadatas, the metadata variable is not initialized and the plugin throws an error and falls back to another plugin, which is not the desired behaviour. So just ignore metadata if we don't have them
--- scripts/lib/recipetool/create_buildsys_python.py | 1 + 1 file changed, 1 insertion(+)
scripts/lib/recipetool/create_buildsys_python.py +1 -0
@@ -726,6 +726,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler): def process(self, srctree, classes, lines_before, lines_after, handled, extravalues): info = {} + metadata = {} if 'buildsystem' in handled: return False
Signature · 1 line
2.42.0
PATCH RESEND 3/7 recipetool: create: add trailing newlines
In reply to Julien Stephancreate_recipe() function relies on oe.recipeutils.patch_recipe_lines() which relies on bb.utils.edit_metadata(). edit_metada expect lines to have trailing newlines, so add it to each lines before calling patch_recipe_lines, otherwise edit_metadata will not be able to squash blank line if there are two consecutive blanks after a removal
--- scripts/lib/recipetool/create.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
scripts/lib/recipetool/create.py +5 -3
@@ -873,8 +873,10 @@ def create_recipe(args): outlines.append('') outlines.extend(lines_after) + outlines = [ line.rstrip('\n') +"\n" for line in outlines] + if extravalues: - _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=False) + _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=True) if args.extract_to: scriptutils.git_convert_standalone_clone(srctree)
@@ -890,7 +892,7 @@ def create_recipe(args): log_info_cond('Source extracted to %s' % args.extract_to, args.devtool) if outfile == '-': - sys.stdout.write('\n'.join(outlines) + '\n') + sys.stdout.write(''.join(outlines) + '\n') else: with open(outfile, 'w') as f: lastline = None
@@ -898,7 +900,7 @@ def create_recipe(args): if not lastline and not line: # Skip extra blank lines continue - f.write('%s\n' % line) + f.write('%s' % line) lastline = line log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool) tinfoil.modified_files()
Signature · 1 line
2.42.0
PATCH RESEND 4/7 recipetool: create: add new optional process_url callback for plugins
In reply to Julien StephanAdd a new process_url callback that plugins can optionally implement if they which to handle url.
Plugins can implement this callback for example, to:
* transform the url
* add special variables using extravalues
* add extra classes
* ...
If a plugin handles the url, it must append 'url' to the handled list and must return the fetchuri
No functional changes expected for plugins non implementing this optional callback
--- scripts/lib/recipetool/create.py | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-)
scripts/lib/recipetool/create.py +32 -22
@@ -423,6 +423,36 @@ def create_recipe(args): storeTagName = '' pv_srcpv = False + handled = [] + classes = [] + + # Find all plugins that want to register handlers + logger.debug('Loading recipe handlers') + raw_handlers = [] + for plugin in plugins: + if hasattr(plugin, 'register_recipe_handlers'): + plugin.register_recipe_handlers(raw_handlers) + # Sort handlers by priority + handlers = [] + for i, handler in enumerate(raw_handlers): + if isinstance(handler, tuple): + handlers.append((handler[0], handler[1], i)) + else: + handlers.append((handler, 0, i)) + handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) + for handler, priority, _ in handlers: + logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) + setattr(handler, '_devtool', args.devtool) + handlers = [item[0] for item in handlers] + + fetchuri = None + for handler in handlers: + if hasattr(handler, 'process_url'): + ret = handler.process_url(args, classes, handled, extravalues) + if 'url' in handled and ret: + fetchuri = ret + break + if os.path.isfile(source): source = 'file://%s' % os.path.abspath(source)
@@ -431,7 +461,8 @@ def create_recipe(args): if re.match(r'https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source): logger.warning('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).') # Fetch a URL - fetchuri = reformat_git_uri(urldefrag(source)[0]) + if not fetchuri: + fetchuri = reformat_git_uri(urldefrag(source)[0]) if args.binary: # Assume the archive contains the directory structure verbatim # so we need to extract to a subdirectory
@@ -638,8 +669,6 @@ def create_recipe(args): # We'll come back and replace this later in handle_license_vars() lines_before.append('##LICENSE_PLACEHOLDER##') - handled = [] - classes = [] # FIXME This is kind of a hack, we probably ought to be using bitbake to do this pn = None
@@ -718,25 +747,6 @@ def create_recipe(args): if args.npm_dev: extravalues['NPM_INSTALL_DEV'] = 1 - # Find all plugins that want to register handlers - logger.debug('Loading recipe handlers') - raw_handlers = [] - for plugin in plugins: - if hasattr(plugin, 'register_recipe_handlers'): - plugin.register_recipe_handlers(raw_handlers) - # Sort handlers by priority - handlers = [] - for i, handler in enumerate(raw_handlers): - if isinstance(handler, tuple): - handlers.append((handler[0], handler[1], i)) - else: - handlers.append((handler, 0, i)) - handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True) - for handler, priority, _ in handlers: - logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority)) - setattr(handler, '_devtool', args.devtool) - handlers = [item[0] for item in handlers] - # Apply the handlers if args.binary: classes.append('bin_package')
Signature · 1 line
2.42.0
PATCH RESEND 5/7 recipetool: create_buildsys_python: add pypi support
In reply to Julien StephanToday, we can use devtool/recipetool to create recipes for python projects using the github url or the direct release tarball of the project, but the create_buildsys_python plugin doesn't support the pypi class, since we cannot know from the extracted source if the package is available on pypi or not.
By implementing the new optional process_url callback, we can detect that the url is a pypi one (i.e 'https://pypi.org/project/<package>') and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)
In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx' variables from the created recipe as they will be handled by the pypi class and add the PYPI_PACKAGE variable
This helps to produce cleaner recipes when package is hosted on pypi.
If the url points to a github url or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching url, to keep legacy behaviour.
To create a recipe for a pypi package, one can now use one of the new following syntax (using recipetool create / devtool add):
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>
or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>
--- scripts/lib/devtool/standard.py | 3 + scripts/lib/recipetool/create.py | 1 + .../lib/recipetool/create_buildsys_python.py | 72 +++++++++++++++++++ 3 files changed, 76 insertions(+)
scripts/lib/devtool/standard.py +3 -0
@@ -147,6 +147,8 @@ def add(args, config, basepath, workspace): extracmdopts += ' -a' if args.npm_dev: extracmdopts += ' --npm-dev' + if args.no_pypi: + extracmdopts += ' --no-pypi' if args.mirrors: extracmdopts += ' --mirrors' if args.srcrev:
@@ -2260,6 +2262,7 @@ def register_commands(subparsers, context): group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true") parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI') parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true") + parser_add.add_argument('--no-pypi', help='Do not inherit pypi class', action="store_true") parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)') parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true") group = parser_add.add_mutually_exclusive_group()
scripts/lib/recipetool/create.py +1 -0
@@ -1413,6 +1413,7 @@ def register_commands(subparsers): parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)') parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies') + parser_create.add_argument('--no-pypi', action="store_true", help='Do not inherit pypi class') parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS) parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).') parser_create.set_defaults(func=create_recipe)
scripts/lib/recipetool/create_buildsys_python.py +72 -0
85 lines not on this page · raw message
Signature · 1 line
2.42.0
Re: [OE-core] [RESEND][PATCH 5/7] recipetool: create_buildsys_python: add pypi support
Quoted 179 lines · Julien Stephan
Today, we can use devtool/recipetool to create recipes for python projects using the github url or the direct release tarball of the project, but the
create_buildsys_pythonplugin doesn't support the pypi class, since we cannot
know from the extracted source if the package is available on pypi or not.By implementing the new optional
process_urlcallback, we can detect that the url is a pypi one (i.e 'https://pypi.org/project/<package>') and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/..
.)In both cases, adds the pypi class, remove 'S' and '
SRC_URIxxx' variables from the created recipe as they will be handled by the pypi class and add thePYPI_PACKAGEvariableThis helps to produce cleaner recipes when package is hosted on pypi.
If the url points to a github url or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching url, to keep legacy behaviour.
To create a recipe for a pypi package, one can now use one of the new following syntax (using recipetool create / devtool add):
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>Signed-off-by Julien Stephan <[email protected]>--- scripts/lib/devtool/standard.py | 3 + scripts/lib/recipetool/create.py | 1 + .../lib/recipetool/create_buildsys_python.py | 72 +++++++++++++++++++ 3 files changed, 76 insertions(+)scripts/lib/devtool/standard.py +2
-0@@ -147,6 +147,8 @@ def add(args, config, basepath, workspace): extracmdopts += ' -a' if args.npm_dev: extracmdopts += ' --npm-dev' + if args.no_pypi: + extracmdopts += ' --no-pypi' if args.mirrors: extracmdopts += ' --mirrors' if args.srcrev:@@ -2260,6 +2262,7 @@ def register_commands(subparsers, context): group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true") parser_add.add_argument('--fetch', '-f', help='Fetch the specifiedURI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
parser_add.add_argument('--npm-dev', help='For npm, also fetch
devDependencies', action="store_true")
+ parser_add.add_argument('--no-pypi', help='Do not inherit pypi
class', action="store_true")
parser_add.add_argument('--version', '-V', help='Version to use
within recipe (PV)')parser_add.add_argument('--no-git', '-g', help='If fetching source,do not set up source tree as a git repository', action="store_true")
group =parser_add.add_mutually_exclusive_group()scripts/lib/recipetool/create.py +1
-0@@ -1413,6 +1413,7 @@ def register_commands(subparsers): parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)') parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)') parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies') + parser_create.add_argument('--no-pypi', action="store_true", help='Donot inherit pypi class')
parser_create.add_argument('--devtool', action="store_true",
help=argparse.SUPPRESS)
parser_create.add_argument('--mirrors', action="store_true",
help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
parser_create.set_defaults(func=create_recipe)scripts/lib/recipetool/create_buildsys_python.py +7
-0@@ -18,7 +18,11 @@ import os import re import sys import subprocess +import json +import urllib.request from recipetool.create import RecipeHandler +from urllib.parse import urldefrag +from recipetool.create import determine_from_url logger = logging.getLogger('recipetool')@@ -111,6 +115,74 @@ class PythonRecipeHandler(RecipeHandler): def __init__(self): pass + def process_url(self, args, classes, handled, extravalues): + """ + Convert any pypi url https://pypi.org/project/<package>/<version>into https://files.pythonhosted.org/packages/source/...
scripts/lib/recipetool/create_buildsys_python.py +10
-0+ which corresponds to the archive location, and add pypi class + """ + + if 'url' in handled: + return None + + fetch_uri = None + source = args.source + required_version = args.version if args.version else None + match = re.match(r'https?://pypi.org/project/([<http://pypi.org/project/(%5B>^/]+)(?:/([^/]+))?/?$',
urldefrag(source)[0])scripts/lib/recipetool/create_buildsys_python.py +3
-0+ if match: + package = match.group(1) + version = match.group(2) if match.group(2) else
required_versionscripts/lib/recipetool/create_buildsys_python.py +15
-0+ + json_url = f"https://pypi.org/pypi/%s/json" % package + response = urllib.request.urlopen(json_url) + if response.status == 200: + data = json.loads(response.read()) + if not version: + # grab latest version + version = data["info"]["version"] + pypi_package = data["info"]["name"] + for release in reversed(data["releases"][version]): + if release["packagetype"] == "sdist": + fetch_uri = release["url"] + break + else: + logger.warning("Cannot handle pypi url %s: cannot fetchpackage information using %s", source,
json_url)scripts/lib/recipetool/create_buildsys_python.py +3
-0+ return None + else: + match = re.match(r'^https?://files.pythonhosted.org/packages.*/(.*)-.*$', source)
scripts/lib/recipetool/create_buildsys_python.py +8
-0+ if match: + fetch_uri = source + pypi_package = match.group(1) + _, version = determine_from_url(fetch_uri) + + if match and not args.no_pypi: + if required_version and version != required_version: + raise Exception("Version specified using --version/-V(%s) and version specified in the url (%s) do not match" % (
required_version, version))
+ # This is optionnal if BPN looks like "python-<pypi_package>"
or "python3-<pypi_package>" (see pypi.bbclass)
+ # but at this point we cannot know because because user can
specify the output name of the recipe on the command linescripts/lib/recipetool/create_buildsys_python.py +2
-0+ extravalues["PYPI_PACKAGE"] = pypi_package + # If the tarball extension is not 'tar.gz' (default value inpypi.bblcass) whe should set
PYPI_PACKAGE_EXTin the recipe
+pypi_package_ext= re.match(r'.*%s-%s\.(.*)$' %
(pypi_package, version),fetch_uri)scripts/lib/recipetool/create_buildsys_python.py +6
-0+ if pypi_package_ext: + pypi_package_ext = pypi_package_ext.group(1) + if pypi_package_ext != "tar.gz": + extravalues["PYPI_PACKAGE_EXT"] = pypi_package_ext + + # Pypi class will handle S and SRC_URIxxx variables, soremove them
+ # TODO: allowoe.recipeutils.patch_recipe_lines()to accept
regexp so we can simplify the following to:
scripts/lib/recipetool/create_buildsys_python.py class PythonRecipeHandler(RecipeHandler): · line 180
Quoted 9 lines
scripts/lib/recipetool/create_buildsys_python.py +9
-0+ # extravalues['SRC_URI(?:\[.*?\])?'] = None + extravalues['S'] = None + extravalues['SRC_URI'] = None + extravalues['SRC_URI[md5sum]'] = None + extravalues['SRC_URI[sha1sum]'] = None + extravalues['SRC_URI[sha256sum]'] = None + extravalues['SRC_URI[sha384sum]'] = None + extravalues['SRC_URI[sha512sum]'] = None +
this was not correct. The pypi.bbclass does not create the SRC_URI[sha256sum]. It does create the SRC_URI. So new recipes will not have any check on the validity of the downloaded tarball.
The only test case which checked for this was with —nopypi
https://git.yoctoproject.org/poky-contrib/tree/meta/lib/oeqa/selftest/cases/recipetool.py#n752
Quoted 24 lines
scripts/lib/recipetool/create_buildsys_python.py +5
-0+ classes.append('pypi') + + handled.append('url') + return fetch_uri + def handle_classifier_license(self, classifiers,existing_licenses=""):
licenses = []--
2.42.0-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#191759):
https://lists.openembedded.org/g/openembedded-core/message/191759
Mute This Topic: https://lists.openembedded.org/mt/102972952/924729
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
PATCH RESEND 6/7 oeqa/selftest/recipetool: remove spaces on empty lines
In reply to Julien Stephan--- meta/lib/oeqa/selftest/cases/recipetool.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/recipetool.py +1 -1
@@ -853,7 +853,7 @@ class RecipetoolTests(RecipetoolBase): self._test_recipe_contents(deps_require_file, checkvars, []) - + def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths): dstdir = basedstdir self.assertTrue(os.path.exists(dstdir))
Signature · 1 line
2.42.0
Patchtest results for [RESEND][PATCH 6/7] oeqa/selftest/recipetool: remove spaces on empty lines
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/RESEND-6-7-oeqa-selftest-recipetool-remove-spaces-on-empty-lines.patch
FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)
PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
PATCH RESEND 7/7 oeqa/selftest/recipetool/devtool: add test for pypi class
In reply to Julien Stephanrecipetool now supports the pypi class and python recipes can by created using the new following syntax:
* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>
or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>
So add tests for the new syntax and modify old tests
--- meta/lib/oeqa/selftest/cases/devtool.py | 4 +- meta/lib/oeqa/selftest/cases/recipetool.py | 112 +++++++++++++++------ 2 files changed, 85 insertions(+), 31 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +2 -2
@@ -543,7 +543,7 @@ class DevtoolAddTests(DevtoolBase): self.track_for_cleanup(self.workspacedir) self.add_command_to_tearDown('bitbake -c cleansstate %s' % testrecipe) self.add_command_to_tearDown('bitbake-layers remove-layer */workspace') - result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url)) + result = runCmd('devtool add --no-pypi %s %s -f %s' % (testrecipe, srcdir, url)) self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created. %s' % result.output) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created')
@@ -562,7 +562,7 @@ class DevtoolAddTests(DevtoolBase): result = runCmd('devtool reset -n %s' % testrecipe) shutil.rmtree(srcdir) fakever = '1.9' - result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) + result = runCmd('devtool add --no-pypi %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever)) self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory') # Test devtool status result = runCmd('devtool status')
meta/lib/oeqa/selftest/cases/recipetool.py +83 -29
154 lines not on this page · raw message
Signature · 1 line
2.42.0