Skip to contentSkip to conversation

Devtool/Recipetool: adding pypi support

PATCH Superseded Julien Stephan 3 participants 12 messages Nov 17, 2023 – Nov 18, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover Devtool/Recipetool: adding pypi support

Hello all,

This series adds basic support for the pypi class in devtool/recipetool.

The idea is to be able to detect that a package is available on pypi and in that case inherits from pypi class. This helps to produce cleaner recipes for pypi packages.

To do this, I am adding a new optional callback "process_url" that plugin can register to implement some logic on matching url.

By implementing this callback in create_buildsys_python we can match on pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".

To create a recipe taking advantage of the pypi class we can use one of the following new syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:

* recipetool create https://files.pythonhosted.org/packages/<...>

If the URL points to a github URL or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching URL, to keep legacy behaviour.

This series also contains some bug fixes I found during my testing.

Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support

Cheers
Julien

Julien Stephan (7):
bitbake: utils: remove spaces on empty lines
recipetool: create_buildsys_python.py: initialize metadata
recipetool: create: add trailing newlines
recipetool: create: add new optional process_url callback for plugins
recipetool: create_buildsys_python: add pypi support oeqa/selftest/recipetool: remove spaces on empty lines oeqa/selftest/recipetool/devtool: add test for pypi class

 bitbake/lib/bb/utils.py                       |  16 +--
 meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
 scripts/lib/devtool/standard.py               |   3 +
 scripts/lib/recipetool/create.py              |  63 ++++++----
 .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
 6 files changed, 208 insertions(+), 65 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/7 bitbake: utils: remove spaces on empty lines

---
 bitbake/lib/bb/utils.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
bitbake/lib/bb/utils.py +8 -8
index b401fa5ec7a..61ffad92ce3 100644
@@ -50,7 +50,7 @@ def clean_context():
 
 def get_context():
     return _context
-    
+
 
 def set_context(ctx):
     _context = ctx
@@ -212,8 +212,8 @@ def explode_dep_versions2(s, *, sort=True):
             inversion = True
             # This list is based on behavior and supported comparisons from deb, opkg and rpm.
             #
-            # Even though =<, <<, ==, !=, =>, and >> may not be supported, 
-            # we list each possibly valid item. 
+            # Even though =<, <<, ==, !=, =>, and >> may not be supported,
+            # we list each possibly valid item.
             # The build system is responsible for validation of what it supports.
             if i.startswith(('<=', '=<', '<<', '==', '!=', '>=', '=>', '>>')):
                 lastcmp = i[0:2]
@@ -347,7 +347,7 @@ def _print_exception(t, value, tb, realfile, text, context):
         exception = traceback.format_exception_only(t, value)
         error.append('Error executing a python function in %s:\n' % realfile)
 
-        # Strip 'us' from the stack (better_exec call) unless that was where the 
+        # Strip 'us' from the stack (better_exec call) unless that was where the
         # error came from
         if tb.tb_next is not None:
             tb = tb.tb_next
@@ -746,9 +746,9 @@ def prunedir(topdir, ionice=False):
 # but thats possibly insane and suffixes is probably going to be small
 #
 def prune_suffix(var, suffixes, d):
-    """ 
+    """
     See if var ends with any of the suffixes listed and
-    remove it if found 
+    remove it if found
     """
     for suffix in suffixes:
         if suffix and var.endswith(suffix):
@@ -1001,9 +1001,9 @@ def umask(new_mask):
         os.umask(current_mask)
 
 def to_boolean(string, default=None):
-    """ 
+    """
     Check input string and return boolean value True/False/None
-    depending upon the checks 
+    depending upon the checks
     """
     if not string:
         return default
Signature · 1 line
2.42.0
#2 lore raw

Patchtest results for [PATCH 1/7] bitbake: utils: remove spaces on empty lines

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/1-7-bitbake-utils-remove-spaces-on-empty-lines.patch

FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#3 lore raw

PATCH 2/7 recipetool: create_buildsys_python.py: initialize metadata

In reply to Julien Stephan

In the case pyproject.toml doesn't contains metadatas, the metadata variable is not initialized and the plugin throws an error and falls back to another plugin, which is not the desired behaviour. So just ignore metadata if we don't have them

---
 scripts/lib/recipetool/create_buildsys_python.py | 1 +
 1 file changed, 1 insertion(+)
scripts/lib/recipetool/create_buildsys_python.py +1 -0
index 9312e4abf13..b620e3271b1 100644
@@ -726,6 +726,7 @@ class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
 
     def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
         info = {}
+        metadata = {}
 
         if 'buildsystem' in handled:
             return False
Signature · 1 line
2.42.0
#4 lore raw

PATCH 3/7 recipetool: create: add trailing newlines

In reply to Julien Stephan

create_recipe() function relies on oe.recipeutils.patch_recipe_lines() which relies on bb.utils.edit_metadata(). edit_metada expect lines to have trailing newlines, so add it to each line before calling patch_recipe_lines, otherwise edit_metadata will not be able to squash blank line if there are two consecutive blanks after a removal

---
 scripts/lib/recipetool/create.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
scripts/lib/recipetool/create.py +5 -3
index 293198d1c88..f5d541eb6c1 100644
@@ -873,8 +873,10 @@ def create_recipe(args):
         outlines.append('')
     outlines.extend(lines_after)
 
+    outlines = [ line.rstrip('\n') +"\n" for line in outlines]
+
     if extravalues:
-        _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=False)
+        _, outlines = oe.recipeutils.patch_recipe_lines(outlines, extravalues, trailing_newline=True)
 
     if args.extract_to:
         scriptutils.git_convert_standalone_clone(srctree)
@@ -890,7 +892,7 @@ def create_recipe(args):
         log_info_cond('Source extracted to %s' % args.extract_to, args.devtool)
 
     if outfile == '-':
-        sys.stdout.write('\n'.join(outlines) + '\n')
+        sys.stdout.write(''.join(outlines) + '\n')
     else:
         with open(outfile, 'w') as f:
             lastline = None
@@ -898,7 +900,7 @@ def create_recipe(args):
                 if not lastline and not line:
                     # Skip extra blank lines
                     continue
-                f.write('%s\n' % line)
+                f.write('%s' % line)
                 lastline = line
         log_info_cond('Recipe %s has been created; further editing may be required to make it fully functional' % outfile, args.devtool)
         tinfoil.modified_files()
Signature · 1 line
2.42.0
#5 lore raw

PATCH 4/7 recipetool: create: add new optional process_url callback for plugins

In reply to Julien Stephan

Add a new process_url callback that plugins can optionally implement if they which to handle url.

Plugins can implement this callback for example, to:
* transform the url
* add special variables using extravalues
* add extra classes
* ...

If a plugin handles the url, it must append 'url' to the handled list and must return the fetchuri

No functional changes expected for plugins non implementing this optional callback

---
 scripts/lib/recipetool/create.py | 54 +++++++++++++++++++-------------
 1 file changed, 32 insertions(+), 22 deletions(-)
scripts/lib/recipetool/create.py +32 -22
index f5d541eb6c1..5c5ac7ae403 100644
@@ -423,6 +423,36 @@ def create_recipe(args):
     storeTagName = ''
     pv_srcpv = False
 
+    handled = []
+    classes = []
+
+    # Find all plugins that want to register handlers
+    logger.debug('Loading recipe handlers')
+    raw_handlers = []
+    for plugin in plugins:
+        if hasattr(plugin, 'register_recipe_handlers'):
+            plugin.register_recipe_handlers(raw_handlers)
+    # Sort handlers by priority
+    handlers = []
+    for i, handler in enumerate(raw_handlers):
+        if isinstance(handler, tuple):
+            handlers.append((handler[0], handler[1], i))
+        else:
+            handlers.append((handler, 0, i))
+    handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True)
+    for handler, priority, _ in handlers:
+        logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority))
+        setattr(handler, '_devtool', args.devtool)
+    handlers = [item[0] for item in handlers]
+
+    fetchuri = None
+    for handler in handlers:
+        if hasattr(handler, 'process_url'):
+            ret = handler.process_url(args, classes, handled, extravalues)
+            if 'url' in handled and ret:
+                fetchuri = ret
+                break
+
     if os.path.isfile(source):
         source = 'file://%s' % os.path.abspath(source)
 
@@ -431,7 +461,8 @@ def create_recipe(args):
         if re.match(r'https?://github.com/[^/]+/[^/]+/archive/.+(\.tar\..*|\.zip)$', source):
             logger.warning('github archive files are not guaranteed to be stable and may be re-generated over time. If the latter occurs, the checksums will likely change and the recipe will fail at do_fetch. It is recommended that you point to an actual commit or tag in the repository instead (using the repository URL in conjunction with the -S/--srcrev option).')
         # Fetch a URL
-        fetchuri = reformat_git_uri(urldefrag(source)[0])
+        if not fetchuri:
+            fetchuri = reformat_git_uri(urldefrag(source)[0])
         if args.binary:
             # Assume the archive contains the directory structure verbatim
             # so we need to extract to a subdirectory
@@ -638,8 +669,6 @@ def create_recipe(args):
     # We'll come back and replace this later in handle_license_vars()
     lines_before.append('##LICENSE_PLACEHOLDER##')
 
-    handled = []
-    classes = []
 
     # FIXME This is kind of a hack, we probably ought to be using bitbake to do this
     pn = None
@@ -718,25 +747,6 @@ def create_recipe(args):
     if args.npm_dev:
         extravalues['NPM_INSTALL_DEV'] = 1
 
-    # Find all plugins that want to register handlers
-    logger.debug('Loading recipe handlers')
-    raw_handlers = []
-    for plugin in plugins:
-        if hasattr(plugin, 'register_recipe_handlers'):
-            plugin.register_recipe_handlers(raw_handlers)
-    # Sort handlers by priority
-    handlers = []
-    for i, handler in enumerate(raw_handlers):
-        if isinstance(handler, tuple):
-            handlers.append((handler[0], handler[1], i))
-        else:
-            handlers.append((handler, 0, i))
-    handlers.sort(key=lambda item: (item[1], -item[2]), reverse=True)
-    for handler, priority, _ in handlers:
-        logger.debug('Handler: %s (priority %d)' % (handler.__class__.__name__, priority))
-        setattr(handler, '_devtool', args.devtool)
-    handlers = [item[0] for item in handlers]
-
     # Apply the handlers
     if args.binary:
         classes.append('bin_package')
Signature · 1 line
2.42.0
#6 lore raw

PATCH 5/7 recipetool: create_buildsys_python: add pypi support

In reply to Julien Stephan

Today, we can use devtool/recipetool to create recipes for python projects using the github url or the direct realse tarball of the project, but the create_buildsys_python plugin doesn't support the pypi class, since we cannot know from the extracted source if the package is available on pypi or not.

By implementing the new optional process_url callback, we can detect that the url is a pypi one (i.e 'https://pypi.org/project/<package>') and retrieve the release tarball location.
Also detect if the url points to a release tarball hosted on "files.pythonhosted.iorg" (i.e https://files.pythonhosted.org/packages/...)

In both cases, adds the pypi class, remove 'S' and 'SRC_URIxxx' variables from the created recipe as they will be handled by the pypi class and add the PYPI_PACKAGE variable

This helps to produce cleaner recipes when package is hosted on pypi.

If the url points to a github url or a release tarball not coming from "files.pythonhosted.org" website, the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to devtool/recipetool to NOT inherit from pypi class on matching url, to keep legacy behaviour.

To create a recipe for a pypi package, one can now use one of the new following syntax (using recipetool create / devtool add):

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>

---
 scripts/lib/devtool/standard.py               |  3 +
 scripts/lib/recipetool/create.py              |  1 +
 .../lib/recipetool/create_buildsys_python.py  | 72 +++++++++++++++++++
 3 files changed, 76 insertions(+)
scripts/lib/devtool/standard.py +3 -0
index d53fb810071..f18ebaa70d3 100644
@@ -147,6 +147,8 @@ def add(args, config, basepath, workspace):
         extracmdopts += ' -a'
     if args.npm_dev:
         extracmdopts += ' --npm-dev'
+    if args.no_pypi:
+        extracmdopts += ' --no-pypi'
     if args.mirrors:
         extracmdopts += ' --mirrors'
     if args.srcrev:
@@ -2260,6 +2262,7 @@ def register_commands(subparsers, context):
     group.add_argument('--no-same-dir', help='Force build in a separate build directory', action="store_true")
     parser_add.add_argument('--fetch', '-f', help='Fetch the specified URI and extract it to create the source tree (deprecated - pass as positional argument instead)', metavar='URI')
     parser_add.add_argument('--npm-dev', help='For npm, also fetch devDependencies', action="store_true")
+    parser_add.add_argument('--no-pypi', help='Do not inherit pypi class', action="store_true")
     parser_add.add_argument('--version', '-V', help='Version to use within recipe (PV)')
     parser_add.add_argument('--no-git', '-g', help='If fetching source, do not set up source tree as a git repository', action="store_true")
     group = parser_add.add_mutually_exclusive_group()
scripts/lib/recipetool/create.py +1 -0
index 5c5ac7ae403..963aa91421e 100644
@@ -1413,6 +1413,7 @@ def register_commands(subparsers):
     parser_create.add_argument('-B', '--srcbranch', help='Branch in source repository if fetching from an SCM such as git (default master)')
     parser_create.add_argument('--keep-temp', action="store_true", help='Keep temporary directory (for debugging)')
     parser_create.add_argument('--npm-dev', action="store_true", help='For npm, also fetch devDependencies')
+    parser_create.add_argument('--no-pypi', action="store_true", help='Do not inherit pypi class')
     parser_create.add_argument('--devtool', action="store_true", help=argparse.SUPPRESS)
     parser_create.add_argument('--mirrors', action="store_true", help='Enable PREMIRRORS and MIRRORS for source tree fetching (disabled by default).')
     parser_create.set_defaults(func=create_recipe)
scripts/lib/recipetool/create_buildsys_python.py +72 -0

85 lines not on this page · raw message

Signature · 1 line
2.42.0
#7 lore raw

PATCH 6/7 oeqa/selftest/recipetool: remove spaces on empty lines

In reply to Julien Stephan
---
 meta/lib/oeqa/selftest/cases/recipetool.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/recipetool.py +1 -1
index 55cbba9ca74..4bc28a4f2ee 100644
@@ -853,7 +853,7 @@ class RecipetoolTests(RecipetoolBase):
         self._test_recipe_contents(deps_require_file, checkvars, [])
 
 
-        
+
     def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
         dstdir = basedstdir
         self.assertTrue(os.path.exists(dstdir))
Signature · 1 line
2.42.0
#8 lore raw

Patchtest results for [PATCH 6/7] oeqa/selftest/recipetool: remove spaces on empty lines

Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:

---
Testing patch /home/patchtest/share/mboxes/6-7-oeqa-selftest-recipetool-remove-spaces-on-empty-lines.patch

FAIL: test commit message presence: Please include a commit message on your patch explaining the change (test_mbox.TestMbox.test_commit_message_presence)

PASS: pretest pylint (test_python_pylint.PyLint.pretest_pylint)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test max line length (test_metadata.TestMetadata.test_max_line_length)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test pylint (test_python_pylint.PyLint.test_pylint)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)

SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
SKIP: test target mailing list: Series merged, no reason to check other mailing lists (test_mbox.TestMbox.test_target_mailing_list)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!

#9 lore raw

PATCH 7/7 oeqa/selftest/recipetool/devtool: add test for pypi class

In reply to Julien Stephan

recipetool now supports the pypi class and python recipes can by created using the new following syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:
* recipetool create https://files.pythonhosted.org/packages/<...>

So add tests for the new syntax and modify old tests

---
 meta/lib/oeqa/selftest/cases/devtool.py    |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py | 112 +++++++++++++++------
 2 files changed, 85 insertions(+), 31 deletions(-)
meta/lib/oeqa/selftest/cases/devtool.py +2 -2
index b5c488be8e8..3d9a4901ea5 100644
@@ -543,7 +543,7 @@ class DevtoolAddTests(DevtoolBase):
         self.track_for_cleanup(self.workspacedir)
         self.add_command_to_tearDown('bitbake -c cleansstate %s' % testrecipe)
         self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
-        result = runCmd('devtool add %s %s -f %s' % (testrecipe, srcdir, url))
+        result = runCmd('devtool add --no-pypi %s %s -f %s' % (testrecipe, srcdir, url))
         self.assertExists(os.path.join(self.workspacedir, 'conf', 'layer.conf'), 'Workspace directory not created. %s' % result.output)
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
         self.assertTrue(os.path.isdir(os.path.join(srcdir, '.git')), 'git repository for external source tree was not created')
@@ -562,7 +562,7 @@ class DevtoolAddTests(DevtoolBase):
         result = runCmd('devtool reset -n %s' % testrecipe)
         shutil.rmtree(srcdir)
         fakever = '1.9'
-        result = runCmd('devtool add %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever))
+        result = runCmd('devtool add --no-pypi %s %s -f %s -V %s' % (testrecipe, srcdir, url, fakever))
         self.assertTrue(os.path.isfile(os.path.join(srcdir, 'setup.py')), 'Unable to find setup.py in source directory')
         # Test devtool status
         result = runCmd('devtool status')
meta/lib/oeqa/selftest/cases/recipetool.py +83 -29

154 lines not on this page · raw message

Signature · 1 line
2.42.0
#10 lore raw
In reply to Julien Stephan

Hello,

The tests seem to be failing on the AB:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6065/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6062/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6013/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6078/steps/14/logs/stdio

Quoted 67 lines · Julien Stephan

Hello all,

This series adds basic support for the pypi class in devtool/recipetool.

The idea is to be able to detect that a package is available on pypi and in that case inherits from pypi class. This helps to produce cleaner recipes for pypi packages.

To do this, I am adding a new optional callback "process_url" that plugin can register to implement some logic on matching url.

By implementing this callback in create_buildsys_python we can match on pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".

To create a recipe taking advantage of the pypi class we can use one of the following new syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:

* recipetool create https://files.pythonhosted.org/packages/<...>

If the URL points to a github URL or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching URL, to keep legacy behaviour.

This series also contains some bug fixes I found during my testing.

Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support

Cheers
Julien

Julien Stephan (7):
bitbake: utils: remove spaces on empty lines
recipetool: create_buildsys_python.py: initialize metadata
recipetool: create: add trailing newlines
recipetool: create: add new optional process_url callback for plugins
recipetool: create_buildsys_python: add pypi support oeqa/selftest/recipetool: remove spaces on empty lines oeqa/selftest/recipetool/devtool: add test for pypi class

 bitbake/lib/bb/utils.py                       |  16 +--
 meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
 scripts/lib/devtool/standard.py               |   3 +
 scripts/lib/recipetool/create.py              |  63 ++++++----
 .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
 6 files changed, 208 insertions(+), 65 deletions(-)
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190836): https://lists.openembedded.org/g/openembedded-core/message/190836
Mute This Topic: https://lists.openembedded.org/mt/102645283/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#12 lore raw
Quoted 8 lines · Alexandre Belloni via lists.openembedded.org

Hello,

The tests seem to be failing on the AB:

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/6065/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6062/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6013/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6078/steps/14/logs/stdio

Actually this seems to have been caused by something else.

Quoted 79 lines · Julien Stephan
Quoted 61 lines

Hello all,

This series adds basic support for the pypi class in devtool/recipetool.

The idea is to be able to detect that a package is available on pypi and in that case inherits from pypi class. This helps to produce cleaner recipes for pypi packages.

To do this, I am adding a new optional callback "process_url" that plugin can register to implement some logic on matching url.

By implementing this callback in create_buildsys_python we can match on pypi URLs directly or on release tarballs hosted on "files.pythonhosted.org".

To create a recipe taking advantage of the pypi class we can use one of the following new syntax:

* recipetool create https://pypi.org/project/<package>
* recipetool create https://pypi.org/project/<package>/<version>
* recipetool create https://pypi.org/project/<package> --version <version>

or the old syntax:

* recipetool create https://files.pythonhosted.org/packages/<...>

If the URL points to a github URL or a release tarball not coming from "files.pythonhosted.org", the created recipe is the same as before. One can also use the newly added "--no-pypi" switch to NOT inherit from pypi class on matching URL, to keep legacy behaviour.

This series also contains some bug fixes I found during my testing.

Pushed my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-pypi-support

Cheers
Julien

Julien Stephan (7):
bitbake: utils: remove spaces on empty lines
recipetool: create_buildsys_python.py: initialize metadata
recipetool: create: add trailing newlines
recipetool: create: add new optional process_url callback for plugins
recipetool: create_buildsys_python: add pypi support oeqa/selftest/recipetool: remove spaces on empty lines oeqa/selftest/recipetool/devtool: add test for pypi class

 bitbake/lib/bb/utils.py                       |  16 +--
 meta/lib/oeqa/selftest/cases/devtool.py       |   4 +-
 meta/lib/oeqa/selftest/cases/recipetool.py    | 114 +++++++++++++-----
 scripts/lib/devtool/standard.py               |   3 +
 scripts/lib/recipetool/create.py              |  63 ++++++----
 .../lib/recipetool/create_buildsys_python.py  |  73 +++++++++++
 6 files changed, 208 insertions(+), 65 deletions(-)
Signature · 1 line
2.42.0
Signature · 12 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#190862): https://lists.openembedded.org/g/openembedded-core/message/190862
Mute This Topic: https://lists.openembedded.org/mt/102645283/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com