Skip to contentSkip to conversation

scripts:recipetool:create_buildsys_python: fix license note

PATCH In review Julien Stephan 4 participants 13 messages Oct 19, 2023 – Oct 20, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/4 scripts:recipetool:create_buildsys_python: fix license note

License field of setup is not always standardized, so we usually use the classifier to determine the correct license format to use in the recipe.

A warning note is added above the LICENSE field of the create recipe in case a license is provided in setup. But when the plugin is called, "LICENSE =" is not yet present so we can never display this note. Replace the "LICENSE =" condition with "##LICENSE_PLACEHOLDER##" to actually be able to display the note message

---
 scripts/lib/recipetool/create_buildsys_python.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/recipetool/create_buildsys_python.py +1 -1
index 92468b22546..321d0ba257d 100644
@@ -254,7 +254,7 @@ class PythonRecipeHandler(RecipeHandler):
 
         if license_str:
             for i, line in enumerate(lines_before):
-                if line.startswith('LICENSE = '):
+                if line.startswith('##LICENSE_PLACEHOLDER##'):
                     lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
                     break
 
Signature · 1 line
2.42.0
#1 lore raw

PATCH v2 2/4 scripts:recipetool:create_buildsys_python: prefix created recipes with python3-

By convention, all python recipes start with "python3-" so update create_buildsys_python to do this

This rule doesn't apply for packages already starting with "python"

---
 scripts/lib/recipetool/create_buildsys_python.py | 5 +++++
 1 file changed, 5 insertions(+)
scripts/lib/recipetool/create_buildsys_python.py +5 -0
index 321d0ba257d..502e1dfbc3d 100644
@@ -297,6 +297,11 @@ class PythonRecipeHandler(RecipeHandler):
                 value = ' '.join(str(v) for v in values if v)
 
             bbvar = self.bbvar_map[field]
+            if bbvar == "PN":
+                # by convention python recipes start with "python3-"
+                if not value.startswith('python'):
+                    value = 'python3-' + value
+
             if bbvar not in extravalues and value:
                 extravalues[bbvar] = value
 
Signature · 1 line
2.42.0
#2 lore raw

PATCH v2 3/4 scripts:recipetool:create_buildsys_python: refactor code for futur PEP517 addition

In reply to Julien Stephan

In order to prepare the support for pyproject.toml (PEP517 [1]) enabled projects, refactor the code and move setup.py specific code into a specific class in order to allow sharing the PythonRecipeHandler class

No functionnal changes expected

[1]: https://peps.python.org/pep-0517/#source-tree

---
 .../lib/recipetool/create_buildsys_python.py  | 748 +++++++++---------
 1 file changed, 385 insertions(+), 363 deletions(-)
scripts/lib/recipetool/create_buildsys_python.py +385 -363
index 502e1dfbc3d..69f6f5ca511 100644
@@ -37,63 +37,8 @@ class PythonRecipeHandler(RecipeHandler):
     assume_provided = ['builtins', 'os.path']
     # Assumes that the host python3 builtin_module_names is sane for target too
     assume_provided = assume_provided + list(sys.builtin_module_names)
+    excluded_fields = []
 
-    bbvar_map = {
-        'Name': 'PN',
-        'Version': 'PV',
-        'Home-page': 'HOMEPAGE',
-        'Summary': 'SUMMARY',
-        'Description': 'DESCRIPTION',
-        'License': 'LICENSE',
-        'Requires': 'RDEPENDS:${PN}',
-        'Provides': 'RPROVIDES:${PN}',
-        'Obsoletes': 'RREPLACES:${PN}',
-    }
-    # PN/PV are already set by recipetool core & desc can be extremely long
-    excluded_fields = [
-        'Description',
-    ]
-    setup_parse_map = {
-        'Url': 'Home-page',
-        'Classifiers': 'Classifier',
-        'Description': 'Summary',
-    }
-    setuparg_map = {
-        'Home-page': 'url',
-        'Classifier': 'classifiers',
-        'Summary': 'description',
-        'Description': 'long-description',
-    }
-    # Values which are lists, used by the setup.py argument based metadata
-    # extraction method, to determine how to process the setup.py output.
-    setuparg_list_fields = [
-        'Classifier',
-        'Requires',
-        'Provides',
-        'Obsoletes',
-        'Platform',
-        'Supported-Platform',
-    ]
-    setuparg_multi_line_values = ['Description']
-    replacements = [
-        ('License', r' +$', ''),
-        ('License', r'^ +', ''),
-        ('License', r' ', '-'),
-        ('License', r'^GNU-', ''),
-        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
-        ('License', r'^UNKNOWN$', ''),
-
-        # Remove currently unhandled version numbers from these variables
-        ('Requires', r' *\([^)]*\)', ''),
-        ('Provides', r' *\([^)]*\)', ''),
-        ('Obsoletes', r' *\([^)]*\)', ''),
-        ('Install-requires', r'^([^><= ]+).*', r'\1'),
-        ('Extras-require', r'^([^><= ]+).*', r'\1'),
-        ('Tests-require', r'^([^><= ]+).*', r'\1'),
-
-        # Remove unhandled dependency on particular features (e.g. foo[PDF])
-        ('Install-requires', r'\[[^\]]+\]$', ''),
-    ]
 
     classifier_license_map = {
         'License :: OSI Approved :: Academic Free License (AFL)': 'AFL',
@@ -166,122 +111,34 @@ class PythonRecipeHandler(RecipeHandler):
     def __init__(self):
         pass
 
-    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
-        if 'buildsystem' in handled:
-            return False
-
-        # Check for non-zero size setup.py files
-        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
-        for fn in setupfiles:
-            if os.path.getsize(fn):
-                break
-        else:
-            return False
-
-        # setup.py is always parsed to get at certain required information, such as
-        # distutils vs setuptools
-        #
-        # If egg info is available, we use it for both its PKG-INFO metadata
-        # and for its requires.txt for install_requires.
-        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
-        # the parsed setup.py, but use the install_requires info from the
-        # parsed setup.py.
-
-        setupscript = os.path.join(srctree, 'setup.py')
-        try:
-            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
-        except Exception:
-            logger.exception("Failed to parse setup.py")
-            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
-
-        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
-        if egginfo:
-            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
-            requires_txt = os.path.join(egginfo[0], 'requires.txt')
-            if os.path.exists(requires_txt):
-                with codecs.open(requires_txt) as f:
-                    inst_req = []
-                    extras_req = collections.defaultdict(list)
-                    current_feature = None
-                    for line in f.readlines():
-                        line = line.rstrip()
-                        if not line:
-                            continue
-
-                        if line.startswith('['):
-                            # PACKAGECONFIG must not contain expressions or whitespace
-                            line = line.replace(" ", "")
-                            line = line.replace(':', "")
-                            line = line.replace('.', "-dot-")
-                            line = line.replace('"', "")
-                            line = line.replace('<', "-smaller-")
-                            line = line.replace('>', "-bigger-")
-                            line = line.replace('_', "-")
-                            line = line.replace('(', "")
-                            line = line.replace(')', "")
-                            line = line.replace('!', "-not-")
-                            line = line.replace('=', "-equals-")
-                            current_feature = line[1:-1]
-                        elif current_feature:
-                            extras_req[current_feature].append(line)
-                        else:
-                            inst_req.append(line)
-                    info['Install-requires'] = inst_req
-                    info['Extras-require'] = extras_req
-        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
-            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
-
-            if setup_info:
-                if 'Install-requires' in setup_info:
-                    info['Install-requires'] = setup_info['Install-requires']
-                if 'Extras-require' in setup_info:
-                    info['Extras-require'] = setup_info['Extras-require']
-        else:
-            if setup_info:
-                info = setup_info
-            else:
-                info = self.get_setup_args_info(setupscript)
-
-        # Grab the license value before applying replacements
-        license_str = info.get('License', '').strip()
-
-        self.apply_info_replacements(info)
-
-        if uses_setuptools:
-            classes.append('setuptools3')
-        else:
-            classes.append('distutils3')
-
-        if license_str:
-            for i, line in enumerate(lines_before):
-                if line.startswith('##LICENSE_PLACEHOLDER##'):
-                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
-                    break
-
-        if 'Classifier' in info:
-            existing_licenses = info.get('License', '')
-            licenses = []
-            for classifier in info['Classifier']:
-                if classifier in self.classifier_license_map:
-                    license = self.classifier_license_map[classifier]
-                    if license == 'Apache' and 'Apache-2.0' in existing_licenses:
-                        license = 'Apache-2.0'
-                    elif license == 'GPL':
-                        if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
-                            license = 'GPL-2.0'
-                        elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
-                            license = 'GPL-3.0'
-                    elif license == 'LGPL':
-                        if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
-                            license = 'LGPL-2.1'
-                        elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
-                            license = 'LGPL-2.0'
-                        elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
-                            license = 'LGPL-3.0'
-                    licenses.append(license)
-
-            if licenses:
-                info['License'] = ' & '.join(licenses)
+    def handle_classifier_license(self, classifiers, existing_licenses=""):
+
+        licenses = []
+        for classifier in classifiers:
+            if classifier in self.classifier_license_map:
+                license = self.classifier_license_map[classifier]
+                if license == 'Apache' and 'Apache-2.0' in existing_licenses:
+                    license = 'Apache-2.0'
+                elif license == 'GPL':
+                    if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
+                        license = 'GPL-2.0'
+                    elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
+                        license = 'GPL-3.0'
+                elif license == 'LGPL':
+                    if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
+                        license = 'LGPL-2.1'
+                    elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
+                        license = 'LGPL-2.0'
+                    elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
+                        license = 'LGPL-3.0'
+                licenses.append(license)
+
+        if licenses:
+            return ' & '.join(licenses)
+
+        return None
+
+    def map_info_to_bbvar(self, info, extravalues):
 
         # Map PKG-INFO & setup.py fields to bitbake variables
         for field, values in info.items():
@@ -305,85 +162,220 @@ class PythonRecipeHandler(RecipeHandler):
             if bbvar not in extravalues and value:
                 extravalues[bbvar] = value
 
-        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
-
-        extras_req = set()
-        if 'Extras-require' in info:
-            extras_req = info['Extras-require']
-            if extras_req:
-                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
-                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
-                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
-                lines_after.append('#')
-                lines_after.append('# Uncomment this line to enable all the optional features.')
-                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
-                for feature, feature_reqs in extras_req.items():
-                    unmapped_deps.difference_update(feature_reqs)
-
-                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
-                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
-
-        inst_reqs = set()
-        if 'Install-requires' in info:
-            if extras_req:
-                lines_after.append('')
-            inst_reqs = info['Install-requires']
-            if inst_reqs:
-                unmapped_deps.difference_update(inst_reqs)
-
-                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
-                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
-                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+    def apply_info_replacements(self, info):
+        if not self.replacements:
+            return
 
-        if mapped_deps:
-            name = info.get('Name')
-            if name and name[0] in mapped_deps:
-                # Attempt to avoid self-reference
-                mapped_deps.remove(name[0])
-            mapped_deps -= set(self.excluded_pkgdeps)
-            if inst_reqs or extras_req:
-                lines_after.append('')
-            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
-            lines_after.append('# python sources, and might not be 100% accurate.')
-            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+        for variable, search, replace in self.replacements:
+            if variable not in info:
+                continue
 
-        unmapped_deps -= set(extensions)
-        unmapped_deps -= set(self.assume_provided)
-        if unmapped_deps:
-            if mapped_deps:
-                lines_after.append('')
-            lines_after.append('# WARNING: We were unable to map the following python package/module')
-            lines_after.append('# dependencies to the bitbake packages which include them:')
-            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+            def replace_value(search, replace, value):
+                if replace is None:
+                    if re.search(search, value):
+                        return None
+                else:
+                    new_value = re.sub(search, replace, value)
+                    if value != new_value:
+                        return new_value
+                return value
 
-        handled.append('buildsystem')
+            value = info[variable]
+            if isinstance(value, str):
+                new_value = replace_value(search, replace, value)
+                if new_value is None:
+                    del info[variable]
+                elif new_value != value:
+                    info[variable] = new_value
+            elif hasattr(value, 'items'):
+                for dkey, dvalue in list(value.items()):
+                    new_list = []
+                    for pos, a_value in enumerate(dvalue):
+                        new_value = replace_value(search, replace, a_value)
+                        if new_value is not None and new_value != value:
+                            new_list.append(new_value)
 
-    def get_pkginfo(self, pkginfo_fn):
-        msg = email.message_from_file(open(pkginfo_fn, 'r'))
-        msginfo = {}
-        for field in msg.keys():
-            values = msg.get_all(field)
-            if len(values) == 1:
-                msginfo[field] = values[0]
+                    if value != new_list:
+                        value[dkey] = new_list
             else:
-                msginfo[field] = values
-        return msginfo
+                new_list = []
+                for pos, a_value in enumerate(value):
+                    new_value = replace_value(search, replace, a_value)
+                    if new_value is not None and new_value != value:
+                        new_list.append(new_value)
 
-    def parse_setup_py(self, setupscript='./setup.py'):
-        with codecs.open(setupscript) as f:
-            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+                if value != new_list:
+                    info[variable] = new_list
 
-        def _map(key):
-            key = key.replace('_', '-')
-            key = key[0].upper() + key[1:]
-            if key in self.setup_parse_map:
-                key = self.setup_parse_map[key]
-            return key
 
-        # Naive mapping of setup() arguments to PKG-INFO field names
-        for d in [info, non_literals]:
+    def scan_python_dependencies(self, paths):
+        deps = set()
+        try:
+            dep_output = self.run_command(['pythondeps', '-d'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            for line in dep_output.splitlines():
+                line = line.rstrip()
+                dep, filename = line.split('\t', 1)
+                if filename.endswith('/setup.py'):
+                    continue
+                deps.add(dep)
+
+        try:
+            provides_output = self.run_command(['pythondeps', '-p'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            provides_lines = (l.rstrip() for l in provides_output.splitlines())
+            provides = set(l for l in provides_lines if l and l != 'setup')
+            deps -= provides
+
+        return deps
+
+    def parse_pkgdata_for_python_packages(self):
+        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+
+        ldata = tinfoil.config_data.createCopy()
+        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
+        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+
+        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
+        python_dirs = [python_sitedir + os.sep,
+                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
+                       os.path.dirname(python_sitedir) + os.sep]
+        packages = {}
+        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
+            files_info = None
+            with open(pkgdatafile, 'r') as f:
+                for line in f.readlines():
+                    field, value = line.split(': ', 1)
+                    if field.startswith('FILES_INFO'):
+                        files_info = ast.literal_eval(value)
+                        break
+                else:
+                    continue
+
+            for fn in files_info:
+                for suffix in importlib.machinery.all_suffixes():
+                    if fn.endswith(suffix):
+                        break
+                else:
+                    continue
+
+                if fn.startswith(dynload_dir + os.sep):
+                    if '/.debug/' in fn:
+                        continue
+                    base = os.path.basename(fn)
+                    provided = base.split('.', 1)[0]
+                    packages[provided] = os.path.basename(pkgdatafile)
+                    continue
+
+                for python_dir in python_dirs:
+                    if fn.startswith(python_dir):
+                        relpath = fn[len(python_dir):]
+                        relstart, _, relremaining = relpath.partition(os.sep)
+                        if relstart.endswith('.egg'):
+                            relpath = relremaining
+                        base, _ = os.path.splitext(relpath)
+
+                        if '/.debug/' in base:
+                            continue
+                        if os.path.basename(base) == '__init__':
+                            base = os.path.dirname(base)
+                        base = base.replace(os.sep + os.sep, os.sep)
+                        provided = base.replace(os.sep, '.')
+                        packages[provided] = os.path.basename(pkgdatafile)
+        return packages
+
+    @classmethod
+    def run_command(cls, cmd, **popenargs):
+        if 'stderr' not in popenargs:
+            popenargs['stderr'] = subprocess.STDOUT
+        try:
+            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
+        except OSError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
+            raise
+        except subprocess.CalledProcessError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
+            raise
+
+class PythonSetupPyRecipeHandler(PythonRecipeHandler):
+    bbvar_map = {
+        'Name': 'PN',
+        'Version': 'PV',
+        'Home-page': 'HOMEPAGE',
+        'Summary': 'SUMMARY',
+        'Description': 'DESCRIPTION',
+        'License': 'LICENSE',
+        'Requires': 'RDEPENDS:${PN}',
+        'Provides': 'RPROVIDES:${PN}',
+        'Obsoletes': 'RREPLACES:${PN}',
+    }
+    # PN/PV are already set by recipetool core & desc can be extremely long
+    excluded_fields = [
+        'Description',
+    ]
+    setup_parse_map = {
+        'Url': 'Home-page',
+        'Classifiers': 'Classifier',
+        'Description': 'Summary',
+    }
+    setuparg_map = {
+        'Home-page': 'url',
+        'Classifier': 'classifiers',
+        'Summary': 'description',
+        'Description': 'long-description',
+    }
+    # Values which are lists, used by the setup.py argument based metadata
+    # extraction method, to determine how to process the setup.py output.
+    setuparg_list_fields = [
+        'Classifier',
+        'Requires',
+        'Provides',
+        'Obsoletes',
+        'Platform',
+        'Supported-Platform',
+    ]
+    setuparg_multi_line_values = ['Description']
+
+    replacements = [
+        ('License', r' +$', ''),
+        ('License', r'^ +', ''),
+        ('License', r' ', '-'),
+        ('License', r'^GNU-', ''),
+        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
+        ('License', r'^UNKNOWN$', ''),
+
+        # Remove currently unhandled version numbers from these variables
+        ('Requires', r' *\([^)]*\)', ''),
+        ('Provides', r' *\([^)]*\)', ''),
+        ('Obsoletes', r' *\([^)]*\)', ''),
+        ('Install-requires', r'^([^><= ]+).*', r'\1'),
+        ('Extras-require', r'^([^><= ]+).*', r'\1'),
+        ('Tests-require', r'^([^><= ]+).*', r'\1'),
+
+        # Remove unhandled dependency on particular features (e.g. foo[PDF])
+        ('Install-requires', r'\[[^\]]+\]$', ''),
+    ]
+
+    def __init__(self):
+        pass
+
+    def parse_setup_py(self, setupscript='./setup.py'):
+        with codecs.open(setupscript) as f:
+            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+
+        def _map(key):
+            key = key.replace('_', '-')
+            key = key[0].upper() + key[1:]
+            if key in self.setup_parse_map:
+                key = self.setup_parse_map[key]
+            return key
+
+        # Naive mapping of setup() arguments to PKG-INFO field names
+        for d in [info, non_literals]:
             for key, value in list(d.items()):
                 if key is None:
                     continue
@@ -445,47 +437,16 @@ class PythonRecipeHandler(RecipeHandler):
                 info[fields[lineno]] = line
         return info
 
-    def apply_info_replacements(self, info):
-        for variable, search, replace in self.replacements:
-            if variable not in info:
-                continue
-
-            def replace_value(search, replace, value):
-                if replace is None:
-                    if re.search(search, value):
-                        return None
-                else:
-                    new_value = re.sub(search, replace, value)
-                    if value != new_value:
-                        return new_value
-                return value
-
-            value = info[variable]
-            if isinstance(value, str):
-                new_value = replace_value(search, replace, value)
-                if new_value is None:
-                    del info[variable]
-                elif new_value != value:
-                    info[variable] = new_value
-            elif hasattr(value, 'items'):
-                for dkey, dvalue in list(value.items()):
-                    new_list = []
-                    for pos, a_value in enumerate(dvalue):
-                        new_value = replace_value(search, replace, a_value)
-                        if new_value is not None and new_value != value:
-                            new_list.append(new_value)
-
-                    if value != new_list:
-                        value[dkey] = new_list
+    def get_pkginfo(self, pkginfo_fn):
+        msg = email.message_from_file(open(pkginfo_fn, 'r'))
+        msginfo = {}
+        for field in msg.keys():
+            values = msg.get_all(field)
+            if len(values) == 1:
+                msginfo[field] = values[0]
             else:
-                new_list = []
-                for pos, a_value in enumerate(value):
-                    new_value = replace_value(search, replace, a_value)
-                    if new_value is not None and new_value != value:
-                        new_list.append(new_value)
-
-                if value != new_list:
-                    info[variable] = new_list
+                msginfo[field] = values
+        return msginfo
 
     def scan_setup_python_deps(self, srctree, setup_info, setup_non_literals):
         if 'Package-dir' in setup_info:
@@ -540,99 +501,160 @@ class PythonRecipeHandler(RecipeHandler):
                 unmapped_deps.add(dep)
         return mapped_deps, unmapped_deps
 
-    def scan_python_dependencies(self, paths):
-        deps = set()
-        try:
-            dep_output = self.run_command(['pythondeps', '-d'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
         else:
-            for line in dep_output.splitlines():
-                line = line.rstrip()
-                dep, filename = line.split('\t', 1)
-                if filename.endswith('/setup.py'):
-                    continue
-                deps.add(dep)
+            return False
+
+        # setup.py is always parsed to get at certain required information, such as
+        # distutils vs setuptools
+        #
+        # If egg info is available, we use it for both its PKG-INFO metadata
+        # and for its requires.txt for install_requires.
+        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
+        # the parsed setup.py, but use the install_requires info from the
+        # parsed setup.py.
 
+        setupscript = os.path.join(srctree, 'setup.py')
         try:
-            provides_output = self.run_command(['pythondeps', '-p'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
+        except Exception:
+            logger.exception("Failed to parse setup.py")
+            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
+
+        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
+        if egginfo:
+            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
+            requires_txt = os.path.join(egginfo[0], 'requires.txt')
+            if os.path.exists(requires_txt):
+                with codecs.open(requires_txt) as f:
+                    inst_req = []
+                    extras_req = collections.defaultdict(list)
+                    current_feature = None
+                    for line in f.readlines():
+                        line = line.rstrip()
+                        if not line:
+                            continue
+
+                        if line.startswith('['):
+                            # PACKAGECONFIG must not contain expressions or whitespace
+                            line = line.replace(" ", "")
+                            line = line.replace(':', "")
+                            line = line.replace('.', "-dot-")
+                            line = line.replace('"', "")
+                            line = line.replace('<', "-smaller-")
+                            line = line.replace('>', "-bigger-")
+                            line = line.replace('_', "-")
+                            line = line.replace('(', "")
+                            line = line.replace(')', "")
+                            line = line.replace('!', "-not-")
+                            line = line.replace('=', "-equals-")
+                            current_feature = line[1:-1]
+                        elif current_feature:
+                            extras_req[current_feature].append(line)
+                        else:
+                            inst_req.append(line)
+                    info['Install-requires'] = inst_req
+                    info['Extras-require'] = extras_req
+        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
+            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
+
+            if setup_info:
+                if 'Install-requires' in setup_info:
+                    info['Install-requires'] = setup_info['Install-requires']
+                if 'Extras-require' in setup_info:
+                    info['Extras-require'] = setup_info['Extras-require']
         else:
-            provides_lines = (l.rstrip() for l in provides_output.splitlines())
-            provides = set(l for l in provides_lines if l and l != 'setup')
-            deps -= provides
+            if setup_info:
+                info = setup_info
+            else:
+                info = self.get_setup_args_info(setupscript)
 
-        return deps
+        # Grab the license value before applying replacements
+        license_str = info.get('License', '').strip()
 
-    def parse_pkgdata_for_python_packages(self):
-        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+        self.apply_info_replacements(info)
 
-        ldata = tinfoil.config_data.createCopy()
-        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
-        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+        if uses_setuptools:
+            classes.append('setuptools3')
+        else:
+            classes.append('distutils3')
 
-        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
-        python_dirs = [python_sitedir + os.sep,
-                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
-                       os.path.dirname(python_sitedir) + os.sep]
-        packages = {}
-        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
-            files_info = None
-            with open(pkgdatafile, 'r') as f:
-                for line in f.readlines():
-                    field, value = line.split(': ', 1)
-                    if field.startswith('FILES_INFO'):
-                        files_info = ast.literal_eval(value)
-                        break
-                else:
-                    continue
+        if license_str:
+            for i, line in enumerate(lines_before):
+                if line.startswith('##LICENSE_PLACEHOLDER##'):
+                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
+                    break
 
-            for fn in files_info:
-                for suffix in importlib.machinery.all_suffixes():
-                    if fn.endswith(suffix):
-                        break
-                else:
-                    continue
+        if 'Classifier' in info:
+            license = self.handle_classifier_license(info['Classifier'], info.get('License', ''))
+            if license:
+                info['License'] = license
 
-                if fn.startswith(dynload_dir + os.sep):
-                    if '/.debug/' in fn:
-                        continue
-                    base = os.path.basename(fn)
-                    provided = base.split('.', 1)[0]
-                    packages[provided] = os.path.basename(pkgdatafile)
-                    continue
+        self.map_info_to_bbvar(info, extravalues)
 
-                for python_dir in python_dirs:
-                    if fn.startswith(python_dir):
-                        relpath = fn[len(python_dir):]
-                        relstart, _, relremaining = relpath.partition(os.sep)
-                        if relstart.endswith('.egg'):
-                            relpath = relremaining
-                        base, _ = os.path.splitext(relpath)
+        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
 
-                        if '/.debug/' in base:
-                            continue
-                        if os.path.basename(base) == '__init__':
-                            base = os.path.dirname(base)
-                        base = base.replace(os.sep + os.sep, os.sep)
-                        provided = base.replace(os.sep, '.')
-                        packages[provided] = os.path.basename(pkgdatafile)
-        return packages
+        extras_req = set()
+        if 'Extras-require' in info:
+            extras_req = info['Extras-require']
+            if extras_req:
+                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
+                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
+                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
+                lines_after.append('#')
+                lines_after.append('# Uncomment this line to enable all the optional features.')
+                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
+                for feature, feature_reqs in extras_req.items():
+                    unmapped_deps.difference_update(feature_reqs)
 
-    @classmethod
-    def run_command(cls, cmd, **popenargs):
-        if 'stderr' not in popenargs:
-            popenargs['stderr'] = subprocess.STDOUT
-        try:
-            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
-        except OSError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
-            raise
-        except subprocess.CalledProcessError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
-            raise
+                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
+                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
 
+        inst_reqs = set()
+        if 'Install-requires' in info:
+            if extras_req:
+                lines_after.append('')
+            inst_reqs = info['Install-requires']
+            if inst_reqs:
+                unmapped_deps.difference_update(inst_reqs)
+
+                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
+                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
+                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+
+        if mapped_deps:
+            name = info.get('Name')
+            if name and name[0] in mapped_deps:
+                # Attempt to avoid self-reference
+                mapped_deps.remove(name[0])
+            mapped_deps -= set(self.excluded_pkgdeps)
+            if inst_reqs or extras_req:
+                lines_after.append('')
+            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
+            lines_after.append('# python sources, and might not be 100% accurate.')
+            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+
+        unmapped_deps -= set(extensions)
+        unmapped_deps -= set(self.assume_provided)
+        if unmapped_deps:
+            if mapped_deps:
+                lines_after.append('')
+            lines_after.append('# WARNING: We were unable to map the following python package/module')
+            lines_after.append('# dependencies to the bitbake packages which include them:')
+            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+
+        handled.append('buildsystem')
 
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
@@ -748,4 +770,4 @@ def has_non_literals(value):
 
 def register_recipe_handlers(handlers):
     # We need to make sure this is ahead of the makefile fallback handler
-    handlers.append((PythonRecipeHandler(), 70))
+    handlers.append((PythonSetupPyRecipeHandler(), 70))
Signature · 1 line
2.42.0
#3 lore raw

Re: [OE-core] [PATCH v2 3/4] scripts:recipetool:create_buildsys_python: refactor code for futur PEP517 addition

Hello,

Quoted 6 lines · Julien Stephan

In order to prepare the support for pyproject.toml (PEP517 [1]) enabled projects, refactor the code and move setup.py specific code into a specific class in order to allow sharing the PythonRecipeHandler class

No functionnal changes expected

I tested with only the first 3 patches and unfortunately, thre were functional changes:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio

2023-10-19 07:23:07,712 - oe-selftest - INFO - 1: 20/39 149/543 (20.20s) (0 failed) (recipetool.RecipetoolCreateTests.test_recipetool_create_github)
2023-10-19 07:23:07,712 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 451, in test_recipetool_create_github
    self.assertTrue(os.path.isfile(recipefile))
  File "/usr/lib/python3.11/unittest/case.py", line 715, in assertTrue
    raise self.failureException(msg)

AssertionError: False is not true

Quoted 835 lines

[1]: https://peps.python.org/pep-0517/#source-tree

---
 .../lib/recipetool/create_buildsys_python.py  | 748 +++++++++---------
 1 file changed, 385 insertions(+), 363 deletions(-)
scripts/lib/recipetool/create_buildsys_python.py +385 -363
index 502e1dfbc3d..69f6f5ca511 100644
@@ -37,63 +37,8 @@ class PythonRecipeHandler(RecipeHandler):
     assume_provided = ['builtins', 'os.path']
     # Assumes that the host python3 builtin_module_names is sane for target too
     assume_provided = assume_provided + list(sys.builtin_module_names)
+    excluded_fields = []
 
-    bbvar_map = {
-        'Name': 'PN',
-        'Version': 'PV',
-        'Home-page': 'HOMEPAGE',
-        'Summary': 'SUMMARY',
-        'Description': 'DESCRIPTION',
-        'License': 'LICENSE',
-        'Requires': 'RDEPENDS:${PN}',
-        'Provides': 'RPROVIDES:${PN}',
-        'Obsoletes': 'RREPLACES:${PN}',
-    }
-    # PN/PV are already set by recipetool core & desc can be extremely long
-    excluded_fields = [
-        'Description',
-    ]
-    setup_parse_map = {
-        'Url': 'Home-page',
-        'Classifiers': 'Classifier',
-        'Description': 'Summary',
-    }
-    setuparg_map = {
-        'Home-page': 'url',
-        'Classifier': 'classifiers',
-        'Summary': 'description',
-        'Description': 'long-description',
-    }
-    # Values which are lists, used by the setup.py argument based metadata
-    # extraction method, to determine how to process the setup.py output.
-    setuparg_list_fields = [
-        'Classifier',
-        'Requires',
-        'Provides',
-        'Obsoletes',
-        'Platform',
-        'Supported-Platform',
-    ]
-    setuparg_multi_line_values = ['Description']
-    replacements = [
-        ('License', r' +$', ''),
-        ('License', r'^ +', ''),
-        ('License', r' ', '-'),
-        ('License', r'^GNU-', ''),
-        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
-        ('License', r'^UNKNOWN$', ''),
-
-        # Remove currently unhandled version numbers from these variables
-        ('Requires', r' *\([^)]*\)', ''),
-        ('Provides', r' *\([^)]*\)', ''),
-        ('Obsoletes', r' *\([^)]*\)', ''),
-        ('Install-requires', r'^([^><= ]+).*', r'\1'),
-        ('Extras-require', r'^([^><= ]+).*', r'\1'),
-        ('Tests-require', r'^([^><= ]+).*', r'\1'),
-
-        # Remove unhandled dependency on particular features (e.g. foo[PDF])
-        ('Install-requires', r'\[[^\]]+\]$', ''),
-    ]
 
     classifier_license_map = {
         'License :: OSI Approved :: Academic Free License (AFL)': 'AFL',
@@ -166,122 +111,34 @@ class PythonRecipeHandler(RecipeHandler):
     def __init__(self):
         pass
 
-    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
-        if 'buildsystem' in handled:
-            return False
-
-        # Check for non-zero size setup.py files
-        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
-        for fn in setupfiles:
-            if os.path.getsize(fn):
-                break
-        else:
-            return False
-
-        # setup.py is always parsed to get at certain required information, such as
-        # distutils vs setuptools
-        #
-        # If egg info is available, we use it for both its PKG-INFO metadata
-        # and for its requires.txt for install_requires.
-        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
-        # the parsed setup.py, but use the install_requires info from the
-        # parsed setup.py.
-
-        setupscript = os.path.join(srctree, 'setup.py')
-        try:
-            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
-        except Exception:
-            logger.exception("Failed to parse setup.py")
-            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
-
-        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
-        if egginfo:
-            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
-            requires_txt = os.path.join(egginfo[0], 'requires.txt')
-            if os.path.exists(requires_txt):
-                with codecs.open(requires_txt) as f:
-                    inst_req = []
-                    extras_req = collections.defaultdict(list)
-                    current_feature = None
-                    for line in f.readlines():
-                        line = line.rstrip()
-                        if not line:
-                            continue
-
-                        if line.startswith('['):
-                            # PACKAGECONFIG must not contain expressions or whitespace
-                            line = line.replace(" ", "")
-                            line = line.replace(':', "")
-                            line = line.replace('.', "-dot-")
-                            line = line.replace('"', "")
-                            line = line.replace('<', "-smaller-")
-                            line = line.replace('>', "-bigger-")
-                            line = line.replace('_', "-")
-                            line = line.replace('(', "")
-                            line = line.replace(')', "")
-                            line = line.replace('!', "-not-")
-                            line = line.replace('=', "-equals-")
-                            current_feature = line[1:-1]
-                        elif current_feature:
-                            extras_req[current_feature].append(line)
-                        else:
-                            inst_req.append(line)
-                    info['Install-requires'] = inst_req
-                    info['Extras-require'] = extras_req
-        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
-            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
-
-            if setup_info:
-                if 'Install-requires' in setup_info:
-                    info['Install-requires'] = setup_info['Install-requires']
-                if 'Extras-require' in setup_info:
-                    info['Extras-require'] = setup_info['Extras-require']
-        else:
-            if setup_info:
-                info = setup_info
-            else:
-                info = self.get_setup_args_info(setupscript)
-
-        # Grab the license value before applying replacements
-        license_str = info.get('License', '').strip()
-
-        self.apply_info_replacements(info)
-
-        if uses_setuptools:
-            classes.append('setuptools3')
-        else:
-            classes.append('distutils3')
-
-        if license_str:
-            for i, line in enumerate(lines_before):
-                if line.startswith('##LICENSE_PLACEHOLDER##'):
-                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
-                    break
-
-        if 'Classifier' in info:
-            existing_licenses = info.get('License', '')
-            licenses = []
-            for classifier in info['Classifier']:
-                if classifier in self.classifier_license_map:
-                    license = self.classifier_license_map[classifier]
-                    if license == 'Apache' and 'Apache-2.0' in existing_licenses:
-                        license = 'Apache-2.0'
-                    elif license == 'GPL':
-                        if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
-                            license = 'GPL-2.0'
-                        elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
-                            license = 'GPL-3.0'
-                    elif license == 'LGPL':
-                        if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
-                            license = 'LGPL-2.1'
-                        elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
-                            license = 'LGPL-2.0'
-                        elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
-                            license = 'LGPL-3.0'
-                    licenses.append(license)
-
-            if licenses:
-                info['License'] = ' & '.join(licenses)
+    def handle_classifier_license(self, classifiers, existing_licenses=""):
+
+        licenses = []
+        for classifier in classifiers:
+            if classifier in self.classifier_license_map:
+                license = self.classifier_license_map[classifier]
+                if license == 'Apache' and 'Apache-2.0' in existing_licenses:
+                    license = 'Apache-2.0'
+                elif license == 'GPL':
+                    if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
+                        license = 'GPL-2.0'
+                    elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
+                        license = 'GPL-3.0'
+                elif license == 'LGPL':
+                    if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
+                        license = 'LGPL-2.1'
+                    elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
+                        license = 'LGPL-2.0'
+                    elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
+                        license = 'LGPL-3.0'
+                licenses.append(license)
+
+        if licenses:
+            return ' & '.join(licenses)
+
+        return None
+
+    def map_info_to_bbvar(self, info, extravalues):
 
         # Map PKG-INFO & setup.py fields to bitbake variables
         for field, values in info.items():
@@ -305,85 +162,220 @@ class PythonRecipeHandler(RecipeHandler):
             if bbvar not in extravalues and value:
                 extravalues[bbvar] = value
 
-        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
-
-        extras_req = set()
-        if 'Extras-require' in info:
-            extras_req = info['Extras-require']
-            if extras_req:
-                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
-                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
-                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
-                lines_after.append('#')
-                lines_after.append('# Uncomment this line to enable all the optional features.')
-                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
-                for feature, feature_reqs in extras_req.items():
-                    unmapped_deps.difference_update(feature_reqs)
-
-                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
-                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
-
-        inst_reqs = set()
-        if 'Install-requires' in info:
-            if extras_req:
-                lines_after.append('')
-            inst_reqs = info['Install-requires']
-            if inst_reqs:
-                unmapped_deps.difference_update(inst_reqs)
-
-                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
-                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
-                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+    def apply_info_replacements(self, info):
+        if not self.replacements:
+            return
 
-        if mapped_deps:
-            name = info.get('Name')
-            if name and name[0] in mapped_deps:
-                # Attempt to avoid self-reference
-                mapped_deps.remove(name[0])
-            mapped_deps -= set(self.excluded_pkgdeps)
-            if inst_reqs or extras_req:
-                lines_after.append('')
-            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
-            lines_after.append('# python sources, and might not be 100% accurate.')
-            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+        for variable, search, replace in self.replacements:
+            if variable not in info:
+                continue
 
-        unmapped_deps -= set(extensions)
-        unmapped_deps -= set(self.assume_provided)
-        if unmapped_deps:
-            if mapped_deps:
-                lines_after.append('')
-            lines_after.append('# WARNING: We were unable to map the following python package/module')
-            lines_after.append('# dependencies to the bitbake packages which include them:')
-            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+            def replace_value(search, replace, value):
+                if replace is None:
+                    if re.search(search, value):
+                        return None
+                else:
+                    new_value = re.sub(search, replace, value)
+                    if value != new_value:
+                        return new_value
+                return value
 
-        handled.append('buildsystem')
+            value = info[variable]
+            if isinstance(value, str):
+                new_value = replace_value(search, replace, value)
+                if new_value is None:
+                    del info[variable]
+                elif new_value != value:
+                    info[variable] = new_value
+            elif hasattr(value, 'items'):
+                for dkey, dvalue in list(value.items()):
+                    new_list = []
+                    for pos, a_value in enumerate(dvalue):
+                        new_value = replace_value(search, replace, a_value)
+                        if new_value is not None and new_value != value:
+                            new_list.append(new_value)
 
-    def get_pkginfo(self, pkginfo_fn):
-        msg = email.message_from_file(open(pkginfo_fn, 'r'))
-        msginfo = {}
-        for field in msg.keys():
-            values = msg.get_all(field)
-            if len(values) == 1:
-                msginfo[field] = values[0]
+                    if value != new_list:
+                        value[dkey] = new_list
             else:
-                msginfo[field] = values
-        return msginfo
+                new_list = []
+                for pos, a_value in enumerate(value):
+                    new_value = replace_value(search, replace, a_value)
+                    if new_value is not None and new_value != value:
+                        new_list.append(new_value)
 
-    def parse_setup_py(self, setupscript='./setup.py'):
-        with codecs.open(setupscript) as f:
-            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+                if value != new_list:
+                    info[variable] = new_list
 
-        def _map(key):
-            key = key.replace('_', '-')
-            key = key[0].upper() + key[1:]
-            if key in self.setup_parse_map:
-                key = self.setup_parse_map[key]
-            return key
 
-        # Naive mapping of setup() arguments to PKG-INFO field names
-        for d in [info, non_literals]:
+    def scan_python_dependencies(self, paths):
+        deps = set()
+        try:
+            dep_output = self.run_command(['pythondeps', '-d'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            for line in dep_output.splitlines():
+                line = line.rstrip()
+                dep, filename = line.split('\t', 1)
+                if filename.endswith('/setup.py'):
+                    continue
+                deps.add(dep)
+
+        try:
+            provides_output = self.run_command(['pythondeps', '-p'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            provides_lines = (l.rstrip() for l in provides_output.splitlines())
+            provides = set(l for l in provides_lines if l and l != 'setup')
+            deps -= provides
+
+        return deps
+
+    def parse_pkgdata_for_python_packages(self):
+        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+
+        ldata = tinfoil.config_data.createCopy()
+        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
+        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+
+        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
+        python_dirs = [python_sitedir + os.sep,
+                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
+                       os.path.dirname(python_sitedir) + os.sep]
+        packages = {}
+        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
+            files_info = None
+            with open(pkgdatafile, 'r') as f:
+                for line in f.readlines():
+                    field, value = line.split(': ', 1)
+                    if field.startswith('FILES_INFO'):
+                        files_info = ast.literal_eval(value)
+                        break
+                else:
+                    continue
+
+            for fn in files_info:
+                for suffix in importlib.machinery.all_suffixes():
+                    if fn.endswith(suffix):
+                        break
+                else:
+                    continue
+
+                if fn.startswith(dynload_dir + os.sep):
+                    if '/.debug/' in fn:
+                        continue
+                    base = os.path.basename(fn)
+                    provided = base.split('.', 1)[0]
+                    packages[provided] = os.path.basename(pkgdatafile)
+                    continue
+
+                for python_dir in python_dirs:
+                    if fn.startswith(python_dir):
+                        relpath = fn[len(python_dir):]
+                        relstart, _, relremaining = relpath.partition(os.sep)
+                        if relstart.endswith('.egg'):
+                            relpath = relremaining
+                        base, _ = os.path.splitext(relpath)
+
+                        if '/.debug/' in base:
+                            continue
+                        if os.path.basename(base) == '__init__':
+                            base = os.path.dirname(base)
+                        base = base.replace(os.sep + os.sep, os.sep)
+                        provided = base.replace(os.sep, '.')
+                        packages[provided] = os.path.basename(pkgdatafile)
+        return packages
+
+    @classmethod
+    def run_command(cls, cmd, **popenargs):
+        if 'stderr' not in popenargs:
+            popenargs['stderr'] = subprocess.STDOUT
+        try:
+            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
+        except OSError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
+            raise
+        except subprocess.CalledProcessError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
+            raise
+
+class PythonSetupPyRecipeHandler(PythonRecipeHandler):
+    bbvar_map = {
+        'Name': 'PN',
+        'Version': 'PV',
+        'Home-page': 'HOMEPAGE',
+        'Summary': 'SUMMARY',
+        'Description': 'DESCRIPTION',
+        'License': 'LICENSE',
+        'Requires': 'RDEPENDS:${PN}',
+        'Provides': 'RPROVIDES:${PN}',
+        'Obsoletes': 'RREPLACES:${PN}',
+    }
+    # PN/PV are already set by recipetool core & desc can be extremely long
+    excluded_fields = [
+        'Description',
+    ]
+    setup_parse_map = {
+        'Url': 'Home-page',
+        'Classifiers': 'Classifier',
+        'Description': 'Summary',
+    }
+    setuparg_map = {
+        'Home-page': 'url',
+        'Classifier': 'classifiers',
+        'Summary': 'description',
+        'Description': 'long-description',
+    }
+    # Values which are lists, used by the setup.py argument based metadata
+    # extraction method, to determine how to process the setup.py output.
+    setuparg_list_fields = [
+        'Classifier',
+        'Requires',
+        'Provides',
+        'Obsoletes',
+        'Platform',
+        'Supported-Platform',
+    ]
+    setuparg_multi_line_values = ['Description']
+
+    replacements = [
+        ('License', r' +$', ''),
+        ('License', r'^ +', ''),
+        ('License', r' ', '-'),
+        ('License', r'^GNU-', ''),
+        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
+        ('License', r'^UNKNOWN$', ''),
+
+        # Remove currently unhandled version numbers from these variables
+        ('Requires', r' *\([^)]*\)', ''),
+        ('Provides', r' *\([^)]*\)', ''),
+        ('Obsoletes', r' *\([^)]*\)', ''),
+        ('Install-requires', r'^([^><= ]+).*', r'\1'),
+        ('Extras-require', r'^([^><= ]+).*', r'\1'),
+        ('Tests-require', r'^([^><= ]+).*', r'\1'),
+
+        # Remove unhandled dependency on particular features (e.g. foo[PDF])
+        ('Install-requires', r'\[[^\]]+\]$', ''),
+    ]
+
+    def __init__(self):
+        pass
+
+    def parse_setup_py(self, setupscript='./setup.py'):
+        with codecs.open(setupscript) as f:
+            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+
+        def _map(key):
+            key = key.replace('_', '-')
+            key = key[0].upper() + key[1:]
+            if key in self.setup_parse_map:
+                key = self.setup_parse_map[key]
+            return key
+
+        # Naive mapping of setup() arguments to PKG-INFO field names
+        for d in [info, non_literals]:
             for key, value in list(d.items()):
                 if key is None:
                     continue
@@ -445,47 +437,16 @@ class PythonRecipeHandler(RecipeHandler):
                 info[fields[lineno]] = line
         return info
 
-    def apply_info_replacements(self, info):
-        for variable, search, replace in self.replacements:
-            if variable not in info:
-                continue
-
-            def replace_value(search, replace, value):
-                if replace is None:
-                    if re.search(search, value):
-                        return None
-                else:
-                    new_value = re.sub(search, replace, value)
-                    if value != new_value:
-                        return new_value
-                return value
-
-            value = info[variable]
-            if isinstance(value, str):
-                new_value = replace_value(search, replace, value)
-                if new_value is None:
-                    del info[variable]
-                elif new_value != value:
-                    info[variable] = new_value
-            elif hasattr(value, 'items'):
-                for dkey, dvalue in list(value.items()):
-                    new_list = []
-                    for pos, a_value in enumerate(dvalue):
-                        new_value = replace_value(search, replace, a_value)
-                        if new_value is not None and new_value != value:
-                            new_list.append(new_value)
-
-                    if value != new_list:
-                        value[dkey] = new_list
+    def get_pkginfo(self, pkginfo_fn):
+        msg = email.message_from_file(open(pkginfo_fn, 'r'))
+        msginfo = {}
+        for field in msg.keys():
+            values = msg.get_all(field)
+            if len(values) == 1:
+                msginfo[field] = values[0]
             else:
-                new_list = []
-                for pos, a_value in enumerate(value):
-                    new_value = replace_value(search, replace, a_value)
-                    if new_value is not None and new_value != value:
-                        new_list.append(new_value)
-
-                if value != new_list:
-                    info[variable] = new_list
+                msginfo[field] = values
+        return msginfo
 
     def scan_setup_python_deps(self, srctree, setup_info, setup_non_literals):
         if 'Package-dir' in setup_info:
@@ -540,99 +501,160 @@ class PythonRecipeHandler(RecipeHandler):
                 unmapped_deps.add(dep)
         return mapped_deps, unmapped_deps
 
-    def scan_python_dependencies(self, paths):
-        deps = set()
-        try:
-            dep_output = self.run_command(['pythondeps', '-d'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
         else:
-            for line in dep_output.splitlines():
-                line = line.rstrip()
-                dep, filename = line.split('\t', 1)
-                if filename.endswith('/setup.py'):
-                    continue
-                deps.add(dep)
+            return False
+
+        # setup.py is always parsed to get at certain required information, such as
+        # distutils vs setuptools
+        #
+        # If egg info is available, we use it for both its PKG-INFO metadata
+        # and for its requires.txt for install_requires.
+        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
+        # the parsed setup.py, but use the install_requires info from the
+        # parsed setup.py.
 
+        setupscript = os.path.join(srctree, 'setup.py')
         try:
-            provides_output = self.run_command(['pythondeps', '-p'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
+        except Exception:
+            logger.exception("Failed to parse setup.py")
+            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
+
+        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
+        if egginfo:
+            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
+            requires_txt = os.path.join(egginfo[0], 'requires.txt')
+            if os.path.exists(requires_txt):
+                with codecs.open(requires_txt) as f:
+                    inst_req = []
+                    extras_req = collections.defaultdict(list)
+                    current_feature = None
+                    for line in f.readlines():
+                        line = line.rstrip()
+                        if not line:
+                            continue
+
+                        if line.startswith('['):
+                            # PACKAGECONFIG must not contain expressions or whitespace
+                            line = line.replace(" ", "")
+                            line = line.replace(':', "")
+                            line = line.replace('.', "-dot-")
+                            line = line.replace('"', "")
+                            line = line.replace('<', "-smaller-")
+                            line = line.replace('>', "-bigger-")
+                            line = line.replace('_', "-")
+                            line = line.replace('(', "")
+                            line = line.replace(')', "")
+                            line = line.replace('!', "-not-")
+                            line = line.replace('=', "-equals-")
+                            current_feature = line[1:-1]
+                        elif current_feature:
+                            extras_req[current_feature].append(line)
+                        else:
+                            inst_req.append(line)
+                    info['Install-requires'] = inst_req
+                    info['Extras-require'] = extras_req
+        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
+            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
+
+            if setup_info:
+                if 'Install-requires' in setup_info:
+                    info['Install-requires'] = setup_info['Install-requires']
+                if 'Extras-require' in setup_info:
+                    info['Extras-require'] = setup_info['Extras-require']
         else:
-            provides_lines = (l.rstrip() for l in provides_output.splitlines())
-            provides = set(l for l in provides_lines if l and l != 'setup')
-            deps -= provides
+            if setup_info:
+                info = setup_info
+            else:
+                info = self.get_setup_args_info(setupscript)
 
-        return deps
+        # Grab the license value before applying replacements
+        license_str = info.get('License', '').strip()
 
-    def parse_pkgdata_for_python_packages(self):
-        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+        self.apply_info_replacements(info)
 
-        ldata = tinfoil.config_data.createCopy()
-        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
-        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+        if uses_setuptools:
+            classes.append('setuptools3')
+        else:
+            classes.append('distutils3')
 
-        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
-        python_dirs = [python_sitedir + os.sep,
-                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
-                       os.path.dirname(python_sitedir) + os.sep]
-        packages = {}
-        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
-            files_info = None
-            with open(pkgdatafile, 'r') as f:
-                for line in f.readlines():
-                    field, value = line.split(': ', 1)
-                    if field.startswith('FILES_INFO'):
-                        files_info = ast.literal_eval(value)
-                        break
-                else:
-                    continue
+        if license_str:
+            for i, line in enumerate(lines_before):
+                if line.startswith('##LICENSE_PLACEHOLDER##'):
+                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
+                    break
 
-            for fn in files_info:
-                for suffix in importlib.machinery.all_suffixes():
-                    if fn.endswith(suffix):
-                        break
-                else:
-                    continue
+        if 'Classifier' in info:
+            license = self.handle_classifier_license(info['Classifier'], info.get('License', ''))
+            if license:
+                info['License'] = license
 
-                if fn.startswith(dynload_dir + os.sep):
-                    if '/.debug/' in fn:
-                        continue
-                    base = os.path.basename(fn)
-                    provided = base.split('.', 1)[0]
-                    packages[provided] = os.path.basename(pkgdatafile)
-                    continue
+        self.map_info_to_bbvar(info, extravalues)
 
-                for python_dir in python_dirs:
-                    if fn.startswith(python_dir):
-                        relpath = fn[len(python_dir):]
-                        relstart, _, relremaining = relpath.partition(os.sep)
-                        if relstart.endswith('.egg'):
-                            relpath = relremaining
-                        base, _ = os.path.splitext(relpath)
+        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
 
-                        if '/.debug/' in base:
-                            continue
-                        if os.path.basename(base) == '__init__':
-                            base = os.path.dirname(base)
-                        base = base.replace(os.sep + os.sep, os.sep)
-                        provided = base.replace(os.sep, '.')
-                        packages[provided] = os.path.basename(pkgdatafile)
-        return packages
+        extras_req = set()
+        if 'Extras-require' in info:
+            extras_req = info['Extras-require']
+            if extras_req:
+                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
+                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
+                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
+                lines_after.append('#')
+                lines_after.append('# Uncomment this line to enable all the optional features.')
+                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
+                for feature, feature_reqs in extras_req.items():
+                    unmapped_deps.difference_update(feature_reqs)
 
-    @classmethod
-    def run_command(cls, cmd, **popenargs):
-        if 'stderr' not in popenargs:
-            popenargs['stderr'] = subprocess.STDOUT
-        try:
-            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
-        except OSError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
-            raise
-        except subprocess.CalledProcessError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
-            raise
+                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
+                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
 
+        inst_reqs = set()
+        if 'Install-requires' in info:
+            if extras_req:
+                lines_after.append('')
+            inst_reqs = info['Install-requires']
+            if inst_reqs:
+                unmapped_deps.difference_update(inst_reqs)
+
+                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
+                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
+                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+
+        if mapped_deps:
+            name = info.get('Name')
+            if name and name[0] in mapped_deps:
+                # Attempt to avoid self-reference
+                mapped_deps.remove(name[0])
+            mapped_deps -= set(self.excluded_pkgdeps)
+            if inst_reqs or extras_req:
+                lines_after.append('')
+            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
+            lines_after.append('# python sources, and might not be 100% accurate.')
+            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+
+        unmapped_deps -= set(extensions)
+        unmapped_deps -= set(self.assume_provided)
+        if unmapped_deps:
+            if mapped_deps:
+                lines_after.append('')
+            lines_after.append('# WARNING: We were unable to map the following python package/module')
+            lines_after.append('# dependencies to the bitbake packages which include them:')
+            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+
+        handled.append('buildsystem')
 
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
@@ -748,4 +770,4 @@ def has_non_literals(value):
 
 def register_recipe_handlers(handlers):
     # We need to make sure this is ahead of the makefile fallback handler
-    handlers.append((PythonRecipeHandler(), 70))
+    handlers.append((PythonSetupPyRecipeHandler(), 70))
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189430): https://lists.openembedded.org/g/openembedded-core/message/189430
Mute This Topic: https://lists.openembedded.org/mt/102055998/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

Re: [OE-core] [PATCH v2 3/4] scripts:recipetool:create_buildsys_python: refactor code for futur PEP517 addition

Quoted 20 lines · Alexandre Belloni

Hello,

Quoted 6 lines · Julien Stephan

In order to prepare the support for pyproject.toml (PEP517 [1]) enabled projects, refactor the code and move setup.py specific code into a specific class in order to allow sharing the PythonRecipeHandler class

No functionnal changes expected

I tested with only the first 3 patches and unfortunately, thre were functional changes:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5886/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5935/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5952/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5936/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2296/steps/14/logs/stdio

Quoted 7 lines
2023-10-19 07:23:07,712 - oe-selftest - INFO - 1: 20/39 149/543 (20.20s) (0 failed) (recipetool.RecipetoolCreateTests.test_recipetool_create_github)
2023-10-19 07:23:07,712 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/recipetool.py", line 451, in test_recipetool_create_github
    self.assertTrue(os.path.isfile(recipefile))
  File "/usr/lib/python3.11/unittest/case.py", line 715, in assertTrue
    raise self.failureException(msg)

AssertionError: False is not true

Hi Alexandre,

I am sorry, I did run a full self test for devtool but I forgot to run it for recipetool..
I found the issue. This is not this commit, but the one where I now prepend "python3-" for all created recipes. I will update the self tests accordingly :)

Cheers
Julien

Quoted 842 lines
Quoted 828 lines

[1]: https://peps.python.org/pep-0517/#source-tree

---
 .../lib/recipetool/create_buildsys_python.py  | 748 +++++++++---------
 1 file changed, 385 insertions(+), 363 deletions(-)
scripts/lib/recipetool/create_buildsys_python.py +385 -364
index 502e1dfbc3d..69f6f5ca511 100644
@@ -37,63 +37,8 @@ class PythonRecipeHandler(RecipeHandler):
     assume_provided = ['builtins', 'os.path']
     # Assumes that the host python3 builtin_module_names is sane for target too
     assume_provided = assume_provided + list(sys.builtin_module_names)
+    excluded_fields = []
 
-    bbvar_map = {
-        'Name': 'PN',
-        'Version': 'PV',
-        'Home-page': 'HOMEPAGE',
-        'Summary': 'SUMMARY',
-        'Description': 'DESCRIPTION',
-        'License': 'LICENSE',
-        'Requires': 'RDEPENDS:${PN}',
-        'Provides': 'RPROVIDES:${PN}',
-        'Obsoletes': 'RREPLACES:${PN}',
-    }
-    # PN/PV are already set by recipetool core & desc can be extremely long
-    excluded_fields = [
-        'Description',
-    ]
-    setup_parse_map = {
-        'Url': 'Home-page',
-        'Classifiers': 'Classifier',
-        'Description': 'Summary',
-    }
-    setuparg_map = {
-        'Home-page': 'url',
-        'Classifier': 'classifiers',
-        'Summary': 'description',
-        'Description': 'long-description',
-    }
-    # Values which are lists, used by the setup.py argument based metadata
-    # extraction method, to determine how to process the setup.py output.
-    setuparg_list_fields = [
-        'Classifier',
-        'Requires',
-        'Provides',
-        'Obsoletes',
-        'Platform',
-        'Supported-Platform',
-    ]
-    setuparg_multi_line_values = ['Description']
-    replacements = [
-        ('License', r' +$', ''),
-        ('License', r'^ +', ''),
-        ('License', r' ', '-'),
-        ('License', r'^GNU-', ''),
-        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
-        ('License', r'^UNKNOWN$', ''),
-
-        # Remove currently unhandled version numbers from these variables
-        ('Requires', r' *\([^)]*\)', ''),
-        ('Provides', r' *\([^)]*\)', ''),
-        ('Obsoletes', r' *\([^)]*\)', ''),
-        ('Install-requires', r'^([^><= ]+).*', r'\1'),
-        ('Extras-require', r'^([^><= ]+).*', r'\1'),
-        ('Tests-require', r'^([^><= ]+).*', r'\1'),
-
-        # Remove unhandled dependency on particular features (e.g. foo[PDF])
-        ('Install-requires', r'\[[^\]]+\]$', ''),
-    ]
 
     classifier_license_map = {
         'License :: OSI Approved :: Academic Free License (AFL)': 'AFL',
@@ -166,122 +111,34 @@ class PythonRecipeHandler(RecipeHandler):
     def __init__(self):
         pass
 
-    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
-        if 'buildsystem' in handled:
-            return False
-
-        # Check for non-zero size setup.py files
-        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
-        for fn in setupfiles:
-            if os.path.getsize(fn):
-                break
-        else:
-            return False
-
-        # setup.py is always parsed to get at certain required information, such as
-        # distutils vs setuptools
-        #
-        # If egg info is available, we use it for both its PKG-INFO metadata
-        # and for its requires.txt for install_requires.
-        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
-        # the parsed setup.py, but use the install_requires info from the
-        # parsed setup.py.
-
-        setupscript = os.path.join(srctree, 'setup.py')
-        try:
-            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
-        except Exception:
-            logger.exception("Failed to parse setup.py")
-            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
-
-        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
-        if egginfo:
-            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
-            requires_txt = os.path.join(egginfo[0], 'requires.txt')
-            if os.path.exists(requires_txt):
-                with codecs.open(requires_txt) as f:
-                    inst_req = []
-                    extras_req = collections.defaultdict(list)
-                    current_feature = None
-                    for line in f.readlines():
-                        line = line.rstrip()
-                        if not line:
-                            continue
-
-                        if line.startswith('['):
-                            # PACKAGECONFIG must not contain expressions or whitespace
-                            line = line.replace(" ", "")
-                            line = line.replace(':', "")
-                            line = line.replace('.', "-dot-")
-                            line = line.replace('"', "")
-                            line = line.replace('<', "-smaller-")
-                            line = line.replace('>', "-bigger-")
-                            line = line.replace('_', "-")
-                            line = line.replace('(', "")
-                            line = line.replace(')', "")
-                            line = line.replace('!', "-not-")
-                            line = line.replace('=', "-equals-")
-                            current_feature = line[1:-1]
-                        elif current_feature:
-                            extras_req[current_feature].append(line)
-                        else:
-                            inst_req.append(line)
-                    info['Install-requires'] = inst_req
-                    info['Extras-require'] = extras_req
-        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
-            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
-
-            if setup_info:
-                if 'Install-requires' in setup_info:
-                    info['Install-requires'] = setup_info['Install-requires']
-                if 'Extras-require' in setup_info:
-                    info['Extras-require'] = setup_info['Extras-require']
-        else:
-            if setup_info:
-                info = setup_info
-            else:
-                info = self.get_setup_args_info(setupscript)
-
-        # Grab the license value before applying replacements
-        license_str = info.get('License', '').strip()
-
-        self.apply_info_replacements(info)
-
-        if uses_setuptools:
-            classes.append('setuptools3')
-        else:
-            classes.append('distutils3')
-
-        if license_str:
-            for i, line in enumerate(lines_before):
-                if line.startswith('##LICENSE_PLACEHOLDER##'):
-                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
-                    break
-
-        if 'Classifier' in info:
-            existing_licenses = info.get('License', '')
-            licenses = []
-            for classifier in info['Classifier']:
-                if classifier in self.classifier_license_map:
-                    license = self.classifier_license_map[classifier]
-                    if license == 'Apache' and 'Apache-2.0' in existing_licenses:
-                        license = 'Apache-2.0'
-                    elif license == 'GPL':
-                        if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
-                            license = 'GPL-2.0'
-                        elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
-                            license = 'GPL-3.0'
-                    elif license == 'LGPL':
-                        if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
-                            license = 'LGPL-2.1'
-                        elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
-                            license = 'LGPL-2.0'
-                        elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
-                            license = 'LGPL-3.0'
-                    licenses.append(license)
-
-            if licenses:
-                info['License'] = ' & '.join(licenses)
+    def handle_classifier_license(self, classifiers, existing_licenses=""):
+
+        licenses = []
+        for classifier in classifiers:
+            if classifier in self.classifier_license_map:
+                license = self.classifier_license_map[classifier]
+                if license == 'Apache' and 'Apache-2.0' in existing_licenses:
+                    license = 'Apache-2.0'
+                elif license == 'GPL':
+                    if 'GPL-2.0' in existing_licenses or 'GPLv2' in existing_licenses:
+                        license = 'GPL-2.0'
+                    elif 'GPL-3.0' in existing_licenses or 'GPLv3' in existing_licenses:
+                        license = 'GPL-3.0'
+                elif license == 'LGPL':
+                    if 'LGPL-2.1' in existing_licenses or 'LGPLv2.1' in existing_licenses:
+                        license = 'LGPL-2.1'
+                    elif 'LGPL-2.0' in existing_licenses or 'LGPLv2' in existing_licenses:
+                        license = 'LGPL-2.0'
+                    elif 'LGPL-3.0' in existing_licenses or 'LGPLv3' in existing_licenses:
+                        license = 'LGPL-3.0'
+                licenses.append(license)
+
+        if licenses:
+            return ' & '.join(licenses)
+
+        return None
+
+    def map_info_to_bbvar(self, info, extravalues):
 
         # Map PKG-INFO & setup.py fields to bitbake variables
         for field, values in info.items():
@@ -305,85 +162,220 @@ class PythonRecipeHandler(RecipeHandler):
             if bbvar not in extravalues and value:
                 extravalues[bbvar] = value
 
-        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
-
-        extras_req = set()
-        if 'Extras-require' in info:
-            extras_req = info['Extras-require']
-            if extras_req:
-                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
-                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
-                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
-                lines_after.append('#')
-                lines_after.append('# Uncomment this line to enable all the optional features.')
-                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
-                for feature, feature_reqs in extras_req.items():
-                    unmapped_deps.difference_update(feature_reqs)
-
-                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
-                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
-
-        inst_reqs = set()
-        if 'Install-requires' in info:
-            if extras_req:
-                lines_after.append('')
-            inst_reqs = info['Install-requires']
-            if inst_reqs:
-                unmapped_deps.difference_update(inst_reqs)
-
-                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
-                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
-                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
-                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+    def apply_info_replacements(self, info):
+        if not self.replacements:
+            return
 
-        if mapped_deps:
-            name = info.get('Name')
-            if name and name[0] in mapped_deps:
-                # Attempt to avoid self-reference
-                mapped_deps.remove(name[0])
-            mapped_deps -= set(self.excluded_pkgdeps)
-            if inst_reqs or extras_req:
-                lines_after.append('')
-            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
-            lines_after.append('# python sources, and might not be 100% accurate.')
-            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+        for variable, search, replace in self.replacements:
+            if variable not in info:
+                continue
 
-        unmapped_deps -= set(extensions)
-        unmapped_deps -= set(self.assume_provided)
-        if unmapped_deps:
-            if mapped_deps:
-                lines_after.append('')
-            lines_after.append('# WARNING: We were unable to map the following python package/module')
-            lines_after.append('# dependencies to the bitbake packages which include them:')
-            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+            def replace_value(search, replace, value):
+                if replace is None:
+                    if re.search(search, value):
+                        return None
+                else:
+                    new_value = re.sub(search, replace, value)
+                    if value != new_value:
+                        return new_value
+                return value
 
-        handled.append('buildsystem')
+            value = info[variable]
+            if isinstance(value, str):
+                new_value = replace_value(search, replace, value)
+                if new_value is None:
+                    del info[variable]
+                elif new_value != value:
+                    info[variable] = new_value
+            elif hasattr(value, 'items'):
+                for dkey, dvalue in list(value.items()):
+                    new_list = []
+                    for pos, a_value in enumerate(dvalue):
+                        new_value = replace_value(search, replace, a_value)
+                        if new_value is not None and new_value != value:
+                            new_list.append(new_value)
 
-    def get_pkginfo(self, pkginfo_fn):
-        msg = email.message_from_file(open(pkginfo_fn, 'r'))
-        msginfo = {}
-        for field in msg.keys():
-            values = msg.get_all(field)
-            if len(values) == 1:
-                msginfo[field] = values[0]
+                    if value != new_list:
+                        value[dkey] = new_list
             else:
-                msginfo[field] = values
-        return msginfo
+                new_list = []
+                for pos, a_value in enumerate(value):
+                    new_value = replace_value(search, replace, a_value)
+                    if new_value is not None and new_value != value:
+                        new_list.append(new_value)
 
-    def parse_setup_py(self, setupscript='./setup.py'):
-        with codecs.open(setupscript) as f:
-            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+                if value != new_list:
+                    info[variable] = new_list
 
-        def _map(key):
-            key = key.replace('_', '-')
-            key = key[0].upper() + key[1:]
-            if key in self.setup_parse_map:
-                key = self.setup_parse_map[key]
-            return key
 
-        # Naive mapping of setup() arguments to PKG-INFO field names
-        for d in [info, non_literals]:
+    def scan_python_dependencies(self, paths):
+        deps = set()
+        try:
+            dep_output = self.run_command(['pythondeps', '-d'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            for line in dep_output.splitlines():
+                line = line.rstrip()
+                dep, filename = line.split('\t', 1)
+                if filename.endswith('/setup.py'):
+                    continue
+                deps.add(dep)
+
+        try:
+            provides_output = self.run_command(['pythondeps', '-p'] + paths)
+        except (OSError, subprocess.CalledProcessError):
+            pass
+        else:
+            provides_lines = (l.rstrip() for l in provides_output.splitlines())
+            provides = set(l for l in provides_lines if l and l != 'setup')
+            deps -= provides
+
+        return deps
+
+    def parse_pkgdata_for_python_packages(self):
+        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+
+        ldata = tinfoil.config_data.createCopy()
+        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
+        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+
+        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
+        python_dirs = [python_sitedir + os.sep,
+                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
+                       os.path.dirname(python_sitedir) + os.sep]
+        packages = {}
+        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
+            files_info = None
+            with open(pkgdatafile, 'r') as f:
+                for line in f.readlines():
+                    field, value = line.split(': ', 1)
+                    if field.startswith('FILES_INFO'):
+                        files_info = ast.literal_eval(value)
+                        break
+                else:
+                    continue
+
+            for fn in files_info:
+                for suffix in importlib.machinery.all_suffixes():
+                    if fn.endswith(suffix):
+                        break
+                else:
+                    continue
+
+                if fn.startswith(dynload_dir + os.sep):
+                    if '/.debug/' in fn:
+                        continue
+                    base = os.path.basename(fn)
+                    provided = base.split('.', 1)[0]
+                    packages[provided] = os.path.basename(pkgdatafile)
+                    continue
+
+                for python_dir in python_dirs:
+                    if fn.startswith(python_dir):
+                        relpath = fn[len(python_dir):]
+                        relstart, _, relremaining = relpath.partition(os.sep)
+                        if relstart.endswith('.egg'):
+                            relpath = relremaining
+                        base, _ = os.path.splitext(relpath)
+
+                        if '/.debug/' in base:
+                            continue
+                        if os.path.basename(base) == '__init__':
+                            base = os.path.dirname(base)
+                        base = base.replace(os.sep + os.sep, os.sep)
+                        provided = base.replace(os.sep, '.')
+                        packages[provided] = os.path.basename(pkgdatafile)
+        return packages
+
+    @classmethod
+    def run_command(cls, cmd, **popenargs):
+        if 'stderr' not in popenargs:
+            popenargs['stderr'] = subprocess.STDOUT
+        try:
+            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
+        except OSError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
+            raise
+        except subprocess.CalledProcessError as exc:
+            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
+            raise
+
+class PythonSetupPyRecipeHandler(PythonRecipeHandler):
+    bbvar_map = {
+        'Name': 'PN',
+        'Version': 'PV',
+        'Home-page': 'HOMEPAGE',
+        'Summary': 'SUMMARY',
+        'Description': 'DESCRIPTION',
+        'License': 'LICENSE',
+        'Requires': 'RDEPENDS:${PN}',
+        'Provides': 'RPROVIDES:${PN}',
+        'Obsoletes': 'RREPLACES:${PN}',
+    }
+    # PN/PV are already set by recipetool core & desc can be extremely long
+    excluded_fields = [
+        'Description',
+    ]
+    setup_parse_map = {
+        'Url': 'Home-page',
+        'Classifiers': 'Classifier',
+        'Description': 'Summary',
+    }
+    setuparg_map = {
+        'Home-page': 'url',
+        'Classifier': 'classifiers',
+        'Summary': 'description',
+        'Description': 'long-description',
+    }
+    # Values which are lists, used by the setup.py argument based metadata
+    # extraction method, to determine how to process the setup.py output.
+    setuparg_list_fields = [
+        'Classifier',
+        'Requires',
+        'Provides',
+        'Obsoletes',
+        'Platform',
+        'Supported-Platform',
+    ]
+    setuparg_multi_line_values = ['Description']
+
+    replacements = [
+        ('License', r' +$', ''),
+        ('License', r'^ +', ''),
+        ('License', r' ', '-'),
+        ('License', r'^GNU-', ''),
+        ('License', r'-[Ll]icen[cs]e(,?-[Vv]ersion)?', ''),
+        ('License', r'^UNKNOWN$', ''),
+
+        # Remove currently unhandled version numbers from these variables
+        ('Requires', r' *\([^)]*\)', ''),
+        ('Provides', r' *\([^)]*\)', ''),
+        ('Obsoletes', r' *\([^)]*\)', ''),
+        ('Install-requires', r'^([^><= ]+).*', r'\1'),
+        ('Extras-require', r'^([^><= ]+).*', r'\1'),
+        ('Tests-require', r'^([^><= ]+).*', r'\1'),
+
+        # Remove unhandled dependency on particular features (e.g. foo[PDF])
+        ('Install-requires', r'\[[^\]]+\]$', ''),
+    ]
+
+    def __init__(self):
+        pass
+
+    def parse_setup_py(self, setupscript='./setup.py'):
+        with codecs.open(setupscript) as f:
+            info, imported_modules, non_literals, extensions = gather_setup_info(f)
+
+        def _map(key):
+            key = key.replace('_', '-')
+            key = key[0].upper() + key[1:]
+            if key in self.setup_parse_map:
+                key = self.setup_parse_map[key]
+            return key
+
+        # Naive mapping of setup() arguments to PKG-INFO field names
+        for d in [info, non_literals]:
             for key, value in list(d.items()):
                 if key is None:
                     continue
@@ -445,47 +437,16 @@ class PythonRecipeHandler(RecipeHandler):
                 info[fields[lineno]] = line
         return info
 
-    def apply_info_replacements(self, info):
-        for variable, search, replace in self.replacements:
-            if variable not in info:
-                continue
-
-            def replace_value(search, replace, value):
-                if replace is None:
-                    if re.search(search, value):
-                        return None
-                else:
-                    new_value = re.sub(search, replace, value)
-                    if value != new_value:
-                        return new_value
-                return value
-
-            value = info[variable]
-            if isinstance(value, str):
-                new_value = replace_value(search, replace, value)
-                if new_value is None:
-                    del info[variable]
-                elif new_value != value:
-                    info[variable] = new_value
-            elif hasattr(value, 'items'):
-                for dkey, dvalue in list(value.items()):
-                    new_list = []
-                    for pos, a_value in enumerate(dvalue):
-                        new_value = replace_value(search, replace, a_value)
-                        if new_value is not None and new_value != value:
-                            new_list.append(new_value)
-
-                    if value != new_list:
-                        value[dkey] = new_list
+    def get_pkginfo(self, pkginfo_fn):
+        msg = email.message_from_file(open(pkginfo_fn, 'r'))
+        msginfo = {}
+        for field in msg.keys():
+            values = msg.get_all(field)
+            if len(values) == 1:
+                msginfo[field] = values[0]
             else:
-                new_list = []
-                for pos, a_value in enumerate(value):
-                    new_value = replace_value(search, replace, a_value)
-                    if new_value is not None and new_value != value:
-                        new_list.append(new_value)
-
-                if value != new_list:
-                    info[variable] = new_list
+                msginfo[field] = values
+        return msginfo
 
     def scan_setup_python_deps(self, srctree, setup_info, setup_non_literals):
         if 'Package-dir' in setup_info:
@@ -540,99 +501,160 @@ class PythonRecipeHandler(RecipeHandler):
                 unmapped_deps.add(dep)
         return mapped_deps, unmapped_deps
 
-    def scan_python_dependencies(self, paths):
-        deps = set()
-        try:
-            dep_output = self.run_command(['pythondeps', '-d'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ['setup.py'])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
         else:
-            for line in dep_output.splitlines():
-                line = line.rstrip()
-                dep, filename = line.split('\t', 1)
-                if filename.endswith('/setup.py'):
-                    continue
-                deps.add(dep)
+            return False
+
+        # setup.py is always parsed to get at certain required information, such as
+        # distutils vs setuptools
+        #
+        # If egg info is available, we use it for both its PKG-INFO metadata
+        # and for its requires.txt for install_requires.
+        # If PKG-INFO is available but no egg info is, we use that for metadata in preference to
+        # the parsed setup.py, but use the install_requires info from the
+        # parsed setup.py.
 
+        setupscript = os.path.join(srctree, 'setup.py')
         try:
-            provides_output = self.run_command(['pythondeps', '-p'] + paths)
-        except (OSError, subprocess.CalledProcessError):
-            pass
+            setup_info, uses_setuptools, setup_non_literals, extensions = self.parse_setup_py(setupscript)
+        except Exception:
+            logger.exception("Failed to parse setup.py")
+            setup_info, uses_setuptools, setup_non_literals, extensions = {}, True, [], []
+
+        egginfo = glob.glob(os.path.join(srctree, '*.egg-info'))
+        if egginfo:
+            info = self.get_pkginfo(os.path.join(egginfo[0], 'PKG-INFO'))
+            requires_txt = os.path.join(egginfo[0], 'requires.txt')
+            if os.path.exists(requires_txt):
+                with codecs.open(requires_txt) as f:
+                    inst_req = []
+                    extras_req = collections.defaultdict(list)
+                    current_feature = None
+                    for line in f.readlines():
+                        line = line.rstrip()
+                        if not line:
+                            continue
+
+                        if line.startswith('['):
+                            # PACKAGECONFIG must not contain expressions or whitespace
+                            line = line.replace(" ", "")
+                            line = line.replace(':', "")
+                            line = line.replace('.', "-dot-")
+                            line = line.replace('"', "")
+                            line = line.replace('<', "-smaller-")
+                            line = line.replace('>', "-bigger-")
+                            line = line.replace('_', "-")
+                            line = line.replace('(', "")
+                            line = line.replace(')', "")
+                            line = line.replace('!', "-not-")
+                            line = line.replace('=', "-equals-")
+                            current_feature = line[1:-1]
+                        elif current_feature:
+                            extras_req[current_feature].append(line)
+                        else:
+                            inst_req.append(line)
+                    info['Install-requires'] = inst_req
+                    info['Extras-require'] = extras_req
+        elif RecipeHandler.checkfiles(srctree, ['PKG-INFO']):
+            info = self.get_pkginfo(os.path.join(srctree, 'PKG-INFO'))
+
+            if setup_info:
+                if 'Install-requires' in setup_info:
+                    info['Install-requires'] = setup_info['Install-requires']
+                if 'Extras-require' in setup_info:
+                    info['Extras-require'] = setup_info['Extras-require']
         else:
-            provides_lines = (l.rstrip() for l in provides_output.splitlines())
-            provides = set(l for l in provides_lines if l and l != 'setup')
-            deps -= provides
+            if setup_info:
+                info = setup_info
+            else:
+                info = self.get_setup_args_info(setupscript)
 
-        return deps
+        # Grab the license value before applying replacements
+        license_str = info.get('License', '').strip()
 
-    def parse_pkgdata_for_python_packages(self):
-        pkgdata_dir = tinfoil.config_data.getVar('PKGDATA_DIR')
+        self.apply_info_replacements(info)
 
-        ldata = tinfoil.config_data.createCopy()
-        bb.parse.handle('classes-recipe/python3-dir.bbclass', ldata, True)
-        python_sitedir = ldata.getVar('PYTHON_SITEPACKAGES_DIR')
+        if uses_setuptools:
+            classes.append('setuptools3')
+        else:
+            classes.append('distutils3')
 
-        dynload_dir = os.path.join(os.path.dirname(python_sitedir), 'lib-dynload')
-        python_dirs = [python_sitedir + os.sep,
-                       os.path.join(os.path.dirname(python_sitedir), 'dist-packages') + os.sep,
-                       os.path.dirname(python_sitedir) + os.sep]
-        packages = {}
-        for pkgdatafile in glob.glob('{}/runtime/*'.format(pkgdata_dir)):
-            files_info = None
-            with open(pkgdatafile, 'r') as f:
-                for line in f.readlines():
-                    field, value = line.split(': ', 1)
-                    if field.startswith('FILES_INFO'):
-                        files_info = ast.literal_eval(value)
-                        break
-                else:
-                    continue
+        if license_str:
+            for i, line in enumerate(lines_before):
+                if line.startswith('##LICENSE_PLACEHOLDER##'):
+                    lines_before.insert(i, '# NOTE: License in setup.py/PKGINFO is: %s' % license_str)
+                    break
 
-            for fn in files_info:
-                for suffix in importlib.machinery.all_suffixes():
-                    if fn.endswith(suffix):
-                        break
-                else:
-                    continue
+        if 'Classifier' in info:
+            license = self.handle_classifier_license(info['Classifier'], info.get('License', ''))
+            if license:
+                info['License'] = license
 
-                if fn.startswith(dynload_dir + os.sep):
-                    if '/.debug/' in fn:
-                        continue
-                    base = os.path.basename(fn)
-                    provided = base.split('.', 1)[0]
-                    packages[provided] = os.path.basename(pkgdatafile)
-                    continue
+        self.map_info_to_bbvar(info, extravalues)
 
-                for python_dir in python_dirs:
-                    if fn.startswith(python_dir):
-                        relpath = fn[len(python_dir):]
-                        relstart, _, relremaining = relpath.partition(os.sep)
-                        if relstart.endswith('.egg'):
-                            relpath = relremaining
-                        base, _ = os.path.splitext(relpath)
+        mapped_deps, unmapped_deps = self.scan_setup_python_deps(srctree, setup_info, setup_non_literals)
 
-                        if '/.debug/' in base:
-                            continue
-                        if os.path.basename(base) == '__init__':
-                            base = os.path.dirname(base)
-                        base = base.replace(os.sep + os.sep, os.sep)
-                        provided = base.replace(os.sep, '.')
-                        packages[provided] = os.path.basename(pkgdatafile)
-        return packages
+        extras_req = set()
+        if 'Extras-require' in info:
+            extras_req = info['Extras-require']
+            if extras_req:
+                lines_after.append('# The following configs & dependencies are from setuptools extras_require.')
+                lines_after.append('# These dependencies are optional, hence can be controlled via PACKAGECONFIG.')
+                lines_after.append('# The upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('# The configs are might not correct, since PACKAGECONFIG does not support expressions as may used in requires.txt - they are just replaced by text.')
+                lines_after.append('#')
+                lines_after.append('# Uncomment this line to enable all the optional features.')
+                lines_after.append('#PACKAGECONFIG ?= "{}"'.format(' '.join(k.lower() for k in extras_req)))
+                for feature, feature_reqs in extras_req.items():
+                    unmapped_deps.difference_update(feature_reqs)
 
-    @classmethod
-    def run_command(cls, cmd, **popenargs):
-        if 'stderr' not in popenargs:
-            popenargs['stderr'] = subprocess.STDOUT
-        try:
-            return subprocess.check_output(cmd, **popenargs).decode('utf-8')
-        except OSError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc)
-            raise
-        except subprocess.CalledProcessError as exc:
-            logger.error('Unable to run `{}`: {}', ' '.join(cmd), exc.output)
-            raise
+                    feature_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(feature_reqs))
+                    lines_after.append('PACKAGECONFIG[{}] = ",,,{}"'.format(feature.lower(), ' '.join(feature_req_deps)))
 
+        inst_reqs = set()
+        if 'Install-requires' in info:
+            if extras_req:
+                lines_after.append('')
+            inst_reqs = info['Install-requires']
+            if inst_reqs:
+                unmapped_deps.difference_update(inst_reqs)
+
+                inst_req_deps = ('python3-' + r.replace('.', '-').lower() for r in sorted(inst_reqs))
+                lines_after.append('# WARNING: the following rdepends are from setuptools install_requires. These')
+                lines_after.append('# upstream names may not correspond exactly to bitbake package names.')
+                lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(inst_req_deps)))
+
+        if mapped_deps:
+            name = info.get('Name')
+            if name and name[0] in mapped_deps:
+                # Attempt to avoid self-reference
+                mapped_deps.remove(name[0])
+            mapped_deps -= set(self.excluded_pkgdeps)
+            if inst_reqs or extras_req:
+                lines_after.append('')
+            lines_after.append('# WARNING: the following rdepends are determined through basic analysis of the')
+            lines_after.append('# python sources, and might not be 100% accurate.')
+            lines_after.append('RDEPENDS:${{PN}} += "{}"'.format(' '.join(sorted(mapped_deps))))
+
+        unmapped_deps -= set(extensions)
+        unmapped_deps -= set(self.assume_provided)
+        if unmapped_deps:
+            if mapped_deps:
+                lines_after.append('')
+            lines_after.append('# WARNING: We were unable to map the following python package/module')
+            lines_after.append('# dependencies to the bitbake packages which include them:')
+            lines_after.extend('#    {}'.format(d) for d in sorted(unmapped_deps))
+
+        handled.append('buildsystem')
 
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
@@ -748,4 +770,4 @@ def has_non_literals(value):
 
 def register_recipe_handlers(handlers):
     # We need to make sure this is ahead of the makefile fallback handler
-    handlers.append((PythonRecipeHandler(), 70))
+    handlers.append((PythonSetupPyRecipeHandler(), 70))
--

2.42.0

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189430): https://lists.openembedded.org/g/openembedded-core/message/189430
Mute This Topic: https://lists.openembedded.org/mt/102055998/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#5 lore raw

PATCH v2 4/4 scripts:recipetool:create_buildsys_python: add PEP517 support

In reply to Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create_buildsys_python.py +233 -1
index 69f6f5ca511..0b601d50a4b 100644
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml
 from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler):
 
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these variables
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the corresponding bitbake package
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py"
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is: %s" % license_str
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value = bb.utils.md5_file(os.path.join(srctree, licfile))
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values in the right place in the recipe file
+                        try:
+                            pos = lines_before.index("##LICENSE_PLACEHOLDER##")
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile, md5value]))
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require, mapped))
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -= set(self.excluded_native_pkgdeps)
+                    if native_mapped_deps != set():
+                        info["requires"] = " ".join(sorted(native_mapped_deps))
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(native_unmapped_deps)
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency, mapped))
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = " ".join(sorted(mapped_deps))
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(unmapped_deps)
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback handler
+    # We need to make sure these are ahead of the makefile fallback handler
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
Signature · 1 line
2.42.0
#6 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Hello,

Quoted 19 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create_buildsys_python.py +0 -0
index 69f6f5ca511..0b601d50a4b 100644

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 4 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Quoted 261 lines
scripts/lib/recipetool/create_buildsys_python.py +232 -1
 from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler):
 
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these variables
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the corresponding bitbake package
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py"
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is: %s" % license_str
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value = bb.utils.md5_file(os.path.join(srctree, licfile))
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values in the right place in the recipe file
+                        try:
+                            pos = lines_before.index("##LICENSE_PLACEHOLDER##")
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile, md5value]))
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require, mapped))
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -= set(self.excluded_native_pkgdeps)
+                    if native_mapped_deps != set():
+                        info["requires"] = " ".join(sorted(native_mapped_deps))
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(native_unmapped_deps)
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency, mapped))
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = " ".join(sorted(mapped_deps))
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(unmapped_deps)
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback handler
+    # We need to make sure these are ahead of the makefile fallback handler
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
Signature · 11 lines
2.42.0



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189431): https://lists.openembedded.org/g/openembedded-core/message/189431
Mute This Topic: https://lists.openembedded.org/mt/102055999/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Quoted 19 lines · Alexandre Belloni via lists.openembedded.org

Hello,

Quoted 7 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

Review

Quoted 8 lines

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/scripts/lib/recipetool/create_buildsys_python.py

Quoted 13 lines

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 8 lines

index 69f6f5ca511..0b601d50a4b 100644

scripts/lib/recipetool/create_buildsys_python.py +1 -0
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Starting in Python 3.11, we have tomllib
https://docs.python.org/3/library/tomllib.html

However, we currently pin the minimum Python version at 3.8:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/__init__.py#n15

Quoted 298 lines

from recipetool.create import RecipeHandler

scripts/lib/recipetool/create_buildsys_python.py class

Quoted 3 lines
scripts/lib/recipetool/create_buildsys_python.py +0 -0
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class

PythonSetupPyRecipeHandler(PythonRecipeHandler):

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 11 lines
scripts/lib/recipetool/create_buildsys_python.py +8 -0
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621:

https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these

variables

Quoted 13 lines
scripts/lib/recipetool/create_buildsys_python.py +13 -0
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the

corresponding bitbake package

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 12 lines
scripts/lib/recipetool/create_buildsys_python.py +12 -0
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after,

handled, extravalues):

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 7 lines
scripts/lib/recipetool/create_buildsys_python.py +7 -0
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree,

["pyproject.toml"])

Quoted 6 lines
scripts/lib/recipetool/create_buildsys_python.py +6 -0
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use

pyproject.toml. Will try to use legacy setup.py"

Quoted 18 lines
scripts/lib/recipetool/create_buildsys_python.py +18 -0
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is:

%s" % license_str

Quoted 5 lines
scripts/lib/recipetool/create_buildsys_python.py +5 -0
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value =

bb.utils.md5_file(os.path.join(srctree, licfile))

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 8 lines
scripts/lib/recipetool/create_buildsys_python.py +8 -0
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values

in the right place in the recipe file

scripts/lib/recipetool/create_buildsys_python.py class · line 797

Quoted 2 lines
scripts/lib/recipetool/create_buildsys_python.py +2 -0
+                        try:
+                            pos =

lines_before.index("##LICENSE_PLACEHOLDER##")

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 8 lines
scripts/lib/recipetool/create_buildsys_python.py +8 -0
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile,

md5value]))

Quoted 4 lines
scripts/lib/recipetool/create_buildsys_python.py +4 -0
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require,

mapped))

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 12 lines
scripts/lib/recipetool/create_buildsys_python.py +12 -0
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -=

set(self.excluded_native_pkgdeps)

scripts/lib/recipetool/create_buildsys_python.py class · line 833

Quoted 2 lines
scripts/lib/recipetool/create_buildsys_python.py +2 -0
+                    if native_mapped_deps != set():
+                        info["requires"] = "

".join(sorted(native_mapped_deps))

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 5 lines
scripts/lib/recipetool/create_buildsys_python.py +5 -0
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following

python package/module"

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 3 lines
scripts/lib/recipetool/create_buildsys_python.py +3 -0
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which

include them:"

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 3 lines
scripts/lib/recipetool/create_buildsys_python.py +3 -0
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in

sorted(native_unmapped_deps)

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 7 lines
scripts/lib/recipetool/create_buildsys_python.py +7 -0
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency,

mapped))

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +10 -0
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = "

".join(sorted(mapped_deps))

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 5 lines
scripts/lib/recipetool/create_buildsys_python.py +5 -0
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following

python package/module"

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 3 lines
scripts/lib/recipetool/create_buildsys_python.py +3 -0
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages

which include them:"

scripts/lib/recipetool/create_buildsys_python.py class PythonSetupPyRecipeHandler(PythonRecipeHandler):

Quoted 3 lines
scripts/lib/recipetool/create_buildsys_python.py +3 -0
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in

sorted(unmapped_deps)

Quoted 19 lines
scripts/lib/recipetool/create_buildsys_python.py +15 -0
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()

scripts/lib/recipetool/create_buildsys_python.py def has_non_literals(value): · line 772

Quoted 4 lines
scripts/lib/recipetool/create_buildsys_python.py +0 -1
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback

handler

scripts/lib/recipetool/create_buildsys_python.py def has_non_literals(value): · line 1002

Quoted 1 line
scripts/lib/recipetool/create_buildsys_python.py +1 -0
+    # We need to make sure these are ahead of the makefile fallback

handler

Quoted 10 lines
scripts/lib/recipetool/create_buildsys_python.py +2 -1
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
--

2.42.0

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#189462):
https://lists.openembedded.org/g/openembedded-core/message/189462
Mute This Topic: https://lists.openembedded.org/mt/102055999/924729
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#8 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

In reply to Alexandre Belloni
Quoted 18 lines · Alexandre Belloni

Hello,

Quoted 14 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
Quoted 12 lines

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 9 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
index 69f6f5ca511..0b601d50a4b 100644
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Hello,

Sure I 'll do it. Just to confirm, I should add it here:
https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-packages-for-the-build-host
?

Cheers
Julien

Quoted 267 lines
Quoted 253 lines
scripts/lib/recipetool/create_buildsys_python.py +232 -2
 from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler):
 
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these variables
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the corresponding bitbake package
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py"
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is: %s" % license_str
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value = bb.utils.md5_file(os.path.join(srctree, licfile))
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values in the right place in the recipe file
+                        try:
+                            pos = lines_before.index("##LICENSE_PLACEHOLDER##")
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile, md5value]))
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require, mapped))
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -= set(self.excluded_native_pkgdeps)
+                    if native_mapped_deps != set():
+                        info["requires"] = " ".join(sorted(native_mapped_deps))
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(native_unmapped_deps)
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency, mapped))
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = " ".join(sorted(mapped_deps))
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(unmapped_deps)
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback handler
+    # We need to make sure these are ahead of the makefile fallback handler
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
--

2.42.0

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189431): https://lists.openembedded.org/g/openembedded-core/message/189431
Mute This Topic: https://lists.openembedded.org/mt/102055999/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#9 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Quoted 33 lines · Julien Stephan

Le jeu. 19 oct. 2023 � 15:49, Alexandre Belloni
<[email protected]> a �crit :

Quoted 18 lines

Hello,

Quoted 14 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
Quoted 12 lines

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 9 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
index 69f6f5ca511..0b601d50a4b 100644
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Quoted 5 lines

Hello,

Sure I 'll do it. Just to confirm, I should add it here:
https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-packages-for-the-build-host
?

I guess the preferred way would be to depend on python3-toml-native instead of requiring installation on the host.

Quoted 271 lines

Cheers
Julien

Quoted 267 lines
Quoted 253 lines
scripts/lib/recipetool/create_buildsys_python.py +232 -2
 from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler):
 
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these variables
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the corresponding bitbake package
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py"
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is: %s" % license_str
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value = bb.utils.md5_file(os.path.join(srctree, licfile))
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values in the right place in the recipe file
+                        try:
+                            pos = lines_before.index("##LICENSE_PLACEHOLDER##")
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile, md5value]))
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require, mapped))
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -= set(self.excluded_native_pkgdeps)
+                    if native_mapped_deps != set():
+                        info["requires"] = " ".join(sorted(native_mapped_deps))
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(native_unmapped_deps)
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency, mapped))
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = " ".join(sorted(mapped_deps))
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(unmapped_deps)
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback handler
+    # We need to make sure these are ahead of the makefile fallback handler
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
--

2.42.0

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189431): https://lists.openembedded.org/g/openembedded-core/message/189431
Mute This Topic: https://lists.openembedded.org/mt/102055999/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#10 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Quoted 35 lines · Alexandre Belloni
Quoted 33 lines · Julien Stephan
Quoted 30 lines · Alexandre Belloni

Hello,

Quoted 19 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
scripts/lib/recipetool/create_buildsys_python.py +0 -0
index 69f6f5ca511..0b601d50a4b 100644

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 4 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Quoted 9 lines
Quoted 5 lines

Hello,

Sure I 'll do it. Just to confirm, I should add it here:
https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-packages-for-the-build-host
?

I guess the preferred way would be to depend on python3-toml-native instead of requiring installation on the host.

Hi Alexandre,

How am I supposed to do that for a script? Is that even possible? Am I missing something obvious?

Cheers
Julien

Quoted 276 lines
Quoted 271 lines

Cheers
Julien

Quoted 267 lines
Quoted 253 lines
scripts/lib/recipetool/create_buildsys_python.py +232 -2
 from recipetool.create import RecipeHandler
 
 logger = logging.getLogger('recipetool')
@@ -656,6 +657,235 @@ class PythonSetupPyRecipeHandler(PythonRecipeHandler):
 
         handled.append('buildsystem')
 
+class PythonPyprojectTomlRecipeHandler(PythonRecipeHandler):
+    """Base class to support PEP517 and PEP518
+
+    PEP517 https://peps.python.org/pep-0517/#source-trees
+    PEP518 https://peps.python.org/pep-0518/#build-system-table
+    """
+
+    # PEP621: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/
+    # add only the ones that map to a BB var
+    # potentially missing: optional-dependencies
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "Homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+        "dependencies": "RDEPENDS:${PN}",
+        "requires": "DEPENDS",
+    }
+
+    replacements = [
+        ("license", r" +$", ""),
+        ("license", r"^ +", ""),
+        ("license", r" ", "-"),
+        ("license", r"^GNU-", ""),
+        ("license", r"-[Ll]icen[cs]e(,?-[Vv]ersion)?", ""),
+        ("license", r"^UNKNOWN$", ""),
+        # Remove currently unhandled version numbers from these variables
+        ("requires", r"\[[^\]]+\]$", ""),
+        ("requires", r"^([^><= ]+).*", r"\1"),
+        ("dependencies", r"\[[^\]]+\]$", ""),
+        ("dependencies", r"^([^><= ]+).*", r"\1"),
+    ]
+
+    build_backend_map = {
+        "setuptools.build_meta": "python_setuptools_build_meta",
+        "poetry.core.masonry.api": "python_poetry_core",
+        "flit_core.buildapi": "python_flit_core",
+    }
+
+    excluded_native_pkgdeps = [
+        # already provided by python_setuptools_build_meta.bbclass
+        "python3-setuptools-native",
+        "python3-wheel-native",
+        # already provided by python_poetry_core.bbclass
+        "python3-poetry-core-native",
+        # already provided by python_flit_core.bbclass
+        "python3-flit-core-native",
+    ]
+
+    # add here a list of known and often used packages and the corresponding bitbake package
+    known_deps_map = {
+        "setuptools": "python3-setuptools",
+        "wheel": "python3-wheel",
+        "poetry-core": "python3-poetry-core",
+        "flit_core": "python3-flit-core",
+        "setuptools-scm": "python3-setuptools-scm",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(self, srctree, classes, lines_before, lines_after, handled, extravalues):
+        info = {}
+
+        if 'buildsystem' in handled:
+            return False
+
+        # Check for non-zero size setup.py files
+        setupfiles = RecipeHandler.checkfiles(srctree, ["pyproject.toml"])
+        for fn in setupfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        setupscript = os.path.join(srctree, "pyproject.toml")
+
+        try:
+            config = self.parse_pyproject_toml(setupscript)
+            build_backend = config["build-system"]["build-backend"]
+            if build_backend in self.build_backend_map:
+                classes.append(self.build_backend_map[build_backend])
+            else:
+                logger.error(
+                    "Unsupported build-backend: %s, cannot use pyproject.toml. Will try to use legacy setup.py"
+                    % build_backend
+                )
+                return False
+
+            licfile = ""
+            if "project" in config:
+                for field, values in config["project"].items():
+                    if field == "license":
+                        value = values.get("text", "")
+                        if not value:
+                            licfile = values.get("file", "")
+                    elif isinstance(values, dict):
+                        for k, v in values.items():
+                            info[k] = v
+                        continue
+                    else:
+                        value = values
+
+                    info[field] = value
+
+            # Grab the license value before applying replacements
+            license_str = info.get("license", "").strip()
+
+            if license_str:
+                for i, line in enumerate(lines_before):
+                    if line.startswith("##LICENSE_PLACEHOLDER##"):
+                        lines_before.insert(
+                            i, "# NOTE: License in pyproject.toml is: %s" % license_str
+                        )
+                        break
+
+            info["requires"] = config["build-system"]["requires"]
+
+            self.apply_info_replacements(info)
+
+            if "classifiers" in info:
+                license = self.handle_classifier_license(
+                    info["classifiers"], info.get("license", "")
+                )
+                if license:
+                    if licfile:
+                        lines = []
+                        md5value = bb.utils.md5_file(os.path.join(srctree, licfile))
+                        lines.append('LICENSE = "%s"' % license)
+                        lines.append(
+                            'LIC_FILES_CHKSUM = "file://%s;md5=%s"'
+                            % (licfile, md5value)
+                        )
+                        lines.append("")
+
+                        # Replace the placeholder so we get the values in the right place in the recipe file
+                        try:
+                            pos = lines_before.index("##LICENSE_PLACEHOLDER##")
+                        except ValueError:
+                            pos = -1
+                        if pos == -1:
+                            lines_before.extend(lines)
+                        else:
+                            lines_before[pos : pos + 1] = lines
+
+                        handled.append(("license", [license, licfile, md5value]))
+                    else:
+                        info["license"] = license
+
+            provided_packages = self.parse_pkgdata_for_python_packages()
+            provided_packages.update(self.known_deps_map)
+            native_mapped_deps, native_unmapped_deps = set(), set()
+            mapped_deps, unmapped_deps = set(), set()
+
+            if "requires" in info:
+                for require in info["requires"]:
+                    mapped = provided_packages.get(require)
+
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (require, mapped))
+                        native_mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % require)
+                        native_unmapped_deps.add(require)
+
+                info.pop("requires")
+
+                if native_mapped_deps != set():
+                    native_mapped_deps = {
+                        item + "-native" for item in native_mapped_deps
+                    }
+                    native_mapped_deps -= set(self.excluded_native_pkgdeps)
+                    if native_mapped_deps != set():
+                        info["requires"] = " ".join(sorted(native_mapped_deps))
+
+                if native_unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(native_unmapped_deps)
+                    )
+
+            if "dependencies" in info:
+                for dependency in info["dependencies"]:
+                    mapped = provided_packages.get(dependency)
+                    if mapped:
+                        logger.error("Mapped %s to %s" % (dependency, mapped))
+                        mapped_deps.add(mapped)
+                    else:
+                        logger.error("Could not map %s" % dependency)
+                        unmapped_deps.add(dependency)
+
+                info.pop("dependencies")
+
+                if mapped_deps != set():
+                    if mapped_deps != set():
+                        info["dependencies"] = " ".join(sorted(mapped_deps))
+
+                if unmapped_deps:
+                    lines_after.append("")
+                    lines_after.append(
+                        "# WARNING: We were unable to map the following python package/module"
+                    )
+                    lines_after.append(
+                        "# runtime dependencies to the bitbake packages which include them:"
+                    )
+                    lines_after.extend(
+                        "#    {}".format(d) for d in sorted(unmapped_deps)
+                    )
+
+            self.map_info_to_bbvar(info, extravalues)
+
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception("Failed to parse pyproject.toml")
+            return False
+
+    def parse_pyproject_toml(self, setupscript):
+        with open(setupscript, "r") as f:
+            config = toml.load(f)
+        return config
+
+
 def gather_setup_info(fileobj):
     parsed = ast.parse(fileobj.read(), fileobj.name)
     visitor = SetupScriptVisitor()
@@ -769,5 +999,7 @@ def has_non_literals(value):
 
 
 def register_recipe_handlers(handlers):
-    # We need to make sure this is ahead of the makefile fallback handler
+    # We need to make sure these are ahead of the makefile fallback handler
+    # and the pyproject.toml handler ahead of the setup.py handler
+    handlers.append((PythonPyprojectTomlRecipeHandler(), 75))
     handlers.append((PythonSetupPyRecipeHandler(), 70))
--

2.42.0

Quoted 8 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#189431): https://lists.openembedded.org/g/openembedded-core/message/189431
Mute This Topic: https://lists.openembedded.org/mt/102055999/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#11 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Quoted 43 lines · Julien Stephan
Quoted 41 lines · Alexandre Belloni
Quoted 38 lines · Julien Stephan
Quoted 18 lines · Alexandre Belloni

Hello,

Quoted 14 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
Quoted 12 lines

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 9 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
index 69f6f5ca511..0b601d50a4b 100644
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Hello,

Sure I 'll do it. Just to confirm, I should add it here:
https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-packages-for-the-build-host
?

Quoted 8 lines
Quoted 3 lines

I guess the preferred way would be to depend on python3-toml-native instead of requiring installation on the host.

Hi Alexandre,

How am I supposed to do that for a script? Is that even possible? Am I missing something obvious?

As far as I know you're not missing anything obvious. If the toml dependency was in the target recipe this would be easier but needing this from recipetool is harder as it is running under the host python.

Adding dependencies for the host is hard as it needs to be added on the docs, on the autobuilder workers and into our buildtools-extended- tarballs and generally impacts a lot of people/places.

I think moving the import into the code and having recipe tool error and ask the user to install the dependency might be the best solution for now. That does give us a challenge over where/when we can test the code though.

On the autobuilder we could run recipetool for this test using the python3native from a recipe sysroot where the toml dependency is available I guess but that is a bit ugly.

As Tim mentions, with python 3.11 onwards, this problem does go away. We could also make the test conditional upon the host python version I guess so it only runs on newer hosts?

Cheers,

Richard

#12 lore raw

Re: [OE-core] [PATCH v2 4/4] scripts:recipetool:create_buildsys_python: add PEP517 support

Quoted 67 lines · Richard Purdie
Quoted 43 lines · Julien Stephan
Quoted 41 lines · Alexandre Belloni
Quoted 38 lines · Julien Stephan
Quoted 18 lines · Alexandre Belloni

Hello,

Quoted 14 lines · Julien Stephan

add support for PEP517 [1]

if a pyproject.toml file is found, use it to create the recipe, otherwise fallback to the old setup.py method.

[YOCTO #14737]

[1]: https://peps.python.org/pep-0517/

---
 .../lib/recipetool/create_buildsys_python.py  | 234 +++++++++++++++++-
 1 file changed, 233 insertions(+), 1 deletion(-)
Quoted 12 lines

scripts/lib/recipetool/create_buildsys_python.py import os · line 21

Quoted 9 lines
scripts/lib/recipetool/create_buildsys_python.py +1 -0
index 69f6f5ca511..0b601d50a4b 100644
@@ -18,6 +18,7 @@ import os
 import re
 import sys
 import subprocess
+import toml

This fails on the autobuilders because we don't have the toml module installed so I guess you need to add a dependency.

Hello,

Sure I 'll do it. Just to confirm, I should add it here:
https://docs.yoctoproject.org/ref-manual/system-requirements.html#required-packages-for-the-build-host
?

Quoted 8 lines
Quoted 3 lines

I guess the preferred way would be to depend on python3-toml-native instead of requiring installation on the host.

Hi Alexandre,

How am I supposed to do that for a script? Is that even possible? Am I missing something obvious?

As far as I know you're not missing anything obvious. If the toml dependency was in the target recipe this would be easier but needing this from recipetool is harder as it is running under the host python.

Adding dependencies for the host is hard as it needs to be added on the docs, on the autobuilder workers and into our buildtools-extended- tarballs and generally impacts a lot of people/places.

I think moving the import into the code and having recipe tool error and ask the user to install the dependency might be the best solution for now. That does give us a challenge over where/when we can test the code though.

Quoted 8 lines

On the autobuilder we could run recipetool for this test using the python3native from a recipe sysroot where the toml dependency is available I guess but that is a bit ugly.

As Tim mentions, with python 3.11 onwards, this problem does go away. We could also make the test conditional upon the host python version I guess so it only runs on newer hosts?

Hi Richard,

Thank you for the detailed explanation.
I think I will go and use tomllib and skip the test on systems with python < 3.11 if that is okay for everyone.

Makes me realize that I didn't add selftest for pep517, I will add them for the next version of the series.

Have a good week end

Cheers

Julien

Quoted 3 lines

Cheers,

Richard