Skip to contentSkip to conversation

classes/setuptools: use tomllib instead of regex to parse pyproject.toml

PATCH Superseded Ross Burton 2 participants 4 messages Sep 9

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 classes/setuptools: use tomllib instead of regex to parse pyproject.toml

BitBake now vendors in tomli[1] for when the host Python is older than 3.11 so doesn't have tomllib. Rewrite the pyproject.toml parsing from regexs to tomllib.

Also ensure that the functions in setuptools3 and setuptools_legacy are identical, as they had diverged.

[1] bitbake 69810e0c0a ("lib/bb/_vendor: resync to add tomli")

---
 meta/classes-recipe/setuptools3.bbclass       | 24 +++++++++++----
 .../classes-recipe/setuptools3_legacy.bbclass | 29 ++++++++++++++-----
 2 files changed, 40 insertions(+), 13 deletions(-)
meta/classes-recipe/setuptools3.bbclass +18 -6
index b0e4ab5208c..8e0fdff9e9d 100644
@@ -13,19 +13,31 @@ SETUPTOOLS_BUILD_ARGS ?= ""
 SETUPTOOLS_SETUP_PATH ?= "${S}"
 
 python do_check_backend() {
+    """
+    Check if this package has a pyproject.toml that specifies a PEP517 build backend, so should be
+    using a different build class.
+    """
+
     if "pep517-backend" in (d.getVar("INSANE_SKIP") or "").split():
         return
 
-    import re
+    try:
+        import tomllib
+    except ImportError:
+        import bb._vendor_tomli as tomllib
+
     filename = d.expand("${SETUPTOOLS_SETUP_PATH}/pyproject.toml")
     if os.path.exists(filename):
-        for line in open(filename):
-            match = re.match(r"build-backend\s*=\s*\W([\w.]+)\W", line)
-            if not match: continue
+        with open(filename, "rb") as f:
+            toml = tomllib.load(f)
 
-            msg = f"inherits setuptools3 but has pyproject.toml with {match[1]}, use the correct class"
+        try:
+            backend = toml["build-system"]["build-backend"]
+            msg = f"inherits setuptools3 but has pyproject.toml specifying backend {backend}, use the correct class"
             oe.qa.handle_error("pep517-backend", msg, d)
-    oe.qa.exit_if_errors(d)
+            oe.qa.exit_if_errors(d)
+        except KeyError:
+            return
 }
 addtask check_backend after do_patch before do_configure
 
meta/classes-recipe/setuptools3_legacy.bbclass +22 -7
index 6b51b9796bc..102d0827538 100644
@@ -31,16 +31,31 @@ SETUPTOOLS_PYTHON:class-native = "nativepython3"
 SETUPTOOLS_SETUP_PATH ?= "${S}"
 
 python do_check_backend() {
-    import re
+    """
+    Check if this package has a pyproject.toml that specifies a PEP517 build backend, so should be
+    using a different build class.
+    """
+
+    if "pep517-backend" in (d.getVar("INSANE_SKIP") or "").split():
+        return
+
+    try:
+        import tomllib
+    except ImportError:
+        import bb._vendor_tomli as tomllib
+
     filename = d.expand("${SETUPTOOLS_SETUP_PATH}/pyproject.toml")
     if os.path.exists(filename):
-        for line in open(filename):
-            match = re.match(r"build-backend\s*=\s*\W([\w.]+)\W", line)
-            if not match: continue
+        with open(filename, "rb") as f:
+            toml = tomllib.load(f)
 
-            msg = f"inherits setuptools3_legacy but has pyproject.toml with {match[1]}, use the correct class"
-            if "pep517-backend" not in (d.getVar("INSANE_SKIP") or "").split():
-                oe.qa.handle_error("pep517-backend", msg, d)
+        try:
+            backend = toml["build-system"]["build-backend"]
+            msg = f"inherits setuptools3_legacy but has pyproject.toml specifying backend {backend}, use the correct class"
+            oe.qa.handle_error("pep517-backend", msg, d)
+            oe.qa.exit_if_errors(d)
+        except KeyError:
+            return
 }
 addtask check_backend after do_patch before do_configure
 
Signature · 1 line
2.43.0
#1 lore raw

PATCH 2/2 classes/cargo-update-recipe-crates: use tomllib directly

Now that BitBake vendors in a copy of tomli for when the host Python is too old to support tomllib, we can do the crate update logic directly in the task instead of passing an large inline script to python3-native.

The only changes are the removal of nativepython3 and not relying on bitbake's shell expansion to expand variable names.

[1] bitbake 69810e0c0a ("lib/bb/_vendor: resync to add tomli")

---
 .../cargo-update-recipe-crates.bbclass        | 101 ++++++++++--------
 1 file changed, 54 insertions(+), 47 deletions(-)
meta/classes-recipe/cargo-update-recipe-crates.bbclass +54 -47
index 47e845c8221..e62039208c1 100644
@@ -14,7 +14,6 @@
 ## To perform the update: bitbake -c update_crates recipe-name
 
 addtask do_update_crates after do_patch
-do_update_crates[depends] = "python3-native:do_populate_sysroot"
 do_update_crates[nostamp] = "1"
 do_update_crates[doc] = "Update the recipe by reading Cargo.lock and write in ${THISDIR}/${BPN}-crates.inc"
 
@@ -23,58 +22,66 @@ RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates"
 # The directory where to search for Cargo.lock files
 CARGO_LOCK_SRC_DIR ??= "${S}"
 
-do_update_crates() {
-    TARGET_FILE="${THISDIR}/${BPN}-crates.inc"
+python do_update_crates() {
+    cargo_lock_src_dir = d.getVar("CARGO_LOCK_SRC_DIR")
 
-    nativepython3 - <<EOF
+    def get_crates(filename):
+        try:
+            import tomllib
+        except ImportError:
+            import bb._vendor.tomli as tomllib
 
-def get_crates(f):
-    import tomllib
-    c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}')
-    c_list += '\nSRC_URI += " \\\'
-    crates = tomllib.load(open(f, 'rb'))
+        c_list = '# from %s' % os.path.relpath(filename, cargo_lock_src_dir)
+        c_list += '\nSRC_URI += " \\'
+        with open(filename, 'rb') as fp:
+            crates = tomllib.load(fp)
 
-    # Build a list with crates info that have crates.io in the source
-    crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package']))
+        # Build a list with crates info that have crates.io in the source
+        crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package']))
 
-    if not crates_candidates:
-        print("WARNING: Unable to find any candidate crates that use crates.io")
-        return None
+        if not crates_candidates:
+            print("WARNING: Unable to find any candidate crates that use crates.io")
+            return None
 
-    # Update crates uri and their checksum, to avoid name clashing on the checksum
-    # we need to rename crates with name and version to have a unique key
-    cksum_list = ''
-    for c in crates_candidates:
-        rename = "%s-%s" % (c['name'], c['version'])
-        c_list += '\n    crate://crates.io/%s/%s \\\' % (c['name'], c['version'])
-        if 'checksum' in c:
-            cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
+        # Update crates uri and their checksum, to avoid name clashing on the checksum
+        # we need to rename crates with name and version to have a unique key
+        cksum_list = ''
+        for c in crates_candidates:
+            rename = "%s-%s" % (c['name'], c['version'])
+            c_list += '\n    crate://crates.io/%s/%s \\' % (c['name'], c['version'])
+            if 'checksum' in c:
+                cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
 
-    c_list += '\n"\n'
-    c_list += cksum_list
-    c_list += '\n'
-    return c_list
+        c_list += '\n"\n'
+        c_list += cksum_list
+        c_list += '\n'
+        return c_list
 
-import os
-crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    # ignore git and patches directories
-    if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.pc')):
-        continue
-    if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.git')):
-        continue
-    for file in files:
-        if file == 'Cargo.lock':
-            cargo_lock_path = os.path.join(root, file)
-            c = get_crates(cargo_lock_path)
-            if c is not None:
-                crates += c
-if crates is None:
-    raise ValueError("Unable to find any Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
-with open("${TARGET_FILE}", 'w') as f:
-    f.write(crates)
-EOF
+    import os
 
-    bbnote "Successfully update crates inside '${TARGET_FILE}'"
+    pn = d.getVar("PN")
+    crates = f"# Autogenerated with 'bitbake -c update_crates {pn}'\n\n"
+
+    found = False
+    for root, dirs, files in os.walk(cargo_lock_src_dir):
+        # ignore git and patches directories
+        if root.startswith(os.path.join(cargo_lock_src_dir, '.pc')):
+            continue
+        if root.startswith(os.path.join(cargo_lock_src_dir, '.git')):
+            continue
+        for file in files:
+            if file == 'Cargo.lock':
+                cargo_lock_path = os.path.join(root, file)
+                c = get_crates(cargo_lock_path)
+                if c is not None:
+                    crates += c
+
+    if crates is None:
+        raise ValueError(f"Unable to find any Cargo.lock in {cargo_lock_src_dir}")
+
+    target_file = d.expand("${THISDIR}/${BPN}-crates.inc")
+    with open(target_file, 'w') as f:
+        f.write(crates)
+
+    bb.note(f"Successfully updated crates inside {target_file}")
 }
Signature · 1 line
2.43.0
#2 lore raw
In reply to Ross Burton
Quoted 11 lines · Ross Burton via lists.openembedded.org

BitBake now vendors in tomli[1] for when the host Python is older than 3.11 so doesn't have tomllib. Rewrite the pyproject.toml parsing from regexs to tomllib.

Also ensure that the functions in setuptools3 and setuptools_legacy are identical, as they had diverged.

[1] bitbake 69810e0c0a ("lib/bb/_vendor: resync to add tomli")

---

Hi Ross,

I've been getting some error on the autobuilder, event though I'm in sync on the bitbake side:

ERROR: python3-semantic-version-native-2.10.0-r0 do_check_backend: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_check_backend(d)
     0003:

File: '/srv/pokybuild/yocto-worker/qemux86-64-ptest/build/layers/openembedded-core/meta/classes-recipe/setuptools3.bbclass', lineno: 27, function: do_check_backend
0023:
0024: try:
0025: import tomllib
0026: except ImportError:
*** 0027: import bb._vendor_tomli as tomllib

     0028:
     0029:    filename = d.expand("${SETUPTOOLS_SETUP_PATH}/pyproject.toml")
     0030:    if os.path.exists(filename):
     0031:        with open(filename, "rb") as f:

Exception: ModuleNotFoundError: No module named 'bb._vendor_tomli'

https://autobuilder.yoctoproject.org/valkyrie/#/builders/73/builds/4525

Any clue about what might be wrong here?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw
Quoted 36 lines · Mathieu Dubois-Briand
Quoted 11 lines · Ross Burton via lists.openembedded.org

BitBake now vendors in tomli[1] for when the host Python is older than 3.11 so doesn't have tomllib. Rewrite the pyproject.toml parsing from regexs to tomllib.

Also ensure that the functions in setuptools3 and setuptools_legacy are identical, as they had diverged.

[1] bitbake 69810e0c0a ("lib/bb/_vendor: resync to add tomli")

---

Hi Ross,

I've been getting some error on the autobuilder, event though I'm in sync on the bitbake side:

ERROR: python3-semantic-version-native-2.10.0-r0 do_check_backend: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>

    0001:
*** 0002:do_check_backend(d)
    0003:

File: '/srv/pokybuild/yocto-worker/qemux86-64-ptest/build/layers/openembedded-core/meta/classes-recipe/setuptools3.bbclass', lineno: 27, function: do_check_backend
0023:
0024: try:
0025: import tomllib
0026: except ImportError:
*** 0027: import bb._vendor_tomli as tomllib

    0028:
    0029:    filename = d.expand("${SETUPTOOLS_SETUP_PATH}/pyproject.toml")
    0030:    if os.path.exists(filename):
    0031:        with open(filename, "rb") as f:
Quoted 5 lines

Exception: ModuleNotFoundError: No module named 'bb._vendor_tomli'

https://autobuilder.yoctoproject.org/valkyrie/#/builders/73/builds/4525

Any clue about what might be wrong here?

Sigh, typo. v2 incoming.

Ross