Skip to contentSkip to conversation

recipetool: add plugin to create recipe for rust based application

RFC PATCH New Julien Stephan 1 participant 2 messages Nov 27, 2023

lore
Share & tools
lore
#0 lore raw

cover recipetool: add plugin to create recipe for rust based application

Hi all,

This is an RFC to add support of rust recipes in recipetool.

This first version is able to create a minimal working recipe for crates HAVING a Cargo.lock file (needed for reproducibility issues, see [1]) and using the following commands (using simple_genetic as example):

- devtool add https://github.com/ValpsZ/simple_genetic
- bitbake -c update_crates simple-genetic
- devtool finish simple-genetic <layer>

Limitations:
- maybe we should factorise the code inside update_crates to avoid calling it after devtool add
- for now the created recipe contains "include ${BPN}-crate.inc" instead of "require ${BPN}-crate.inc" because when the recipe is created, devtool tries to build the recipe itself and with "require" the parsing fails and the recipe is discarded. Maybe first point can help here. Another solution may be to write an empty ${BPN}-crate.inc next to the created recipe BUT right now I don't think it is possible because the process callback does not know neither the output dir nor ${BPN}. Maybe adding an optionnal post_process callback can help here?

Unfortunately, most of the crates I tried, do NOT version the Cargo.lock file, so we need a solution for that. Maybe the "update_crates" function should do that? If no Cargo.lock file is found, just create one, and create a patch for it. When a recipe is updated, the user upgrading SRCREV (or the version) must be resposible to also run bitbake -c update_crates to update the Cargo.lock and the ${BP}-crate.inc files. I think that should be ok for reproducible builds? What do you think?

What's next:
- add support for "devtool add https://crates.io/crates/simple_genetic/0.1.1"
- add oeqa selftest
- handle the no Cargo.lock case...

NOTE: created recipes are built tested but not tested at runtime for now.

Dev branch on poky-contrib: [2]

[1]: https://git.yoctoproject.org/poky/tree/meta/classes-recipe/cargo.bbclass#n45
[2]: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-add-rust-support

Julien Stephan (1):
recipetool: create: add support for crates

 scripts/lib/recipetool/create_cargo.py | 122 +++++++++++++++++++++++++
 1 file changed, 122 insertions(+)
 create mode 100644 scripts/lib/recipetool/create_cargo.py
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/1 recipetool: create: add support for crates

This commit adds a new plugin for creating recipes for crates (rust based applications).

TODO: add more description here on final version of the patch

---
 scripts/lib/recipetool/create_cargo.py | 125 +++++++++++++++++++++++++
 1 file changed, 125 insertions(+)
 create mode 100644 scripts/lib/recipetool/create_cargo.py
scripts/lib/recipetool/create_cargo.py new file +125 -0
new file mode 100644
index 00000000000..d24ce63509c
@@ -0,0 +1,125 @@
+# Recipe creation tool - cargo support plugin
+#
+# Copyright (c) 2023 BayLibre, SAS
+# Author: Julien Stephan <[email protected]>
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+import logging
+import os
+from recipetool.create import RecipeHandler
+
+logger = logging.getLogger("recipetool")
+
+tinfoil = None
+
+
+def tinfoil_init(instance):
+    global tinfoil
+    tinfoil = instance
+
+
+class CargoRecipeHandler(RecipeHandler):
+    """
+    Base class to support rust crate
+    """
+
+    # The supported metadata list
+    # Exaustive list available here: https://doc.rust-lang.org/cargo/reference/manifest.html#the-manifest-format
+    manifest_package_table = [
+        "name",
+        "version",
+        "homepage",
+        "description",
+        "license",
+        "license-file",
+        "dependencies",
+    ]
+
+    bbvar_map = {
+        "name": "PN",
+        "version": "PV",
+        "homepage": "HOMEPAGE",
+        "description": "SUMMARY",
+        "license": "LICENSE",
+    }
+
+    def __init__(self):
+        pass
+
+    def process(
+        self, srctree, classes, lines_before, lines_after, handled, extravalues
+    ):
+        if "buildsystem" in handled:
+            return False
+
+        # Check for non-zero size Cargo.toml file
+        manifestfiles = RecipeHandler.checkfiles(srctree, ["Cargo.toml"])
+        for fn in manifestfiles:
+            if os.path.getsize(fn):
+                break
+        else:
+            return False
+
+        manifest = os.path.join(srctree, "Cargo.toml")
+
+        lockfile = RecipeHandler.checkfiles(srctree, ["Cargo.lock"])
+        if lockfile == []:
+            logger.error(
+                "No cargo lock found, this is not yet supported, falling back to another method"
+            )
+            return False
+
+        try:
+            import tomllib
+        except ImportError:
+            try:
+                import tomli as tomllib
+            except ImportError:
+                logger.exception(
+                    "Neither 'tomllib' nor 'tomli' could be imported. Please use python3.11 or above or install tomli module"
+                )
+                return False
+            except Exception:
+                logger.exception("Failed to parse pyproject.toml")
+                return False
+
+        try:
+            with open(manifest, "rb") as f:
+                config = tomllib.load(f)
+
+            metadata = config["package"]
+
+            if metadata:
+                for field, value in metadata.items():
+                    if field in self.manifest_package_table:
+                        if field == "license":
+                            # license field is an SPDX 2.1 license expression
+                            # https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields
+                            value = value.replace("AND", "&")
+                            value = value.replace("OR", "|")
+                            # even if deprecated we can still find some old crate using "/" to separate mulitple licenses
+                            value = value.replace("/", " & ")
+                        elif field == "license-file":
+                            # TODO: handle this properly
+                            continue
+
+                        extravalues[self.bbvar_map[field]] = value
+
+            if "dependencies" in extravalues:
+                lines_after.append("include ${BPN}-crates.inc")
+                extravalues.pop("dependencies")
+            classes.append("cargo")
+            classes.append("cargo-update-recipe-crates")
+            handled.append("buildsystem")
+        except Exception:
+            logger.exception(
+                "Failed to correctly handle Cargo.toml, falling back to another method"
+            )
+            return False
+
+
+def register_recipe_handlers(handlers):
+    # We need to make sure these are ahead of the makefile fallback handler
+    handlers.append((CargoRecipeHandler(), 80))
Signature · 1 line
2.42.0