Skip to contentSkip to conversation

rust: generate_rust_analyzer.py: define host crates and scripts

PATCH New Tamir Duberstein 1 participant 14 messages Apr 24, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v6 cover rust: generate_rust_analyzer.py: define host crates and scripts

This series updates rust-project.json to differentiate between host and target crates, where the former are used as dependencies of the macros crate. Please see individual commit messages for details.

The first 3 commits contain mechanical formatting changes and are optional. The series can be taken without them.

I avoided more significant formatting or changes where possible to reduce the diff. Unfortunately scripts/generate_rust_analyzer.py is not consistently formatted before nor after this series.

The 5th commit ("scripts: generate_rust_analyzer.py: use str(pathlib.Path)") can also be considered optional. It removes an inconsistency I noticed while working on this series and which occurs on a line which churns in this series anyway.

The trailing commits can also be considered optional, as they can be submitted separately. I included them in this series because they depend on it, but they can be split out if this is preferred.

---
Changes in v6:
- Rebase on v6.15-rc3.
- Extract local variable crates_cfg. (Daniel Almeida)
- Replace map with generators. (Trevor Gross)
- Replace str.rstrip with Pathlib.path.stem. (Trevor Gross)
- Use Pathlib.path.read_text. (Trevor Gross)
- Explain why cfg_groups only applies to workspace crates. (Trevor
Gross)
- Link to v5: https://lore.kernel.org/r/[email protected]

Changes in v5:
- Split remove "is_proc_macro": false into separate patch. (Daniel
Almeida)
- Move whitespace change from "add type hints" to "add missing
whitespace".
- Add patch to avoid default arguments and force keyword arguments.
- Add patch to add type hints for CLI parameters.
- Replace some instances of str.replace with str.{l,r}strip.
- Rebase on next-20250325. This must be rooted on linux-next because
it's the only place rust-next and rust-fixes have been merged.
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- Fix typo (s/generate/generated/).
- Pull Trevor's suggested change into a separate patch with a
Suggested-by tag.
- Add patch to avoid file descriptor leak.
- Add patch to generate rust-project.json entries for scripts/*.rs.
- Add patch to use cfg_groups to reduce size of rust-project.json by
90%.
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- Rebase on linux-next. This is needed to pick up all the conflicts from
both rust-next and rust-fixes.
- Drop uv from mypy command. (Trevor Gross)
- Add --python-version 3.8 to mypy command. (Trevor Gross)
- from typings import ... directly. (Trevor Gross)
- Extract build_crate and register_crate to avoid peeking into
crates[-1]. (Trevor Gross)
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- Rebased on "rust: fix rust-analyzer configuration for generated files" [1]

- Link to v1: https://lore.kernel.org/r/[email protected]

---
Tamir Duberstein (13):
scripts: generate_rust_analyzer.py: add missing whitespace
scripts: generate_rust_analyzer.py: use double quotes
scripts: generate_rust_analyzer.py: add trailing comma
scripts: generate_rust_analyzer.py: extract {build,register}_crate
scripts: generate_rust_analyzer.py: drop "is_proc_macro": false
scripts: generate_rust_analyzer.py: add type hints
scripts: generate_rust_analyzer.py: avoid optional arguments
scripts: generate_rust_analyzer.py: use str(pathlib.Path)
scripts: generate_rust_analyzer.py: identify crates explicitly
scripts: generate_rust_analyzer.py: define host crates
scripts: generate_rust_analyzer.py: avoid FD leak
scripts: generate_rust_analyzer.py: define scripts
scripts: generate_rust_analyzer.py: use cfg_groups

 scripts/generate_rust_analyzer.py | 334 ++++++++++++++++++++++++++++----------
 1 file changed, 245 insertions(+), 89 deletions(-)

---
base-commit: 9d7a0577c9db35c4cc52db90bc415ea248446472
change-id: 20250209-rust-analyzer-host-43b108655578

Best regards,

Signature · 1 line
Tamir Duberstein <[email protected]>
#1 lore raw

PATCH v6 1/13 scripts: generate_rust_analyzer.py: add missing whitespace

Add a space before the / operator for consistency with surrounding code and code formatting tools. Add a second newline between top-level items in accordance with PEP 8[1]:

Quoted 1 line

Surround top-level function and class definitions with two blank

lines.

This change was made by a code formatting tool.

---
 scripts/generate_rust_analyzer.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +3 -1
index fe663dd0c43b..f66c74dd4e8d 100755
@@ -124,7 +124,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     ):
         append_crate(
             display_name,
-            srctree / "rust"/ display_name / "lib.rs",
+            srctree / "rust" / display_name / "lib.rs",
             deps,
             cfg=cfg,
         )
@@ -173,6 +173,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
     return crates
 
+
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('--verbose', '-v', action='store_true')
@@ -199,5 +200,6 @@ def main():
 
     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
 
+
 if __name__ == "__main__":
     main()
Signature · 1 line
2.49.0
#2 lore raw

PATCH v6 2/13 scripts: generate_rust_analyzer.py: use double quotes

In reply to Tamir Duberstein

Replace inconsistent use of single quotes with double quotes.

This change was made by a code formatting tool.

---
 scripts/generate_rust_analyzer.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/generate_rust_analyzer.py +2 -2
index f66c74dd4e8d..c473e7a2d6f4 100755
@@ -176,8 +176,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument('--verbose', '-v', action='store_true')
-    parser.add_argument('--cfgs', action='append', default=[])
+    parser.add_argument("--verbose", "-v", action="store_true")
+    parser.add_argument("--cfgs", action="append", default=[])
     parser.add_argument("srctree", type=pathlib.Path)
     parser.add_argument("objtree", type=pathlib.Path)
     parser.add_argument("sysroot", type=pathlib.Path)
Signature · 1 line
2.49.0
#3 lore raw

PATCH v6 3/13 scripts: generate_rust_analyzer.py: add trailing comma

In reply to Tamir Duberstein

Add missing trailing comma on multi-line function call as suggested by PEP-8:

Quoted 3 lines

The pattern is to put each value (etc.) on a line by itself, always adding a trailing comma, and add the close parenthesis/bracket/brace on the next line.

This change was made by a code formatting tool.

---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
index c473e7a2d6f4..828e7797d253 100755
@@ -187,7 +187,7 @@ def main():
 
     logging.basicConfig(
         format="[%(asctime)s] [%(levelname)s] %(message)s",
-        level=logging.INFO if args.verbose else logging.WARNING
+        level=logging.INFO if args.verbose else logging.WARNING,
     )
 
     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
Signature · 1 line
2.49.0
#4 lore raw

PATCH v6 4/13 scripts: generate_rust_analyzer.py: extract `{build,register}_crate`

In reply to Tamir Duberstein

Extract helpers from append_crate to avoid the need to peek into crates[-1]. This improves readability.

---
 scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)
scripts/generate_rust_analyzer.py +30 -5
index 828e7797d253..74f7ab5541a1 100755
@@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     crates_indexes = {}
     crates_cfgs = args_crates_cfgs(cfgs)
 
-    def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False):
+    def build_crate(
+        display_name,
+        root_module,
+        deps,
+        cfg=[],
+        is_workspace_member=True,
+        is_proc_macro=False,
+    ):
         crate = {
             "display_name": display_name,
             "root_module": str(root_module),
@@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
                 stdin=subprocess.DEVNULL,
             ).decode('utf-8').strip()
             crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
-        crates_indexes[display_name] = len(crates)
+        return crate
+
+    def register_crate(crate):
+        crates_indexes[crate["display_name"]] = len(crates)
         crates.append(crate)
 
+    def append_crate(
+        display_name,
+        root_module,
+        deps,
+        cfg=[],
+        is_workspace_member=True,
+        is_proc_macro=False,
+    ):
+        register_crate(
+            build_crate(
+                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
+            )
+        )
+
     def append_sysroot_crate(
         display_name,
         deps,
@@ -122,20 +146,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         display_name,
         deps,
     ):
-        append_crate(
+        crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
             deps,
             cfg=cfg,
         )
-        crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True))
-        crates[-1]["source"] = {
+        crate["env"]["OBJTREE"] = str(objtree.resolve(True))
+        crate["source"] = {
             "include_dirs": [
                 str(srctree / "rust" / display_name),
                 str(objtree / "rust")
             ],
             "exclude_dirs": [],
         }
+        register_crate(crate)
 
     append_crate_with_generated("bindings", ["core", "ffi"])
     append_crate_with_generated("uapi", ["core", "ffi"])
Signature · 1 line
2.49.0
#5 lore raw

PATCH v6 5/13 scripts: generate_rust_analyzer.py: drop `"is_proc_macro": false`

In reply to Tamir Duberstein

Add a dedicated append_proc_macro_crate function to reduce overloading in append_crate. This has the effect of removing `"is_proc_macro":
false` from the output; this field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.

---
 scripts/generate_rust_analyzer.py | 55 +++++++++++++++++++++++++++------------
 1 file changed, 39 insertions(+), 16 deletions(-)
scripts/generate_rust_analyzer.py +39 -16
index 74f7ab5541a1..a7f94f3be93e 100755
@@ -41,13 +41,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         deps,
         cfg=[],
         is_workspace_member=True,
-        is_proc_macro=False,
     ):
-        crate = {
+        return {
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "is_proc_macro": is_proc_macro,
             "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
             "cfg": cfg,
             "edition": "2021",
@@ -55,13 +53,6 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
                 "RUST_MODFILE": "This is only for rust-analyzer"
             }
         }
-        if is_proc_macro:
-            proc_macro_dylib_name = subprocess.check_output(
-                [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
-                stdin=subprocess.DEVNULL,
-            ).decode('utf-8').strip()
-            crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}"
-        return crate
 
     def register_crate(crate):
         crates_indexes[crate["display_name"]] = len(crates)
@@ -73,14 +64,48 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         deps,
         cfg=[],
         is_workspace_member=True,
-        is_proc_macro=False,
     ):
         register_crate(
             build_crate(
-                display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro
+                display_name,
+                root_module,
+                deps,
+                cfg,
+                is_workspace_member,
             )
         )
 
+    def append_proc_macro_crate(
+        display_name,
+        root_module,
+        deps,
+        cfg=[],
+    ):
+        crate = build_crate(display_name, root_module, deps, cfg)
+        proc_macro_dylib_name = (
+            subprocess.check_output(
+                [
+                    os.environ["RUSTC"],
+                    "--print",
+                    "file-names",
+                    "--crate-name",
+                    display_name,
+                    "--crate-type",
+                    "proc-macro",
+                    "-",
+                ],
+                stdin=subprocess.DEVNULL,
+            )
+            .decode("utf-8")
+            .strip()
+        )
+        proc_macro_crate = {
+            **crate,
+            "is_proc_macro": True,
+            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
+        }
+        register_crate(proc_macro_crate)
+
     def append_sysroot_crate(
         display_name,
         deps,
@@ -108,11 +133,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         [],
     )
 
-    append_crate(
+    append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
         ["std", "proc_macro"],
-        is_proc_macro=True,
     )
 
     append_crate(
@@ -121,12 +145,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         ["core", "compiler_builtins"],
     )
 
-    append_crate(
+    append_proc_macro_crate(
         "pin_init_internal",
         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
         [],
         cfg=["kernel"],
-        is_proc_macro=True,
     )
 
     append_crate(
Signature · 1 line
2.49.0
#6 lore raw

PATCH v6 6/13 scripts: generate_rust_analyzer.py: add type hints

In reply to Tamir Duberstein

Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.

Python type hints have been present in the kernel since 2019 at the latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests").

Add a subclass of argparse.Namespace to get type checking on the CLI arguments. Move parsing of cfg out of generate_crates to reduce the number of variables in scope with cfg in their name. Use a defaultdict to avoid .get("key", []).

Run mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8 to verify. Note that mypy no longer supports python < 3.8.

---
 scripts/generate_rust_analyzer.py | 163 +++++++++++++++++++++++++-------------
 1 file changed, 106 insertions(+), 57 deletions(-)
scripts/generate_rust_analyzer.py +106 -57
index a7f94f3be93e..6fa4cec0e506 100755
@@ -10,16 +10,48 @@ import os
 import pathlib
 import subprocess
 import sys
+from collections import defaultdict
+from typing import DefaultDict, Dict, Iterable, List, Literal, Optional, TypedDict
 
-def args_crates_cfgs(cfgs):
-    crates_cfgs = {}
-    for cfg in cfgs:
-        crate, vals = cfg.split("=", 1)
-        crates_cfgs[crate] = vals.replace("--cfg", "").split()
 
-    return crates_cfgs
+class Dependency(TypedDict):
+    crate: int
+    name: str
 
-def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
+
+class Source(TypedDict):
+    include_dirs: List[str]
+    exclude_dirs: List[str]
+
+
+class Crate(TypedDict):
+    display_name: str
+    root_module: str
+    is_workspace_member: bool
+    deps: List[Dependency]
+    cfg: List[str]
+    edition: Literal["2021"]
+    env: Dict[str, str]
+
+
+# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
+class ProcMacroCrate(Crate):
+    is_proc_macro: Literal[True]
+    proc_macro_dylib_path: Optional[str]  # `pathlib.Path` is not JSON serializable.
+
+
+# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11.
+class CrateWithGenerated(Crate):
+    source: Optional[Source]
+
+
+def generate_crates(
+    srctree: pathlib.Path,
+    objtree: pathlib.Path,
+    sysroot_src: pathlib.Path,
+    external_src: pathlib.Path,
+    crates_cfgs: DefaultDict[str, List[str]],
+) -> List[Crate]:
     # Generate the configuration list.
     cfg = []
     with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -31,17 +63,16 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Now fill the crates list -- dependencies need to come first.
     #
     # Avoid O(n^2) iterations by keeping a map of indexes.
-    crates = []
-    crates_indexes = {}
-    crates_cfgs = args_crates_cfgs(cfgs)
+    crates: List[Crate] = []
+    crates_indexes: Dict[str, int] = {}
 
     def build_crate(
-        display_name,
-        root_module,
-        deps,
-        cfg=[],
-        is_workspace_member=True,
-    ):
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+        is_workspace_member: bool = True,
+    ) -> Crate:
         return {
             "display_name": display_name,
             "root_module": str(root_module),
@@ -51,36 +82,30 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             "edition": "2021",
             "env": {
                 "RUST_MODFILE": "This is only for rust-analyzer"
-            }
+            },
         }
 
-    def register_crate(crate):
+    def register_crate(crate: Crate) -> None:
         crates_indexes[crate["display_name"]] = len(crates)
         crates.append(crate)
 
     def append_crate(
-        display_name,
-        root_module,
-        deps,
-        cfg=[],
-        is_workspace_member=True,
-    ):
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+        is_workspace_member: bool = True,
+    ) -> None:
         register_crate(
-            build_crate(
-                display_name,
-                root_module,
-                deps,
-                cfg,
-                is_workspace_member,
-            )
+            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
         )
 
     def append_proc_macro_crate(
-        display_name,
-        root_module,
-        deps,
-        cfg=[],
-    ):
+        display_name: str,
+        root_module: pathlib.Path,
+        deps: List[str],
+        cfg: List[str] = [],
+    ) -> None:
         crate = build_crate(display_name, root_module, deps, cfg)
         proc_macro_dylib_name = (
             subprocess.check_output(
@@ -99,7 +124,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             .decode("utf-8")
             .strip()
         )
-        proc_macro_crate = {
+        proc_macro_crate: ProcMacroCrate = {
             **crate,
             "is_proc_macro": True,
             "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
@@ -107,10 +132,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
         register_crate(proc_macro_crate)
 
     def append_sysroot_crate(
-        display_name,
-        deps,
-        cfg=[],
-    ):
+        display_name: str,
+        deps: List[str],
+        cfg: List[str] = [],
+    ) -> None:
         append_crate(
             display_name,
             sysroot_src / display_name / "src" / "lib.rs",
@@ -122,7 +147,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
-    append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", []))
+    append_sysroot_crate("core", [], cfg=crates_cfgs["core"])
     append_sysroot_crate("alloc", ["core"])
     append_sysroot_crate("std", ["alloc", "core"])
     append_sysroot_crate("proc_macro", ["core", "std"])
@@ -166,9 +191,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     )
 
     def append_crate_with_generated(
-        display_name,
-        deps,
-    ):
+        display_name: str,
+        deps: List[str],
+    ) -> None:
         crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
@@ -176,20 +201,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
             cfg=cfg,
         )
         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
-        crate["source"] = {
-            "include_dirs": [
-                str(srctree / "rust" / display_name),
-                str(objtree / "rust")
-            ],
-            "exclude_dirs": [],
+        crate_with_generated: CrateWithGenerated = {
+            **crate,
+            "source": {
+                "include_dirs": [
+                    str(srctree / "rust" / display_name),
+                    str(objtree / "rust")
+                ],
+                "exclude_dirs": [],
+            }
         }
-        register_crate(crate)
+        register_crate(crate_with_generated)
 
     append_crate_with_generated("bindings", ["core", "ffi"])
     append_crate_with_generated("uapi", ["core", "ffi"])
     append_crate_with_generated("kernel", ["core", "macros", "build_error", "pin_init", "ffi", "bindings", "uapi"])
 
-    def is_root_crate(build_file, target):
+    def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
             return f"{target}.o" in open(build_file).read()
         except FileNotFoundError:
@@ -198,7 +226,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     # Then, the rest outside of `rust/`.
     #
     # We explicitly mention the top-level folders we want to cover.
-    extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers"))
+    extra_dirs: Iterable[pathlib.Path] = (srctree / dir for dir in ("rust", "drivers"))
     if external_src is not None:
         extra_dirs = [external_src]
     for folder in extra_dirs:
@@ -222,7 +250,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
     return crates
 
 
-def main():
+def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--verbose", "-v", action="store_true")
     parser.add_argument("--cfgs", action="append", default=[])
@@ -231,7 +259,17 @@ def main():
     parser.add_argument("sysroot", type=pathlib.Path)
     parser.add_argument("sysroot_src", type=pathlib.Path)
     parser.add_argument("exttree", type=pathlib.Path, nargs="?")
-    args = parser.parse_args()
+
+    class Args(argparse.Namespace):
+        verbose: bool
+        cfgs: List[str]
+        srctree: pathlib.Path
+        objtree: pathlib.Path
+        sysroot: pathlib.Path
+        sysroot_src: pathlib.Path
+        exttree: pathlib.Path
+
+    args = parser.parse_args(namespace=Args())
 
     logging.basicConfig(
         format="[%(asctime)s] [%(levelname)s] %(message)s",
@@ -241,8 +279,19 @@ def main():
     # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
     assert args.sysroot in args.sysroot_src.parents
 
+    crates_cfg = {
+        crate: vals.lstrip("--cfg").split()
+        for crate, _, vals in (cfg.partition("=") for cfg in args.cfgs)
+    }
+
     rust_project = {
-        "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs),
+        "crates": generate_crates(
+            args.srctree,
+            args.objtree,
+            args.sysroot_src,
+            args.exttree,
+            defaultdict(list, crates_cfg),
+        ),
         "sysroot": str(args.sysroot),
     }
 
Signature · 1 line
2.49.0
#7 lore raw

PATCH v6 7/13 scripts: generate_rust_analyzer.py: avoid optional arguments

In reply to Tamir Duberstein

Make all arguments required to reduce the probability of incorrect implicit behavior. Use keyword arguments for clarity.

---
 scripts/generate_rust_analyzer.py | 94 +++++++++++++++++++++++++++------------
 1 file changed, 65 insertions(+), 29 deletions(-)
scripts/generate_rust_analyzer.py +65 -29
index 6fa4cec0e506..66e3d5e8c52c 100755
@@ -69,9 +69,10 @@ def generate_crates(
     def build_crate(
         display_name: str,
         root_module: pathlib.Path,
+        *,
         deps: List[str],
-        cfg: List[str] = [],
-        is_workspace_member: bool = True,
+        cfg: List[str],
+        is_workspace_member: bool,
     ) -> Crate:
         return {
             "display_name": display_name,
@@ -92,21 +93,34 @@ def generate_crates(
     def append_crate(
         display_name: str,
         root_module: pathlib.Path,
+        *,
         deps: List[str],
-        cfg: List[str] = [],
-        is_workspace_member: bool = True,
+        cfg: List[str],
     ) -> None:
         register_crate(
-            build_crate(display_name, root_module, deps, cfg, is_workspace_member)
+            build_crate(
+                display_name,
+                root_module,
+                deps=deps,
+                cfg=cfg,
+                is_workspace_member=True,
+            )
         )
 
     def append_proc_macro_crate(
         display_name: str,
         root_module: pathlib.Path,
+        *,
         deps: List[str],
-        cfg: List[str] = [],
+        cfg: List[str],
     ) -> None:
-        crate = build_crate(display_name, root_module, deps, cfg)
+        crate = build_crate(
+            display_name,
+            root_module,
+            deps=deps,
+            cfg=cfg,
+            is_workspace_member=True,
+        )
         proc_macro_dylib_name = (
             subprocess.check_output(
                 [
@@ -133,72 +147,82 @@ def generate_crates(
 
     def append_sysroot_crate(
         display_name: str,
+        *,
         deps: List[str],
-        cfg: List[str] = [],
+        cfg: List[str],
     ) -> None:
-        append_crate(
-            display_name,
-            sysroot_src / display_name / "src" / "lib.rs",
-            deps,
-            cfg,
-            is_workspace_member=False,
+        register_crate(
+            build_crate(
+                display_name,
+                sysroot_src / display_name / "src" / "lib.rs",
+                deps=deps,
+                cfg=cfg,
+                is_workspace_member=False,
+            )
         )
 
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
-    append_sysroot_crate("core", [], cfg=crates_cfgs["core"])
-    append_sysroot_crate("alloc", ["core"])
-    append_sysroot_crate("std", ["alloc", "core"])
-    append_sysroot_crate("proc_macro", ["core", "std"])
+    append_sysroot_crate("core", deps=[], cfg=crates_cfgs["core"])
+    append_sysroot_crate("alloc", deps=["core"], cfg=[])
+    append_sysroot_crate("std", deps=["alloc", "core"], cfg=[])
+    append_sysroot_crate("proc_macro", deps=["core", "std"], cfg=[])
 
     append_crate(
         "compiler_builtins",
         srctree / "rust" / "compiler_builtins.rs",
-        [],
+        deps=[],
+        cfg=[],
     )
 
     append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        ["std", "proc_macro"],
+        deps=["std", "proc_macro"],
+        cfg=[],
     )
 
     append_crate(
         "build_error",
         srctree / "rust" / "build_error.rs",
-        ["core", "compiler_builtins"],
+        deps=["core", "compiler_builtins"],
+        cfg=[],
     )
 
     append_proc_macro_crate(
         "pin_init_internal",
         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
-        [],
+        deps=[],
         cfg=["kernel"],
     )
 
     append_crate(
         "pin_init",
         srctree / "rust" / "pin-init" / "src" / "lib.rs",
-        ["core", "pin_init_internal", "macros"],
+        deps=["core", "pin_init_internal", "macros"],
         cfg=["kernel"],
     )
 
     append_crate(
         "ffi",
         srctree / "rust" / "ffi.rs",
-        ["core", "compiler_builtins"],
+        deps=["core", "compiler_builtins"],
+        cfg=[],
     )
 
     def append_crate_with_generated(
         display_name: str,
+        *,
         deps: List[str],
+        cfg: List[str],
     ) -> None:
         crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
-            deps,
+            deps=deps,
             cfg=cfg,
+            is_workspace_member=True,
         )
         crate["env"]["OBJTREE"] = str(objtree.resolve(True))
         crate_with_generated: CrateWithGenerated = {
@@ -213,9 +237,21 @@ def generate_crates(
         }
         register_crate(crate_with_generated)
 
-    append_crate_with_generated("bindings", ["core", "ffi"])
-    append_crate_with_generated("uapi", ["core", "ffi"])
-    append_crate_with_generated("kernel", ["core", "macros", "build_error", "pin_init", "ffi", "bindings", "uapi"])
+    append_crate_with_generated("bindings", deps=["core", "ffi"], cfg=[])
+    append_crate_with_generated("uapi", deps=["core", "ffi"], cfg=[])
+    append_crate_with_generated(
+        "kernel",
+        deps=[
+            "core",
+            "macros",
+            "build_error",
+            "pin_init",
+            "ffi",
+            "bindings",
+            "uapi",
+        ],
+        cfg=[],
+    )
 
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
@@ -243,7 +279,7 @@ def generate_crates(
             append_crate(
                 name,
                 path,
-                ["core", "kernel"],
+                deps=["core", "kernel"],
                 cfg=cfg,
             )
 
Signature · 1 line
2.49.0
#8 lore raw

PATCH v6 8/13 scripts: generate_rust_analyzer.py: use str(pathlib.Path)

In reply to Tamir Duberstein

Use the / operator on pathlib.Path rather than directly crafting a string. This is consistent with all other path manipulation in this script.

---
 scripts/generate_rust_analyzer.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
index 66e3d5e8c52c..bc23cf58e7a1 100755
@@ -141,7 +141,7 @@ def generate_crates(
         proc_macro_crate: ProcMacroCrate = {
             **crate,
             "is_proc_macro": True,
-            "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
+            "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
         register_crate(proc_macro_crate)
 
Signature · 1 line
2.49.0
#9 lore raw

PATCH v6 9/13 scripts: generate_rust_analyzer.py: identify crates explicitly

In reply to Tamir Duberstein

Use the return of append_crate to declare dependency on that crate. This allows multiple crates with the same display_name be defined, which we'll use to define host crates separately from target crates.

---
 scripts/generate_rust_analyzer.py | 88 +++++++++++++++++++--------------------
 1 file changed, 43 insertions(+), 45 deletions(-)
scripts/generate_rust_analyzer.py +43 -45
index bc23cf58e7a1..b281e87cfa30 100755
@@ -60,17 +60,14 @@ def generate_crates(
             line = line.replace("\n", "")
             cfg.append(line)
 
-    # Now fill the crates list -- dependencies need to come first.
-    #
-    # Avoid O(n^2) iterations by keeping a map of indexes.
+    # Now fill the crates list.
     crates: List[Crate] = []
-    crates_indexes: Dict[str, int] = {}
 
     def build_crate(
         display_name: str,
         root_module: pathlib.Path,
         *,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str],
         is_workspace_member: bool,
     ) -> Crate:
@@ -78,7 +75,7 @@ def generate_crates(
             "display_name": display_name,
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
-            "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps],
+            "deps": deps,
             "cfg": cfg,
             "edition": "2021",
             "env": {
@@ -86,18 +83,19 @@ def generate_crates(
             },
         }
 
-    def register_crate(crate: Crate) -> None:
-        crates_indexes[crate["display_name"]] = len(crates)
+    def register_crate(crate: Crate) -> Dependency:
+        index = len(crates)
         crates.append(crate)
+        return {"crate": index, "name": crate["display_name"]}
 
     def append_crate(
         display_name: str,
         root_module: pathlib.Path,
         *,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str],
-    ) -> None:
-        register_crate(
+    ) -> Dependency:
+        return register_crate(
             build_crate(
                 display_name,
                 root_module,
@@ -111,9 +109,9 @@ def generate_crates(
         display_name: str,
         root_module: pathlib.Path,
         *,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str],
-    ) -> None:
+    ) -> Dependency:
         crate = build_crate(
             display_name,
             root_module,
@@ -143,15 +141,15 @@ def generate_crates(
             "is_proc_macro": True,
             "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name),
         }
-        register_crate(proc_macro_crate)
+        return register_crate(proc_macro_crate)
 
     def append_sysroot_crate(
         display_name: str,
         *,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str],
-    ) -> None:
-        register_crate(
+    ) -> Dependency:
+        return register_crate(
             build_crate(
                 display_name,
                 sysroot_src / display_name / "src" / "lib.rs",
@@ -164,59 +162,59 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
-    append_sysroot_crate("core", deps=[], cfg=crates_cfgs["core"])
-    append_sysroot_crate("alloc", deps=["core"], cfg=[])
-    append_sysroot_crate("std", deps=["alloc", "core"], cfg=[])
-    append_sysroot_crate("proc_macro", deps=["core", "std"], cfg=[])
+    core = append_sysroot_crate("core", deps=[], cfg=crates_cfgs["core"])
+    alloc = append_sysroot_crate("alloc", deps=[core], cfg=[])
+    std = append_sysroot_crate("std", deps=[alloc, core], cfg=[])
+    proc_macro = append_sysroot_crate("proc_macro", deps=[core, std], cfg=[])
 
-    append_crate(
+    compiler_builtins = append_crate(
         "compiler_builtins",
         srctree / "rust" / "compiler_builtins.rs",
         deps=[],
         cfg=[],
     )
 
-    append_proc_macro_crate(
+    macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        deps=["std", "proc_macro"],
+        deps=[std, proc_macro],
         cfg=[],
     )
 
-    append_crate(
+    build_error = append_crate(
         "build_error",
         srctree / "rust" / "build_error.rs",
-        deps=["core", "compiler_builtins"],
+        deps=[core, compiler_builtins],
         cfg=[],
     )
 
-    append_proc_macro_crate(
+    pin_init_internal = append_proc_macro_crate(
         "pin_init_internal",
         srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs",
         deps=[],
         cfg=["kernel"],
     )
 
-    append_crate(
+    pin_init = append_crate(
         "pin_init",
         srctree / "rust" / "pin-init" / "src" / "lib.rs",
-        deps=["core", "pin_init_internal", "macros"],
+        deps=[core, pin_init_internal, macros],
         cfg=["kernel"],
     )
 
-    append_crate(
+    ffi = append_crate(
         "ffi",
         srctree / "rust" / "ffi.rs",
-        deps=["core", "compiler_builtins"],
+        deps=[core, compiler_builtins],
         cfg=[],
     )
 
     def append_crate_with_generated(
         display_name: str,
         *,
-        deps: List[str],
+        deps: List[Dependency],
         cfg: List[str],
-    ) -> None:
+    ) -> Dependency:
         crate = build_crate(
             display_name,
             srctree / "rust" / display_name / "lib.rs",
@@ -235,20 +233,20 @@ def generate_crates(
                 "exclude_dirs": [],
             }
         }
-        register_crate(crate_with_generated)
+        return register_crate(crate_with_generated)
 
-    append_crate_with_generated("bindings", deps=["core", "ffi"], cfg=[])
-    append_crate_with_generated("uapi", deps=["core", "ffi"], cfg=[])
-    append_crate_with_generated(
+    bindings = append_crate_with_generated("bindings", deps=[core, ffi], cfg=[])
+    uapi = append_crate_with_generated("uapi", deps=[core, ffi], cfg=[])
+    kernel = append_crate_with_generated(
         "kernel",
         deps=[
-            "core",
-            "macros",
-            "build_error",
-            "pin_init",
-            "ffi",
-            "bindings",
-            "uapi",
+            core,
+            macros,
+            build_error,
+            pin_init,
+            ffi,
+            bindings,
+            uapi,
         ],
         cfg=[],
     )
@@ -279,7 +277,7 @@ def generate_crates(
             append_crate(
                 name,
                 path,
-                deps=["core", "kernel"],
+                deps=[core, kernel],
                 cfg=cfg,
             )
 
Signature · 1 line
2.49.0
#10 lore raw

PATCH v6 10/13 scripts: generate_rust_analyzer.py: define host crates

In reply to Tamir Duberstein

Define host crates used by the macros crate separately from target crates, now that we can uniquely identify crates with the same name.

This avoids rust-analyzer thinking the host core crate has our target configs applied to it.

---
 scripts/generate_rust_analyzer.py | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
scripts/generate_rust_analyzer.py +10 -4
index b281e87cfa30..0e3bd64f857f 100755
@@ -162,10 +162,16 @@ def generate_crates(
     # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
     # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth
     # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`.
+    host_core = append_sysroot_crate("core", deps=[], cfg=[])
+    host_alloc = append_sysroot_crate("alloc", deps=[host_core], cfg=[])
+    host_std = append_sysroot_crate("std", deps=[host_alloc, host_core], cfg=[])
+    host_proc_macro = append_sysroot_crate(
+        "proc_macro",
+        deps=[host_core, host_std],
+        cfg=[],
+    )
+
     core = append_sysroot_crate("core", deps=[], cfg=crates_cfgs["core"])
-    alloc = append_sysroot_crate("alloc", deps=[core], cfg=[])
-    std = append_sysroot_crate("std", deps=[alloc, core], cfg=[])
-    proc_macro = append_sysroot_crate("proc_macro", deps=[core, std], cfg=[])
 
     compiler_builtins = append_crate(
         "compiler_builtins",
@@ -177,7 +183,7 @@ def generate_crates(
     macros = append_proc_macro_crate(
         "macros",
         srctree / "rust" / "macros" / "lib.rs",
-        deps=[std, proc_macro],
+        deps=[host_std, host_proc_macro],
         cfg=[],
     )
 
Signature · 1 line
2.49.0
#11 lore raw

PATCH v6 11/13 scripts: generate_rust_analyzer.py: avoid FD leak

In reply to Tamir Duberstein

Use a context manager to avoid leaking file descriptors.

---
 scripts/generate_rust_analyzer.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +2 -1
index 0e3bd64f857f..93b96b219a42 100755
@@ -259,7 +259,8 @@ def generate_crates(
 
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
-            return f"{target}.o" in open(build_file).read()
+            with open(build_file) as f:
+                return f"{target}.o" in f.read()
         except FileNotFoundError:
             return False
 
Signature · 1 line
2.49.0
#12 lore raw

PATCH v6 12/13 scripts: generate_rust_analyzer.py: define scripts

In reply to Tamir Duberstein

Generate rust-project.json entries for scripts written in Rust. This is possible now that we have a definition for std built for the host.

Use Pathlib.path.stem for consistency.

---
 scripts/generate_rust_analyzer.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +14 -1
index 93b96b219a42..5f04e3785854 100755
@@ -257,6 +257,19 @@ def generate_crates(
         cfg=[],
     )
 
+    scripts = srctree / "scripts"
+    makefile = (scripts / "Makefile").read_text()
+    for path in scripts.glob("*.rs"):
+        name = path.stem
+        if f"{name}-rust" not in makefile:
+            continue
+        _script = append_crate(
+            name,
+            path,
+            deps=[host_std],
+            cfg=[],
+        )
+
     def is_root_crate(build_file: pathlib.Path, target: str) -> bool:
         try:
             with open(build_file) as f:
@@ -273,7 +286,7 @@ def generate_crates(
     for folder in extra_dirs:
         for path in folder.rglob("*.rs"):
             logging.info("Checking %s", path)
-            name = path.name.replace(".rs", "")
+            name = path.stem
 
             # Skip those that are not crate roots.
             if not is_root_crate(path.parent / "Makefile", name) and \
Signature · 1 line
2.49.0
#13 lore raw

PATCH v6 13/13 scripts: generate_rust_analyzer.py: use `cfg_groups`

In reply to Tamir Duberstein

Declare common cfgs just once to reduce the size of rust-analyzer.json from 30619 to 2624 lines.

---
 scripts/generate_rust_analyzer.py | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
scripts/generate_rust_analyzer.py +12 -9
index 5f04e3785854..aa79b56ff6f4 100755
@@ -29,6 +29,7 @@ class Crate(TypedDict):
     root_module: str
     is_workspace_member: bool
     deps: List[Dependency]
+    cfg_groups: List[str]
     cfg: List[str]
     edition: Literal["2021"]
     env: Dict[str, str]
@@ -51,15 +52,8 @@ def generate_crates(
     sysroot_src: pathlib.Path,
     external_src: pathlib.Path,
     crates_cfgs: DefaultDict[str, List[str]],
+    cfg_groups: List[str],
 ) -> List[Crate]:
-    # Generate the configuration list.
-    cfg = []
-    with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
-        for line in fd:
-            line = line.replace("--cfg=", "")
-            line = line.replace("\n", "")
-            cfg.append(line)
-
     # Now fill the crates list.
     crates: List[Crate] = []
 
@@ -76,6 +70,9 @@ def generate_crates(
             "root_module": str(root_module),
             "is_workspace_member": is_workspace_member,
             "deps": deps,
+            # `cfg_groups` contain the configs generated by the kernel build system; upstream
+            # (sysroot) crates are not aware of them.
+            "cfg_groups": cfg_groups if is_workspace_member else [],
             "cfg": cfg,
             "edition": "2021",
             "env": {
@@ -298,7 +295,7 @@ def generate_crates(
                 name,
                 path,
                 deps=[core, kernel],
-                cfg=cfg,
+                cfg=[],
             )
 
     return crates
@@ -338,6 +335,10 @@ def main() -> None:
         for crate, _, vals in (cfg.partition("=") for cfg in args.cfgs)
     }
 
+    # Generate the configuration list.
+    with open(args.objtree / "include" / "generated" / "rustc_cfg") as fd:
+        cfg_groups = {"rustc_cfg": [line.lstrip("--cfg=").rstrip("\n") for line in fd]}
+
     rust_project = {
         "crates": generate_crates(
             args.srctree,
@@ -345,8 +346,10 @@ def main() -> None:
             args.sysroot_src,
             args.exttree,
             defaultdict(list, crates_cfg),
+            list(cfg_groups.keys()),
         ),
         "sysroot": str(args.sysroot),
+        "cfg_groups": cfg_groups,
     }
 
     json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
Signature · 1 line
2.49.0