rust: generate_rust_analyzer.py: define host crates
Subsystems (2)
v2 cover rust: generate_rust_analyzer.py: define host crates
This series updates rust-project.json to differentiate between host and target crates, where the former are used as dependencies of the macros crate. Please see individual commit messages for details.
The first 3 commits contain mechanical formatting changes and are optional. The series can be taken without them.
I avoided more significant formatting or changes where possible to reduce the diff. Unfortunately scripts/generate_rust_analyzer.py is not consistently formatted before nor after this series.
The 5th commit ("scripts: generate_rust_analyzer.py: use str(pathlib.Path)") can also be considered optional. It removes an inconsistency I noticed while working on this series and which occurs on a line which churns in this series anyway.
---
Changes in v2:
- Rebased on "rust: fix rust-analyzer configuration for generated files" [1]
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Tamir Duberstein (7):
scripts: generate_rust_analyzer.py: add missing whitespace
scripts: generate_rust_analyzer.py: use double quotes
scripts: generate_rust_analyzer.py: add trailing comma
scripts: generate_rust_analyzer.py: add type hints
scripts: generate_rust_analyzer.py: use str(pathlib.Path)
scripts: generate_rust_analyzer.py: identify crates explicitly
scripts: generate_rust_analyzer.py: define host crates
scripts/generate_rust_analyzer.py | 177 ++++++++++++++++++++++++++------------ 1 file changed, 121 insertions(+), 56 deletions(-)
---
base-commit: 510570138476387532129cba425ea7a375308d87
change-id: 20250209-rust-analyzer-host-43b108655578
Best regards,
Signature · 1 line
Tamir Duberstein <[email protected]>
PATCH v2 1/7 scripts: generate_rust_analyzer.py: add missing whitespace
Add a space before the / operator for consistency with surrounding code and code formatting tools. Add a second newline between top-level items in accordance with PEP 8[1]:
Quoted 1 line
Surround top-level function and class definitions with two blank
lines.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +2 -1
@@ -103,7 +103,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ): append_crate( display_name, - srctree / "rust"/ display_name / "lib.rs", + srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, )
@@ -178,5 +178,6 @@ def main(): json.dump(rust_project, sys.stdout, sort_keys=True, indent=4) + if __name__ == "__main__": main()
Signature · 1 line
2.48.1
Re: [PATCH v2 1/7] scripts: generate_rust_analyzer.py: add missing whitespace
Quoted 11 lines · Tamir Duberstein
Add a space before the
/operator for consistency with surrounding code and code formatting tools. Add a second newline between top-level items in accordance with PEP 8[1]:Quoted 1 line
Surround top-level function and class definitions with two blank
lines.
This change was made by a code formatting tool.
Link https://peps.python.org/pep-0008/ [1]Signed-off-by Tamir Duberstein <[email protected]>
Quoted 24 lines
--- scripts/generate_rust_analyzer.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)scripts/generate_rust_analyzer.py +2
-1@@ -103,7 +103,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ): append_crate( display_name, - srctree / "rust"/ display_name / "lib.rs", + srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, )@@ -178,5 +178,6 @@ def main(): json.dump(rust_project, sys.stdout, sort_keys=True, indent=4) + if __name__ == "__main__": main()
PATCH v2 2/7 scripts: generate_rust_analyzer.py: use double quotes
In reply to Tamir DubersteinReplace inconsistent use of single quotes with double quotes.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/generate_rust_analyzer.py +2 -2
@@ -154,8 +154,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): def main(): parser = argparse.ArgumentParser() - parser.add_argument('--verbose', '-v', action='store_true') - parser.add_argument('--cfgs', action='append', default=[]) + parser.add_argument("--verbose", "-v", action="store_true") + parser.add_argument("--cfgs", action="append", default=[]) parser.add_argument("srctree", type=pathlib.Path) parser.add_argument("objtree", type=pathlib.Path) parser.add_argument("sysroot", type=pathlib.Path)
Signature · 1 line
2.48.1
Re: [PATCH v2 2/7] scripts: generate_rust_analyzer.py: use double quotes
Quoted 5 lines · Tamir Duberstein
Replace inconsistent use of single quotes with double quotes.
This change was made by a code formatting tool.
Signed-off-by Tamir Duberstein <[email protected]>
Quoted 19 lines
--- scripts/generate_rust_analyzer.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)scripts/generate_rust_analyzer.py +2
-2@@ -154,8 +154,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): def main(): parser = argparse.ArgumentParser() - parser.add_argument('--verbose', '-v', action='store_true') - parser.add_argument('--cfgs', action='append', default=[]) + parser.add_argument("--verbose", "-v", action="store_true") + parser.add_argument("--cfgs", action="append", default=[]) parser.add_argument("srctree", type=pathlib.Path) parser.add_argument("objtree", type=pathlib.Path) parser.add_argument("sysroot", type=pathlib.Path)
PATCH v2 3/7 scripts: generate_rust_analyzer.py: add trailing comma
In reply to Tamir DubersteinAdd missing trailing comma on multi-line function call as suggested by PEP-8:
Quoted 3 lines
The pattern is to put each value (etc.) on a line by itself, always adding a trailing comma, and add the close parenthesis/bracket/brace on the next line.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
@@ -165,7 +165,7 @@ def main(): logging.basicConfig( format="[%(asctime)s] [%(levelname)s] %(message)s", - level=logging.INFO if args.verbose else logging.WARNING + level=logging.INFO if args.verbose else logging.WARNING, ) # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
Signature · 1 line
2.48.1
Re: [PATCH v2 3/7] scripts: generate_rust_analyzer.py: add trailing comma
Quoted 10 lines · Tamir Duberstein
Add missing trailing comma on multi-line function call as suggested by PEP-8:
Quoted 3 lines
The pattern is to put each value (etc.) on a line by itself, always adding a trailing comma, and add the close parenthesis/bracket/brace on the next line.
This change was made by a code formatting tool.
Signed-off-by Tamir Duberstein <[email protected]>
Quoted 17 lines
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/generate_rust_analyzer.py +1
-1@@ -165,7 +165,7 @@ def main(): logging.basicConfig( format="[%(asctime)s] [%(levelname)s] %(message)s", - level=logging.INFO if args.verbose else logging.WARNING + level=logging.INFO if args.verbose else logging.WARNING, ) # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
PATCH v2 4/7 scripts: generate_rust_analyzer.py: add type hints
In reply to Tamir DubersteinPython type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests").
Run uv tool run mypy --strict scripts/generate_rust_analyzer.py to verify.
This removes "is_proc_macro": false from rust-project.json in exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.
--- scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 34 deletions(-)
scripts/generate_rust_analyzer.py +96 -34
@@ -10,8 +10,10 @@ import os import pathlib import subprocess import sys +import typing as T -def args_crates_cfgs(cfgs): + +def args_crates_cfgs(cfgs: T.Iterable[str]) -> dict[str, list[str]]: crates_cfgs = {} for cfg in cfgs: crate, vals = cfg.split("=", 1)
@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs): return crates_cfgs -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): + +class Dependency(T.TypedDict): + crate: int + name: str + + +class Source(T.TypedDict): + include_dirs: list[str] + exclude_dirs: list[str] + + +class Crate(T.TypedDict): + display_name: str + root_module: str + is_workspace_member: bool + deps: list[Dependency] + cfg: list[str] + edition: T.Literal["2021"] + env: dict[str, str] + + +# `T.NotRequired` fields on `Crate` would be better but `T.NotRequired` was added in 3.11. +class ProcMacroCrate(Crate): + is_proc_macro: T.Literal[True] + proc_macro_dylib_path: T.Optional[str] # `pathlib.Path` is not JSON serializable. + + +# `T.NotRequired` fields on `Crate` would be better but `T.NotRequired` was added in 3.11. +class CrateWithGenerated(Crate): + source: T.Optional[Source] + + +def generate_crates( + srctree: pathlib.Path, + objtree: pathlib.Path, + sysroot_src: pathlib.Path, + external_src: pathlib.Path, + cfgs: list[str], +) -> list[Crate]: # Generate the configuration list. cfg = [] with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -31,37 +71,53 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Now fill the crates list -- dependencies need to come first. # # Avoid O(n^2) iterations by keeping a map of indexes. - crates = [] - crates_indexes = {} + crates: list[Crate] = [] + crates_indexes: dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) - def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False): - crate = { + def append_crate( + display_name: str, + root_module: pathlib.Path, + deps: list[str], + cfg: list[str] = [], + is_workspace_member: bool = True, + ) -> None: + crates_indexes[display_name] = len(crates) + crates.append({ "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "is_proc_macro": is_proc_macro, "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" - } + }, + }) + + def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: list[str], + cfg: list[str] = [], + ) -> None: + append_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output( + [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], + stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + crate: ProcMacroCrate = { + **crates[-1], + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", } - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) - crates.append(crate) + crates[-1] = crate def append_sysroot_crate( - display_name, - deps, - cfg=[], - ): + display_name: str, + deps: list[str], + cfg: list[str] = [], + ) -> None: append_crate( display_name, sysroot_src / display_name / "src" / "lib.rs",
@@ -84,11 +140,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): [], ) - append_crate( + append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", ["std", "proc_macro"], - is_proc_macro=True, ) append_crate(
@@ -98,9 +153,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ) def append_crate_with_generated( - display_name, - deps, - ): + display_name: str, + deps: list[str], + ) -> None: append_crate( display_name, srctree / "rust" / display_name / "lib.rs",
@@ -108,19 +163,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): cfg=cfg, ) crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True)) - crates[-1]["source"] = { - "include_dirs": [ - str(srctree / "rust" / display_name), - str(objtree / "rust") - ], - "exclude_dirs": [], + crate: CrateWithGenerated = { + **crates[-1], + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } + crates[-1] = crate append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"]) append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"]) - def is_root_crate(build_file, target): + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: return f"{target}.o" in open(build_file).read() except FileNotFoundError:
@@ -129,7 +188,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs: T.Iterable[pathlib.Path] = map( + lambda dir: srctree / dir, ("samples", "drivers") + ) if external_src is not None: extra_dirs = [external_src] for folder in extra_dirs:
@@ -152,7 +213,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): return crates -def main(): + +def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--verbose", "-v", action="store_true") parser.add_argument("--cfgs", action="append", default=[])
Signature · 1 line
2.48.1
Re: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints
Quoted 11 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").Run
uv tool run mypy --strict scripts/generate_rust_analyzer.pyto verify.
Quoted 6 lines
This removes
"is_proc_macro": falsefromrust-project.jsonin exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.
Gentle ping. Please have a look if you aren't intimidated by python type hints :)
Trevor Gross
reviewerRe: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints
In reply to Tamir DubersteinQuoted 10 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").Run
uv tool run mypy --strict scripts/generate_rust_analyzer.pyto verify.
From the discussion, it may be better to instead mention the direct invocation (without uv).
Could you also mention the target min version? Since apparently the kernel has a spread. It looks like maybe 3.8 based on what is used here.
Quoted 15 lines
This removes
"is_proc_macro": falsefromrust-project.jsonin exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.Signed-off-by Tamir Duberstein <[email protected]>--- scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 34 deletions(-)scripts/generate_rust_analyzer.py +0
-0
scripts/generate_rust_analyzer.py import os · line 13
Quoted 4 lines
scripts/generate_rust_analyzer.py +1
-0@@ -10,8 +10,10 @@ import os import pathlib import subprocess import sys +import typing as T
Nit: is there any need to keep everything namespaced? I think it should be fine to import Iterable TypedDict etc directly since they aren't confusable.
Same for pathlib.Path since there is no other Path (some of that is preexisting).
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 106
Quoted 9 lines
scripts/generate_rust_analyzer.py +9
-0+ def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: list[str], + cfg: list[str] = [], + ) -> None: + append_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output( + [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
Nit, may as well use this opportunity to wrap the line.
Quoted 6 lines
scripts/generate_rust_analyzer.py +6
-0+ stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + crate: ProcMacroCrate = { + **crates[-1], + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 114
Quoted 10 lines
scripts/generate_rust_analyzer.py +1
-8} - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) - crates.append(crate) + crates[-1] = crate
The unpacking is a bit confusing here, can crates[-1] just be set rather than duplicating and replacing it?
Maybe the body of append_crate should be build_crate(...) -> Crate (which could then be a top-level function), then append_crate, append_crate_with_generated, etc call that and handle modification / appending themselves.
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 176
Quoted 11 lines
scripts/generate_rust_analyzer.py +10
-0+ crate: CrateWithGenerated = { + **crates[-1], + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } + crates[-1] = crate
Same note as above regarding rebuilding the last item.
- Trevor
Re: [PATCH v2 4/7] scripts: generate_rust_analyzer.py: add type hints
Quoted 10 lines · Trevor Gross
Quoted 8 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").
Quoted 5 lines
Quoted 2 lines
Run
uv tool run mypy --strict scripts/generate_rust_analyzer.pyto verify.From the discussion, it may be better to instead mention the direct invocation (without uv).
👍 done.
Quoted 3 lines
Could you also mention the target min version? Since apparently the kernel has a spread. It looks like maybe 3.8 based on what is used here.
Added, it is now 3.8 because mypy doesn't support anything lower.
Quoted 11 lines
Quoted 11 lines
This removes
"is_proc_macro": falsefromrust-project.jsonin exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.Signed-off-by Tamir Duberstein <[email protected]>--- scripts/generate_rust_analyzer.py | 130 ++++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 34 deletions(-)
Quoted 13 lines
scripts/generate_rust_analyzer.py import os · line 13
Quoted 9 lines
scripts/generate_rust_analyzer.py +1
-0@@ -10,8 +10,10 @@ import os import pathlib import subprocess import sys +import typing as TNit: is there any need to keep everything namespaced? I think it should be fine to import
IterableTypedDictetc directly since they aren't confusable.
I saw this style in another project and liked it, but I don't feel strongly. Changed.
Quoted 3 lines
Same for
pathlib.Pathsince there is no otherPath(some of that is preexisting).
I'll leave this one as is to avoid more churn.
Quoted 11 lines
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 106
Quoted 10 lines
scripts/generate_rust_analyzer.py +9
-0+ def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: list[str], + cfg: list[str] = [], + ) -> None: + append_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output( + [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
Quoted 1 line
Nit, may as well use this opportunity to wrap the line.
I considered that, but when you use git with -w this line doesn't change, so I'd rather leave it.
Quoted 18 lines
Quoted 7 lines
scripts/generate_rust_analyzer.py +6
-0+ stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + crate: ProcMacroCrate = { + **crates[-1], + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}",scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 114
Quoted 10 lines
scripts/generate_rust_analyzer.py +1
-8} - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) - crates.append(crate) + crates[-1] = crate
Quoted 7 lines
The unpacking is a bit confusing here, can
crates[-1]just be set rather than duplicating and replacing it?Maybe the body of
append_crateshould bebuild_crate(...) -> Crate(which could then be a top-level function), thenappend_crate,append_crate_with_generated, etc call that and handle modification / appending themselves.
This is a nice improvement. Done.
Quoted 11 lines
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 176
Quoted 11 lines
scripts/generate_rust_analyzer.py +10
-0+ crate: CrateWithGenerated = { + **crates[-1], + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } + crates[-1] = crate
👍
Quoted 2 lines
- Trevor
Thanks for the review!
PATCH v2 5/7 scripts: generate_rust_analyzer.py: use str(pathlib.Path)
In reply to Tamir DubersteinUse the / operator on pathlib.Path rather than directly crafting a string. This is consistent with all other path manipulation in this script.
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
@@ -109,7 +109,7 @@ def generate_crates( crate: ProcMacroCrate = { **crates[-1], "is_proc_macro": True, - "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } crates[-1] = crate
Signature · 1 line
2.48.1
Re: [PATCH v2 5/7] scripts: generate_rust_analyzer.py: use str(pathlib.Path)
Quoted 5 lines · Tamir Duberstein
Use the
/operator onpathlib.Pathrather than directly crafting a string. This is consistent with all other path manipulation in this script.Signed-off-by Tamir Duberstein <[email protected]>
Quoted 16 lines
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)scripts/generate_rust_analyzer.py +1
-1@@ -109,7 +109,7 @@ def generate_crates( crate: ProcMacroCrate = { **crates[-1], "is_proc_macro": True, - "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } crates[-1] = crate
PATCH v2 6/7 scripts: generate_rust_analyzer.py: identify crates explicitly
In reply to Tamir DubersteinUse the return of append_crate to declare dependency on that crate. This allows multiple crates with the same display_name be defined, which we'll use to define host crates separately from target crates.
--- scripts/generate_rust_analyzer.py | 60 +++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 30 deletions(-)
scripts/generate_rust_analyzer.py +30 -30
@@ -68,40 +68,38 @@ def generate_crates( line = line.replace("\n", "") cfg.append(line) - # Now fill the crates list -- dependencies need to come first. - # - # Avoid O(n^2) iterations by keeping a map of indexes. + # Now fill the crates list. crates: list[Crate] = [] - crates_indexes: dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def append_crate( display_name: str, root_module: pathlib.Path, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], is_workspace_member: bool = True, - ) -> None: - crates_indexes[display_name] = len(crates) + ) -> Dependency: + index = len(crates) crates.append({ "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], + "deps": deps, "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" }, }) + return {"crate": index, "name": display_name} def append_proc_macro_crate( display_name: str, root_module: pathlib.Path, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], - ) -> None: - append_crate(display_name, root_module, deps, cfg) + ) -> Dependency: + dep = append_crate(display_name, root_module, deps, cfg) proc_macro_dylib_name = subprocess.check_output( [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], stdin=subprocess.DEVNULL,
@@ -112,13 +110,14 @@ def generate_crates( "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } crates[-1] = crate + return dep def append_sysroot_crate( display_name: str, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], - ) -> None: - append_crate( + ) -> Dependency: + return append_crate( display_name, sysroot_src / display_name / "src" / "lib.rs", deps,
@@ -129,34 +128,34 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. - append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - append_sysroot_crate("alloc", ["core"]) - append_sysroot_crate("std", ["alloc", "core"]) - append_sysroot_crate("proc_macro", ["core", "std"]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) + alloc = append_sysroot_crate("alloc", [core]) + std = append_sysroot_crate("std", [alloc, core]) + proc_macro = append_sysroot_crate("proc_macro", [core, std]) - append_crate( + compiler_builtins = append_crate( "compiler_builtins", srctree / "rust" / "compiler_builtins.rs", [], ) - append_proc_macro_crate( + macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - ["std", "proc_macro"], + [std, proc_macro], ) - append_crate( + build_error = append_crate( "build_error", srctree / "rust" / "build_error.rs", - ["core", "compiler_builtins"], + [core, compiler_builtins], ) def append_crate_with_generated( display_name: str, - deps: list[str], - ) -> None: - append_crate( + deps: list[Dependency], + ) -> Dependency: + dep = append_crate( display_name, srctree / "rust" / display_name / "lib.rs", deps,
@@ -174,10 +173,11 @@ def generate_crates( } } crates[-1] = crate + return dep - append_crate_with_generated("bindings", ["core"]) - append_crate_with_generated("uapi", ["core"]) - append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"]) + bindings = append_crate_with_generated("bindings", [core]) + uapi = append_crate_with_generated("uapi", [core]) + kernel = append_crate_with_generated("kernel", [bindings, build_error, core, macros, uapi]) def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try:
@@ -207,7 +207,7 @@ def generate_crates( append_crate( name, path, - ["core", "kernel"], + [core, kernel], cfg=cfg, )
Signature · 1 line
2.48.1
Re: [PATCH v2 6/7] scripts: generate_rust_analyzer.py: identify crates explicitly
Quoted 5 lines · Tamir Duberstein
Use the return of
append_crateto declare dependency on that crate. This allows multiple crates with the samedisplay_namebe defined, which we'll use to define host crates separately from target crates.Signed-off-by Tamir Duberstein <[email protected]>
Quoted 146 lines
--- scripts/generate_rust_analyzer.py | 60 +++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 30 deletions(-)scripts/generate_rust_analyzer.py +30
-30@@ -68,40 +68,38 @@ def generate_crates( line = line.replace("\n", "") cfg.append(line) - # Now fill the crates list -- dependencies need to come first. - # - # Avoid O(n^2) iterations by keeping a map of indexes. + # Now fill the crates list. crates: list[Crate] = [] - crates_indexes: dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def append_crate( display_name: str, root_module: pathlib.Path, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], is_workspace_member: bool = True, - ) -> None: - crates_indexes[display_name] = len(crates) + ) -> Dependency: + index = len(crates) crates.append({ "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], + "deps": deps, "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" }, }) + return {"crate": index, "name": display_name} def append_proc_macro_crate( display_name: str, root_module: pathlib.Path, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], - ) -> None: - append_crate(display_name, root_module, deps, cfg) + ) -> Dependency: + dep = append_crate(display_name, root_module, deps, cfg) proc_macro_dylib_name = subprocess.check_output( [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], stdin=subprocess.DEVNULL,@@ -112,13 +110,14 @@ def generate_crates( "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } crates[-1] = crate + return dep def append_sysroot_crate( display_name: str, - deps: list[str], + deps: list[Dependency], cfg: list[str] = [], - ) -> None: - append_crate( + ) -> Dependency: + return append_crate( display_name, sysroot_src / display_name / "src" / "lib.rs", deps,@@ -129,34 +128,34 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. - append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - append_sysroot_crate("alloc", ["core"]) - append_sysroot_crate("std", ["alloc", "core"]) - append_sysroot_crate("proc_macro", ["core", "std"]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) + alloc = append_sysroot_crate("alloc", [core]) + std = append_sysroot_crate("std", [alloc, core]) + proc_macro = append_sysroot_crate("proc_macro", [core, std]) - append_crate( + compiler_builtins = append_crate( "compiler_builtins", srctree / "rust" / "compiler_builtins.rs", [], ) - append_proc_macro_crate( + macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - ["std", "proc_macro"], + [std, proc_macro], ) - append_crate( + build_error = append_crate( "build_error", srctree / "rust" / "build_error.rs", - ["core", "compiler_builtins"], + [core, compiler_builtins], ) def append_crate_with_generated( display_name: str, - deps: list[str], - ) -> None: - append_crate( + deps: list[Dependency], + ) -> Dependency: + dep = append_crate( display_name, srctree / "rust" / display_name / "lib.rs", deps,@@ -174,10 +173,11 @@ def generate_crates( } } crates[-1] = crate + return dep - append_crate_with_generated("bindings", ["core"]) - append_crate_with_generated("uapi", ["core"]) - append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "uapi"]) + bindings = append_crate_with_generated("bindings", [core]) + uapi = append_crate_with_generated("uapi", [core]) + kernel = append_crate_with_generated("kernel", [bindings, build_error, core, macros, uapi]) def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try:@@ -207,7 +207,7 @@ def generate_crates( append_crate( name, path, - ["core", "kernel"], + [core, kernel], cfg=cfg, )
PATCH v2 7/7 scripts: generate_rust_analyzer.py: define host crates
In reply to Tamir DubersteinDefine host crates used by the macros crate separately from target crates, now that we can uniquely identify crates with the same name.
This avoids rust-analyzer thinking the host core crate has our target configs applied to it.
--- scripts/generate_rust_analyzer.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/generate_rust_analyzer.py +6 -4
@@ -128,10 +128,12 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. + host_core = append_sysroot_crate("core", []) + host_alloc = append_sysroot_crate("alloc", [host_core]) + host_std = append_sysroot_crate("std", [host_alloc, host_core]) + host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - alloc = append_sysroot_crate("alloc", [core]) - std = append_sysroot_crate("std", [alloc, core]) - proc_macro = append_sysroot_crate("proc_macro", [core, std]) compiler_builtins = append_crate( "compiler_builtins",
@@ -142,7 +144,7 @@ def generate_crates( macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - [std, proc_macro], + [host_std, host_proc_macro], ) build_error = append_crate(
Signature · 1 line
2.48.1
Re: [PATCH v2 7/7] scripts: generate_rust_analyzer.py: define host crates
Quoted 9 lines · Tamir Duberstein
Define host crates used by the
macroscrate separately from target crates, now that we can uniquely identify crates with the same name.This avoids rust-analyzer thinking the host
corecrate has our target configs applied to it.
Quoted 33 lines
--- scripts/generate_rust_analyzer.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)scripts/generate_rust_analyzer.py +6
-4@@ -128,10 +128,12 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. + host_core = append_sysroot_crate("core", []) + host_alloc = append_sysroot_crate("alloc", [host_core]) + host_std = append_sysroot_crate("std", [host_alloc, host_core]) + host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - alloc = append_sysroot_crate("alloc", [core]) - std = append_sysroot_crate("std", [alloc, core]) - proc_macro = append_sysroot_crate("proc_macro", [core, std]) compiler_builtins = append_crate( "compiler_builtins",@@ -142,7 +144,7 @@ def generate_crates( macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - [std, proc_macro], + [host_std, host_proc_macro], ) build_error = append_crate(