rust: generate_rust_analyzer.py: define host crates and scripts
Subsystems (2)
v4 cover rust: generate_rust_analyzer.py: define host crates and scripts
This series updates rust-project.json to differentiate between host and target crates, where the former are used as dependencies of the macros crate. Please see individual commit messages for details.
The first 3 commits contain mechanical formatting changes and are optional. The series can be taken without them.
I avoided more significant formatting or changes where possible to reduce the diff. Unfortunately scripts/generate_rust_analyzer.py is not consistently formatted before nor after this series.
The 5th commit ("scripts: generate_rust_analyzer.py: use str(pathlib.Path)") can also be considered optional. It removes an inconsistency I noticed while working on this series and which occurs on a line which churns in this series anyway.
The last 3 commits can also be considered optional, as they can be submitted separately. I included them in this series because they depend on it, but they can be split out if this is preferred.
---
Changes in v4:
- Fix typo (s/generate/generated/).
- Pull Trevor's suggested change into a separate patch with a
Suggested-by tag.
- Add patch to avoid file descriptor leak.
- Add patch to generate rust-project.json entries for scripts/*.rs.
- Add patch to use cfg_groups to reduce size of rust-project.json by
90%.
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Rebase on linux-next. This is needed to pick up all the conflicts from
both rust-next and rust-fixes.
- Drop uv from mypy command. (Trevor Gross)
- Add --python-version 3.8 to mypy command. (Trevor Gross)
- from typings import ... directly. (Trevor Gross)
- Extract build_crate and register_crate to avoid peeking into
crates[-1]. (Trevor Gross)
- Link to v2: https://lore.kernel.org/r/[email protected]
Changes in v2:
- Rebased on "rust: fix rust-analyzer configuration for generated files" [1]
- Link to v1: https://lore.kernel.org/r/[email protected]
---
Tamir Duberstein (11):
scripts: generate_rust_analyzer.py: add missing whitespace
scripts: generate_rust_analyzer.py: use double quotes
scripts: generate_rust_analyzer.py: add trailing comma
scripts: generate_rust_analyzer.py: extract {build,register}_crate
scripts: generate_rust_analyzer.py: add type hints
scripts: generate_rust_analyzer.py: use str(pathlib.Path)
scripts: generate_rust_analyzer.py: identify crates explicitly
scripts: generate_rust_analyzer.py: define host crates
scripts: generate_rust_analyzer.py: avoid FD leak
scripts: generate_rust_analyzer.py: define scripts
scripts: generate_rust_analyzer.py: use cfg_groups
scripts/generate_rust_analyzer.py | 257 ++++++++++++++++++++++++++------------ 1 file changed, 180 insertions(+), 77 deletions(-)
---
base-commit: 9388ec571cb1adba59d1cded2300eeb11827679c
change-id: 20250209-rust-analyzer-host-43b108655578
Best regards,
Signature · 1 line
Tamir Duberstein <[email protected]>
PATCH v4 1/11 scripts: generate_rust_analyzer.py: add missing whitespace
Add a space before the / operator for consistency with surrounding code and code formatting tools. Add a second newline between top-level items in accordance with PEP 8[1]:
Quoted 1 line
Surround top-level function and class definitions with two blank
lines.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +2 -1
@@ -118,7 +118,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ): append_crate( display_name, - srctree / "rust"/ display_name / "lib.rs", + srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, )
@@ -193,5 +193,6 @@ def main(): json.dump(rust_project, sys.stdout, sort_keys=True, indent=4) + if __name__ == "__main__": main()
Signature · 1 line
2.48.1
Re: [PATCH v4 01/11] scripts: generate_rust_analyzer.py: add missing whitespace
Quoted 39 lines · Tamir Duberstein
Add a space before the
/operator for consistency with surrounding code and code formatting tools. Add a second newline between top-level items in accordance with PEP 8[1]:Quoted 1 line
Surround top-level function and class definitions with two blank
lines.
This change was made by a code formatting tool.
Link https://peps.python.org/pep-0008/ [1]Reviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)scripts/generate_rust_analyzer.py +2
-1@@ -118,7 +118,7 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ): append_crate( display_name, - srctree / "rust"/ display_name / "lib.rs", + srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, )@@ -193,5 +193,6 @@ def main(): json.dump(rust_project, sys.stdout, sort_keys=True, indent=4) +if
__name__== "__main__":
Quoted 6 lines
main()Signature · 1 line
2.48.1
PATCH v4 2/11 scripts: generate_rust_analyzer.py: use double quotes
In reply to Tamir DubersteinReplace inconsistent use of single quotes with double quotes.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/generate_rust_analyzer.py +2 -2
@@ -169,8 +169,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): def main(): parser = argparse.ArgumentParser() - parser.add_argument('--verbose', '-v', action='store_true') - parser.add_argument('--cfgs', action='append', default=[]) + parser.add_argument("--verbose", "-v", action="store_true") + parser.add_argument("--cfgs", action="append", default=[]) parser.add_argument("srctree", type=pathlib.Path) parser.add_argument("objtree", type=pathlib.Path) parser.add_argument("sysroot", type=pathlib.Path)
Signature · 1 line
2.48.1
Re: [PATCH v4 02/11] scripts: generate_rust_analyzer.py: use double quotes
Quoted 34 lines · Tamir Duberstein
Replace inconsistent use of single quotes with double quotes.
This change was made by a code formatting tool.
Reviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)scripts/generate_rust_analyzer.py +0
-0@@ -169,8 +169,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):def
main():
parser =argparse.ArgumentParser()scripts/generate_rust_analyzer.py +2
-2- parser.add_argument('--verbose', '-v', action='store_true') - parser.add_argument('--cfgs', action='append', default=[]) + parser.add_argument("--verbose", "-v", action="store_true") + parser.add_argument("--cfgs", action="append", default=[]) parser.add_argument("srctree", type=pathlib.Path) parser.add_argument("objtree", type=pathlib.Path) parser.add_argument("sysroot", type=pathlib.Path)Signature · 2 lines
-- 2.48.1
PATCH v4 3/11 scripts: generate_rust_analyzer.py: add trailing comma
In reply to Tamir DubersteinAdd missing trailing comma on multi-line function call as suggested by PEP-8:
Quoted 3 lines
The pattern is to put each value (etc.) on a line by itself, always adding a trailing comma, and add the close parenthesis/bracket/brace on the next line.
This change was made by a code formatting tool.
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
@@ -180,7 +180,7 @@ def main(): logging.basicConfig( format="[%(asctime)s] [%(levelname)s] %(message)s", - level=logging.INFO if args.verbose else logging.WARNING + level=logging.INFO if args.verbose else logging.WARNING, ) # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.
Signature · 1 line
2.48.1
Re: [PATCH v4 03/11] scripts: generate_rust_analyzer.py: add trailing comma
Quoted 37 lines · Tamir Duberstein
Add missing trailing comma on multi-line function call as suggested by PEP-8:
Quoted 3 lines
The pattern is to put each value (etc.) on a line by itself, always adding a trailing comma, and add the close parenthesis/bracket/brace on the next line.
This change was made by a code formatting tool.
Reviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)scripts/generate_rust_analyzer.py +1
-1@@ -180,7 +180,7 @@ def main(): logging.basicConfig( format="[%(asctime)s] [%(levelname)s] %(message)s", - level=logging.INFO if args.verbose else logging.WARNING + level=logging.INFO if args.verbose else logging.WARNING, ) # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain.Signature · 2 lines
-- 2.48.1
PATCH v4 4/11 scripts: generate_rust_analyzer.py: extract `{build,register}_crate`
In reply to Tamir DubersteinExtract helpers from append_crate to avoid the need to peek into crates[-1]. This improves readability.
--- scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)
scripts/generate_rust_analyzer.py +30 -5
@@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): crates_indexes = {} crates_cfgs = args_crates_cfgs(cfgs) - def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False): + def build_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): crate = { "display_name": display_name, "root_module": str(root_module),
@@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): stdin=subprocess.DEVNULL, ).decode('utf-8').strip() crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) + return crate + + def register_crate(crate): + crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) + def append_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): + register_crate( + build_crate( + display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro + ) + ) + def append_sysroot_crate( display_name, deps,
@@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): display_name, deps, ): - append_crate( + crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, ) - crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True)) - crates[-1]["source"] = { + crate["env"]["OBJTREE"] = str(objtree.resolve(True)) + crate["source"] = { "include_dirs": [ str(srctree / "rust" / display_name), str(objtree / "rust") ], "exclude_dirs": [], } + register_crate(crate) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"])
Signature · 1 line
2.48.1
Re: [PATCH v4 04/11] scripts: generate_rust_analyzer.py: extract `{build,register}_crate`
Quoted 91 lines · Tamir Duberstein
Extract helpers from
append_crateto avoid the need to peek intocrates[-1]. This improves readability.Suggested-by Trevor Gross <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 35 ++++++++++++++++++++++++++++++-----1 file changed, 30 insertions(+), 5 deletions(-)scripts/generate_rust_analyzer.py +30
-5@@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): crates_indexes = {} crates_cfgs = args_crates_cfgs(cfgs) - def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False): + def build_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): crate = { "display_name": display_name, "root_module": str(root_module),@@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): stdin=subprocess.DEVNULL, ).decode('utf-8').strip() crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) + return crate + + def register_crate(crate): + crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) + def append_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): + register_crate( + build_crate( + display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro + ) + ) + def append_sysroot_crate( display_name, deps,@@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): display_name, deps, ): - append_crate( + crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, ) - crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True)) - crates[-1]["source"] = { + crate["env"]["OBJTREE"] = str(objtree.resolve(True)) + crate["source"] = { "include_dirs": [ str(srctree / "rust" / display_name), str(objtree / "rust") ], "exclude_dirs": [], } + register_crate(crate) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"])Signature · 2 lines
-- 2.48.1
Re: [PATCH v4 04/11] scripts: generate_rust_analyzer.py: extract `{build,register}_crate`
In reply to Tamir DubersteinQuoted 5 lines · Tamir Duberstein
Extract helpers from
append_crateto avoid the need to peek intocrates[-1]. This improves readability.Suggested-by Trevor Gross <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>
Quoted 77 lines
--- scripts/generate_rust_analyzer.py | 35 ++++++++++++++++++++++++++++++----- 1 file changed, 30 insertions(+), 5 deletions(-)scripts/generate_rust_analyzer.py +30
-5@@ -35,7 +35,14 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): crates_indexes = {} crates_cfgs = args_crates_cfgs(cfgs) - def append_crate(display_name, root_module, deps, cfg=[], is_workspace_member=True, is_proc_macro=False): + def build_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): crate = { "display_name": display_name, "root_module": str(root_module),@@ -54,9 +61,26 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): stdin=subprocess.DEVNULL, ).decode('utf-8').strip() crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - crates_indexes[display_name] = len(crates) + return crate + + def register_crate(crate): + crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) + def append_crate( + display_name, + root_module, + deps, + cfg=[], + is_workspace_member=True, + is_proc_macro=False, + ): + register_crate( + build_crate( + display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro + ) + ) + def append_sysroot_crate( display_name, deps,@@ -116,20 +140,21 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): display_name, deps, ): - append_crate( + crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs", deps, cfg=cfg, ) - crates[-1]["env"]["OBJTREE"] = str(objtree.resolve(True)) - crates[-1]["source"] = { + crate["env"]["OBJTREE"] = str(objtree.resolve(True)) + crate["source"] = { "include_dirs": [ str(srctree / "rust" / display_name), str(objtree / "rust") ], "exclude_dirs": [], } + register_crate(crate) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"])
PATCH v4 5/11 scripts: generate_rust_analyzer.py: add type hints
In reply to Tamir DubersteinPython type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests").
Run mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8 to verify. Note that mypy no longer supports python < 3.8.
This removes "is_proc_macro": false from rust-project.json in exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.
--- scripts/generate_rust_analyzer.py | 172 +++++++++++++++++++++++++------------- 1 file changed, 112 insertions(+), 60 deletions(-)
scripts/generate_rust_analyzer.py +112 -60
@@ -10,8 +10,10 @@ import os import pathlib import subprocess import sys +from typing import Dict, Iterable, List, Literal, Optional, TypedDict -def args_crates_cfgs(cfgs): + +def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]: crates_cfgs = {} for cfg in cfgs: crate, vals = cfg.split("=", 1)
@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs): return crates_cfgs -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): + +class Dependency(TypedDict): + crate: int + name: str + + +class Source(TypedDict): + include_dirs: List[str] + exclude_dirs: List[str] + + +class Crate(TypedDict): + display_name: str + root_module: str + is_workspace_member: bool + deps: List[Dependency] + cfg: List[str] + edition: Literal["2021"] + env: Dict[str, str] + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class ProcMacroCrate(Crate): + is_proc_macro: Literal[True] + proc_macro_dylib_path: Optional[str] # `pathlib.Path` is not JSON serializable. + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class CrateWithGenerated(Crate): + source: Optional[Source] + + +def generate_crates( + srctree: pathlib.Path, + objtree: pathlib.Path, + sysroot_src: pathlib.Path, + external_src: pathlib.Path, + cfgs: List[str], +) -> List[Crate]: # Generate the configuration list. cfg = [] with open(objtree / "include" / "generated" / "rustc_cfg") as fd:
@@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Now fill the crates list -- dependencies need to come first. # # Avoid O(n^2) iterations by keeping a map of indexes. - crates = [] - crates_indexes = {} + crates: List[Crate] = [] + crates_indexes: Dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def build_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): - crate = { + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> Crate: + return { "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "is_proc_macro": is_proc_macro, "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" - } + }, } - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - return crate - - def register_crate(crate): + + def register_crate(crate: Crate) -> None: crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) def append_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> None: register_crate( - build_crate( - display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro - ) + build_crate(display_name, root_module, deps, cfg, is_workspace_member) ) + def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + ) -> None: + crate = build_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output( + [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], + stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + proc_macro_crate: ProcMacroCrate = { + **crate, + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + } + register_crate(proc_macro_crate) + def append_sysroot_crate( - display_name, - deps, - cfg=[], - ): - append_crate( - display_name, - sysroot_src / display_name / "src" / "lib.rs", - deps, - cfg, - is_workspace_member=False, + display_name: str, + deps: List[str], + cfg: List[str] = [], + ) -> None: + register_crate( + build_crate( + display_name, + sysroot_src / display_name / "src" / "lib.rs", + deps, + cfg, + is_workspace_member=False, + ) ) # NB: sysroot crates reexport items from one another so setting up our transitive dependencies
@@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): [], ) - append_crate( + append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", ["std", "proc_macro"], - is_proc_macro=True, ) append_crate(
@@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ["core", "compiler_builtins"], ) - append_crate( + append_proc_macro_crate( "pin_init_internal", srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs", [], cfg=["kernel"], - is_proc_macro=True, ) append_crate(
@@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ) def append_crate_with_generated( - display_name, - deps, - ): + display_name: str, + deps: List[str], + ) -> None: crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs",
@@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): cfg=cfg, ) crate["env"]["OBJTREE"] = str(objtree.resolve(True)) - crate["source"] = { - "include_dirs": [ - str(srctree / "rust" / display_name), - str(objtree / "rust") - ], - "exclude_dirs": [], + crate_with_generated: CrateWithGenerated = { + **crate, + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } - register_crate(crate) + register_crate(crate_with_generated) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"]) append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"]) - def is_root_crate(build_file, target): + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: return f"{target}.o" in open(build_file).read() except FileNotFoundError:
@@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs: Iterable[pathlib.Path] = map( + lambda dir: srctree / dir, ("samples", "drivers") + ) if external_src is not None: extra_dirs = [external_src] for folder in extra_dirs:
@@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): return crates -def main(): + +def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--verbose", "-v", action="store_true") parser.add_argument("--cfgs", action="append", default=[])
Signature · 1 line
2.48.1
Daniel Almeida
reviewerRe: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints
Hi Tamir,
Quoted 9 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").
Quoted 6 lines
Run
mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8to verify. Note thatmypyno longer supports python < 3.8.This removes
"is_proc_macro": falsefromrust-project.jsonin exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.
Can this be a separate patch? Not sure how this is related to Python type hints, but it makes the current patch harder to review.
Quoted 151 lines
Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 172 +++++++++++++++++++++++++-------------1 file changed, 112 insertions(+), 60 deletions(-)scripts/generate_rust_analyzer.py +0
-0@@ -10,8 +10,10 @@ import osimport pathlib
import subprocess
import sysscripts/generate_rust_analyzer.py +69
-33+from typing import Dict, Iterable, List, Literal, Optional, TypedDict -def args_crates_cfgs(cfgs): + +def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]: crates_cfgs = {} for cfg in cfgs: crate, vals = cfg.split("=", 1)@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs): return crates_cfgs -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): + +class Dependency(TypedDict): + crate: int + name: str + + +class Source(TypedDict): + include_dirs: List[str] + exclude_dirs: List[str] + + +class Crate(TypedDict): + display_name: str + root_module: str + is_workspace_member: bool + deps: List[Dependency] + cfg: List[str] + edition: Literal["2021"] + env: Dict[str, str] + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class ProcMacroCrate(Crate): + is_proc_macro: Literal[True] + proc_macro_dylib_path: Optional[str] # `pathlib.Path` is not JSON serializable. + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class CrateWithGenerated(Crate): + source: Optional[Source] + + +def generate_crates( + srctree: pathlib.Path, + objtree: pathlib.Path, + sysroot_src: pathlib.Path, + external_src: pathlib.Path, + cfgs: List[str], +) -> List[Crate]: # Generate the configuration list. cfg = [] with open(objtree / "include" / "generated" / "rustc_cfg") as fd:@@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Now fill the crates list -- dependencies need to come first. # # Avoid O(n^2) iterations by keeping a map of indexes. - crates = [] - crates_indexes = {} + crates: List[Crate] = [] + crates_indexes: Dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def build_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): - crate = { + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> Crate: + return { "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "is_proc_macro": is_proc_macro, "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" - } + }, } - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - return crate - - def register_crate(crate): + + def register_crate(crate: Crate) -> None: crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) def append_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> None: register_crate( - build_crate( - display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro - ) + build_crate(display_name, root_module, deps, cfg, is_workspace_member) ) + def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + ) -> None: + crate = build_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output(
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 129
Quoted 10 lines
scripts/generate_rust_analyzer.py +10
-0+ [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], + stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + proc_macro_crate: ProcMacroCrate = { + **crate, + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + } + register_crate(proc_macro_crate) +
Same here. Is this supposed to be related to Python type hints somehow?
Quoted 6 lines
scripts/generate_rust_analyzer.py +0
-5def append_sysroot_crate( - display_name, - deps, - cfg=[], - ): - append_crate(
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 135
Quoted 10 lines
scripts/generate_rust_analyzer.py +5
-5- display_name, - sysroot_src / display_name / "src" / "lib.rs", - deps, - cfg, - is_workspace_member=False, + display_name: str, + deps: List[str], + cfg: List[str] = [], + ) -> None: + register_crate(
Why is register_crate here now? Maybe this change belongs to the preceding patch?
Quoted 94 lines
scripts/generate_rust_analyzer.py +26
-16+ build_crate( + display_name, + sysroot_src / display_name / "src" / "lib.rs", + deps, + cfg, + is_workspace_member=False, + ) ) # NB: sysroot crates reexport items from one another so setting up our transitive dependencies@@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): [], ) - append_crate( + append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", ["std", "proc_macro"], - is_proc_macro=True, ) append_crate(@@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ["core", "compiler_builtins"], ) - append_crate( + append_proc_macro_crate( "pin_init_internal", srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs", [], cfg=["kernel"], - is_proc_macro=True, ) append_crate(@@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ) def append_crate_with_generated( - display_name, - deps, - ): + display_name: str, + deps: List[str], + ) -> None: crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs",@@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): cfg=cfg, ) crate["env"]["OBJTREE"] = str(objtree.resolve(True)) - crate["source"] = { - "include_dirs": [ - str(srctree / "rust" / display_name), - str(objtree / "rust") - ], - "exclude_dirs": [], + crate_with_generated: CrateWithGenerated = { + **crate, + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } - register_crate(crate) + register_crate(crate_with_generated) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"]) append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"]) - def is_root_crate(build_file, target): + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: return f"{target}.o" in open(build_file).read() except FileNotFoundError:@@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs: Iterable[pathlib.Path] = map( + lambda dir: srctree / dir, ("samples", "drivers") + ) if external_src is not None: extra_dirs = [external_src] for folder in extra_dirs:
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 246
Quoted 5 lines
scripts/generate_rust_analyzer.py +1
-1@@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): return crates -def main(): +
What is this extra blank line for? Automatically generated by a formatter?
Quoted 9 lines
scripts/generate_rust_analyzer.py +1
-0+def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--verbose", "-v", action="store_true") parser.add_argument("--cfgs", action="append", default=[])Signature · 2 lines
-- 2.48.1
Other than what I said above, the type hints themselves are fine.
— Daniel
Daniel Almeida
reviewerRe: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints
Ah, by the way:
Quoted 15 lines · Daniel Almeida
Hi Tamir,
Quoted 11 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").Run
mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8to verify. Note thatmypyno longer supports python < 3.8.
$ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8 Success: no issues found in 1 source file
— Daniel
Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints
Quoted 3 lines · Daniel Almeida
$ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8 Success: no issues found in 1 source file
Should this be a Tested-by? Or am I confused and did you mean something else?
Cheers,
Miguel
Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints
Hi Miguel,
Quoted 5 lines · Miguel Ojeda
Quoted 1 line · Daniel Almeida
Quoted 7 lines
Quoted 2 lines
$ mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8 Success: no issues found in 1 source fileShould this be a Tested-by? Or am I confused and did you mean something else?
Cheers,
Miguel
Yes, that was a bit convoluted, sorry.
Re: [PATCH v4 05/11] scripts: generate_rust_analyzer.py: add type hints
In reply to Daniel AlmeidaQuoted 15 lines · Daniel Almeida
Hi Tamir,
Quoted 12 lines · Tamir Duberstein
Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience.
Python type hints have been present in the kernel since 2019 at the latest; see commit
6ebf5866f2e8("kunit: tool: add Python wrappers for running KUnit tests").Run
mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8to verify. Note thatmypyno longer supports python < 3.8.
Quoted 6 lines
Quoted 3 lines
This removes
"is_proc_macro": falsefromrust-project.jsonin exchange for stricter types. This field is interpreted as false if absent[1] so this doesn't change the behavior of rust-analyzer.Can this be a separate patch? Not sure how this is related to Python type hints, but it makes the current patch harder to review.
Yeah, I'll pop it out into another patch.
Quoted 155 lines
Quoted 155 lines
Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 172 +++++++++++++++++++++++++-------------1 file changed, 112 insertions(+), 60 deletions(-)scripts/generate_rust_analyzer.py +0
-0@@ -10,8 +10,10 @@ import osimport pathlib
import subprocess
import sysscripts/generate_rust_analyzer.py +73
-33+from typing import Dict, Iterable, List, Literal, Optional, TypedDict -def args_crates_cfgs(cfgs): + +def args_crates_cfgs(cfgs: Iterable[str]) -> Dict[str, List[str]]: crates_cfgs = {} for cfg in cfgs: crate, vals = cfg.split("=", 1)@@ -19,7 +21,45 @@ def args_crates_cfgs(cfgs): return crates_cfgs -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): + +class Dependency(TypedDict): + crate: int + name: str + + +class Source(TypedDict): + include_dirs: List[str] + exclude_dirs: List[str] + + +class Crate(TypedDict): + display_name: str + root_module: str + is_workspace_member: bool + deps: List[Dependency] + cfg: List[str] + edition: Literal["2021"] + env: Dict[str, str] + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class ProcMacroCrate(Crate): + is_proc_macro: Literal[True] + proc_macro_dylib_path: Optional[str] # `pathlib.Path` is not JSON serializable. + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was added in 3.11. +class CrateWithGenerated(Crate): + source: Optional[Source] + + +def generate_crates( + srctree: pathlib.Path, + objtree: pathlib.Path, + sysroot_src: pathlib.Path, + external_src: pathlib.Path, + cfgs: List[str], +) -> List[Crate]: # Generate the configuration list. cfg = [] with open(objtree / "include" / "generated" / "rustc_cfg") as fd:@@ -31,67 +71,75 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Now fill the crates list -- dependencies need to come first. # # Avoid O(n^2) iterations by keeping a map of indexes. - crates = [] - crates_indexes = {} + crates: List[Crate] = [] + crates_indexes: Dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def build_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): - crate = { + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> Crate: + return { "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "is_proc_macro": is_proc_macro, "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], "cfg": cfg, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" - } + }, } - if is_proc_macro: - proc_macro_dylib_name = subprocess.check_output( - [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], - stdin=subprocess.DEVNULL, - ).decode('utf-8').strip() - crate["proc_macro_dylib_path"] = f"{objtree}/rust/{proc_macro_dylib_name}" - return crate - - def register_crate(crate): + + def register_crate(crate: Crate) -> None: crates_indexes[crate["display_name"]] = len(crates) crates.append(crate) def append_crate( - display_name, - root_module, - deps, - cfg=[], - is_workspace_member=True, - is_proc_macro=False, - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + is_workspace_member: bool = True, + ) -> None: register_crate( - build_crate( - display_name, root_module, deps, cfg, is_workspace_member, is_proc_macro - ) + build_crate(display_name, root_module, deps, cfg, is_workspace_member) ) + def append_proc_macro_crate( + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] = [], + ) -> None: + crate = build_crate(display_name, root_module, deps, cfg) + proc_macro_dylib_name = subprocess.check_output( + [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"], + stdin=subprocess.DEVNULL, + ).decode('utf-8').strip() + proc_macro_crate: ProcMacroCrate = {
Quoted 8 lines
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 129
Quoted 6 lines
scripts/generate_rust_analyzer.py +6
-0+ **crate, + "is_proc_macro": True, + "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + } + register_crate(proc_macro_crate) +Same here. Is this supposed to be related to Python type hints somehow?
Same here - moved into a separate patch.
Quoted 17 lines
Quoted 6 lines
scripts/generate_rust_analyzer.py +0
-5def append_sysroot_crate( - display_name, - deps, - cfg=[], - ): - append_crate(scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 135
Quoted 10 lines
scripts/generate_rust_analyzer.py +5
-5- display_name, - sysroot_src / display_name / "src" / "lib.rs", - deps, - cfg, - is_workspace_member=False, + display_name: str, + deps: List[str], + cfg: List[str] = [], + ) -> None: + register_crate(
Quoted 1 line
Why is
register_cratehere now? Maybe this change belongs to the preceding patch?
Good spot, this didn't need to change. Reverted in this patch.
Quoted 96 lines
Quoted 96 lines
scripts/generate_rust_analyzer.py +26
-16+ build_crate( + display_name, + sysroot_src / display_name / "src" / "lib.rs", + deps, + cfg, + is_workspace_member=False, + ) ) # NB: sysroot crates reexport items from one another so setting up our transitive dependencies@@ -108,11 +156,10 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): [], ) - append_crate( + append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", ["std", "proc_macro"], - is_proc_macro=True, ) append_crate(@@ -121,12 +168,11 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ["core", "compiler_builtins"], ) - append_crate( + append_proc_macro_crate( "pin_init_internal", srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs", [], cfg=["kernel"], - is_proc_macro=True, ) append_crate(@@ -137,9 +183,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): ) def append_crate_with_generated( - display_name, - deps, - ): + display_name: str, + deps: List[str], + ) -> None: crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs",@@ -147,20 +193,23 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): cfg=cfg, ) crate["env"]["OBJTREE"] = str(objtree.resolve(True)) - crate["source"] = { - "include_dirs": [ - str(srctree / "rust" / display_name), - str(objtree / "rust") - ], - "exclude_dirs": [], + crate_with_generated: CrateWithGenerated = { + **crate, + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } - register_crate(crate) + register_crate(crate_with_generated) append_crate_with_generated("bindings", ["core"]) append_crate_with_generated("uapi", ["core"]) append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"]) - def is_root_crate(build_file, target): + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: return f"{target}.o" in open(build_file).read() except FileNotFoundError:@@ -169,7 +218,9 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs = map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs: Iterable[pathlib.Path] = map( + lambda dir: srctree / dir, ("samples", "drivers") + ) if external_src is not None: extra_dirs = [external_src] for folder in extra_dirs:@@ -192,7 +243,8 @@ def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs):
Quoted 6 lines
scripts/generate_rust_analyzer.py def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): · line 246
Quoted 4 lines
scripts/generate_rust_analyzer.py +1
-1return crates -def main(): +What is this extra blank line for? Automatically generated by a formatter?
Yeah, this comes from PEP 8. Moved to "scripts:generate_rust_analyzer.py: add missing whitespace".
Quoted 6 lines
Quoted 6 lines
scripts/generate_rust_analyzer.py +1
-0+def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--verbose", "-v", action="store_true") parser.add_argument("--cfgs", action="append", default=[])
Quoted 8 lines
Quoted 4 lines
--
2.48.1Other than what I said above, the type hints themselves are fine.
— Daniel
Thanks for the reviews!
PATCH v4 6/11 scripts: generate_rust_analyzer.py: use str(pathlib.Path)
In reply to Tamir DubersteinUse the / operator on pathlib.Path rather than directly crafting a string. This is consistent with all other path manipulation in this script.
--- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +1 -1
@@ -123,7 +123,7 @@ def generate_crates( proc_macro_crate: ProcMacroCrate = { **crate, "is_proc_macro": True, - "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } register_crate(proc_macro_crate)
Signature · 1 line
2.48.1
Re: [PATCH v4 06/11] scripts: generate_rust_analyzer.py: use str(pathlib.Path)
Quoted 32 lines · Tamir Duberstein
Use the
/operator onpathlib.Pathrather than directly crafting a string. This is consistent with all other path manipulation in this script.Reviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)scripts/generate_rust_analyzer.py +1
-1@@ -123,7 +123,7 @@ def generate_crates( proc_macro_crate: ProcMacroCrate = { **crate, "is_proc_macro": True, - "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_name}", + "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } register_crate(proc_macro_crate)Signature · 2 lines
-- 2.48.1
PATCH v4 7/11 scripts: generate_rust_analyzer.py: identify crates explicitly
In reply to Tamir DubersteinUse the return of append_crate to declare dependency on that crate. This allows multiple crates with the same display_name be defined, which we'll use to define host crates separately from target crates.
--- scripts/generate_rust_analyzer.py | 70 +++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 36 deletions(-)
scripts/generate_rust_analyzer.py +34 -36
@@ -68,17 +68,14 @@ def generate_crates( line = line.replace("\n", "") cfg.append(line) - # Now fill the crates list -- dependencies need to come first. - # - # Avoid O(n^2) iterations by keeping a map of indexes. + # Now fill the crates list. crates: List[Crate] = [] - crates_indexes: Dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def build_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], is_workspace_member: bool = True, ) -> Crate:
@@ -86,7 +83,7 @@ def generate_crates( "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], + "deps": deps, "cfg": cfg, "edition": "2021", "env": {
@@ -94,27 +91,28 @@ def generate_crates( }, } - def register_crate(crate: Crate) -> None: - crates_indexes[crate["display_name"]] = len(crates) + def register_crate(crate: Crate) -> Dependency: + index = len(crates) crates.append(crate) + return {"crate": index, "name": crate["display_name"]} def append_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], is_workspace_member: bool = True, - ) -> None: - register_crate( + ) -> Dependency: + return register_crate( build_crate(display_name, root_module, deps, cfg, is_workspace_member) ) def append_proc_macro_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], - ) -> None: + ) -> Dependency: crate = build_crate(display_name, root_module, deps, cfg) proc_macro_dylib_name = subprocess.check_output( [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],
@@ -125,14 +123,14 @@ def generate_crates( "is_proc_macro": True, "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } - register_crate(proc_macro_crate) + return register_crate(proc_macro_crate) def append_sysroot_crate( display_name: str, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], - ) -> None: - register_crate( + ) -> Dependency: + return register_crate( build_crate( display_name, sysroot_src / display_name / "src" / "lib.rs",
@@ -145,47 +143,47 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. - append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - append_sysroot_crate("alloc", ["core"]) - append_sysroot_crate("std", ["alloc", "core"]) - append_sysroot_crate("proc_macro", ["core", "std"]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) + alloc = append_sysroot_crate("alloc", [core]) + std = append_sysroot_crate("std", [alloc, core]) + proc_macro = append_sysroot_crate("proc_macro", [core, std]) - append_crate( + compiler_builtins = append_crate( "compiler_builtins", srctree / "rust" / "compiler_builtins.rs", [], ) - append_proc_macro_crate( + macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - ["std", "proc_macro"], + [std, proc_macro], ) - append_crate( + build_error = append_crate( "build_error", srctree / "rust" / "build_error.rs", - ["core", "compiler_builtins"], + [core, compiler_builtins], ) - append_proc_macro_crate( + pin_init_internal = append_proc_macro_crate( "pin_init_internal", srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs", [], cfg=["kernel"], ) - append_crate( + pin_init = append_crate( "pin_init", srctree / "rust" / "pin-init" / "src" / "lib.rs", - ["core", "pin_init_internal", "macros"], + [core, pin_init_internal, macros], cfg=["kernel"], ) def append_crate_with_generated( display_name: str, - deps: List[str], - ) -> None: + deps: List[Dependency], + ) -> Dependency: crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs",
@@ -203,11 +201,11 @@ def generate_crates( "exclude_dirs": [], } } - register_crate(crate_with_generated) + return register_crate(crate_with_generated) - append_crate_with_generated("bindings", ["core"]) - append_crate_with_generated("uapi", ["core"]) - append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"]) + bindings = append_crate_with_generated("bindings", [core]) + uapi = append_crate_with_generated("uapi", [core]) + kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi]) def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try:
@@ -237,7 +235,7 @@ def generate_crates( append_crate( name, path, - ["core", "kernel"], + [core, kernel], cfg=cfg, )
Signature · 1 line
2.48.1
Re: [PATCH v4 07/11] scripts: generate_rust_analyzer.py: identify crates explicitly
Quoted 194 lines · Tamir Duberstein
Use the return of
append_crateto declare dependency on that crate. This allows multiple crates with the samedisplay_namebe defined, which we'll use to define host crates separately from target crates.Reviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 70 +++++++++++++++++++--------------------1 file changed, 34 insertions(+), 36 deletions(-)scripts/generate_rust_analyzer.py +34
-36@@ -68,17 +68,14 @@ def generate_crates( line = line.replace("\n", "") cfg.append(line) - # Now fill the crates list -- dependencies need to come first. - # - # Avoid O(n^2) iterations by keeping a map of indexes. + # Now fill the crates list. crates: List[Crate] = [] - crates_indexes: Dict[str, int] = {} crates_cfgs = args_crates_cfgs(cfgs) def build_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], is_workspace_member: bool = True, ) -> Crate:@@ -86,7 +83,7 @@ def generate_crates( "display_name": display_name, "root_module": str(root_module), "is_workspace_member": is_workspace_member, - "deps": [{"crate": crates_indexes[dep], "name": dep} for dep in deps], + "deps": deps, "cfg": cfg, "edition": "2021", "env": {@@ -94,27 +91,28 @@ def generate_crates( }, } - def register_crate(crate: Crate) -> None: - crates_indexes[crate["display_name"]] = len(crates) + def register_crate(crate: Crate) -> Dependency: + index = len(crates) crates.append(crate) + return {"crate": index, "name": crate["display_name"]} def append_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], is_workspace_member: bool = True, - ) -> None: - register_crate( + ) -> Dependency: + return register_crate( build_crate(display_name, root_module, deps, cfg, is_workspace_member) ) def append_proc_macro_crate( display_name: str, root_module: pathlib.Path, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], - ) -> None: + ) -> Dependency: crate = build_crate(display_name, root_module, deps, cfg) proc_macro_dylib_name = subprocess.check_output( [os.environ["RUSTC"], "--print", "file-names", "--crate-name", display_name, "--crate-type", "proc-macro", "-"],@@ -125,14 +123,14 @@ def generate_crates( "is_proc_macro": True, "proc_macro_dylib_path": str(objtree / "rust" / proc_macro_dylib_name), } - register_crate(proc_macro_crate) + return register_crate(proc_macro_crate) def append_sysroot_crate( display_name: str, - deps: List[str], + deps: List[Dependency], cfg: List[str] = [], - ) -> None: - register_crate( + ) -> Dependency: + return register_crate( build_crate( display_name, sysroot_src / display_name / "src" / "lib.rs",@@ -145,47 +143,47 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. - append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - append_sysroot_crate("alloc", ["core"]) - append_sysroot_crate("std", ["alloc", "core"]) - append_sysroot_crate("proc_macro", ["core", "std"]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) + alloc = append_sysroot_crate("alloc", [core]) + std = append_sysroot_crate("std", [alloc, core]) + proc_macro = append_sysroot_crate("proc_macro", [core, std]) - append_crate( + compiler_builtins = append_crate( "compiler_builtins", srctree / "rust" / "compiler_builtins.rs", [], ) - append_proc_macro_crate( + macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - ["std", "proc_macro"], + [std, proc_macro], ) - append_crate( + build_error = append_crate( "build_error", srctree / "rust" / "build_error.rs", - ["core", "compiler_builtins"], + [core, compiler_builtins], ) - append_proc_macro_crate( + pin_init_internal = append_proc_macro_crate( "pin_init_internal", srctree / "rust" / "pin-init" / "internal" / "src" / "lib.rs", [], cfg=["kernel"], ) - append_crate( + pin_init = append_crate( "pin_init", srctree / "rust" / "pin-init" / "src" / "lib.rs", - ["core", "pin_init_internal", "macros"], + [core, pin_init_internal, macros], cfg=["kernel"], ) def append_crate_with_generated( display_name: str, - deps: List[str], - ) -> None: + deps: List[Dependency], + ) -> Dependency: crate = build_crate( display_name, srctree / "rust" / display_name / "lib.rs",@@ -203,11 +201,11 @@ def generate_crates( "exclude_dirs": [], } } - register_crate(crate_with_generated) + return register_crate(crate_with_generated) - append_crate_with_generated("bindings", ["core"]) - append_crate_with_generated("uapi", ["core"]) - append_crate_with_generated("kernel", ["core", "macros", "build_error", "bindings", "pin_init", "uapi"]) + bindings = append_crate_with_generated("bindings", [core]) + uapi = append_crate_with_generated("uapi", [core]) + kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi]) def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try:@@ -237,7 +235,7 @@ def generate_crates( append_crate( name, path, - ["core", "kernel"], + [core, kernel], cfg=cfg, )Signature · 2 lines
-- 2.48.1
PATCH v4 8/11 scripts: generate_rust_analyzer.py: define host crates
In reply to Tamir DubersteinDefine host crates used by the macros crate separately from target crates, now that we can uniquely identify crates with the same name.
This avoids rust-analyzer thinking the host core crate has our target configs applied to it.
--- scripts/generate_rust_analyzer.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
scripts/generate_rust_analyzer.py +6 -4
@@ -143,10 +143,12 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. + host_core = append_sysroot_crate("core", []) + host_alloc = append_sysroot_crate("alloc", [host_core]) + host_std = append_sysroot_crate("std", [host_alloc, host_core]) + host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - alloc = append_sysroot_crate("alloc", [core]) - std = append_sysroot_crate("std", [alloc, core]) - proc_macro = append_sysroot_crate("proc_macro", [core, std]) compiler_builtins = append_crate( "compiler_builtins",
@@ -157,7 +159,7 @@ def generate_crates( macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - [std, proc_macro], + [host_std, host_proc_macro], ) build_error = append_crate(
Signature · 1 line
2.48.1
Re: [PATCH v4 08/11] scripts: generate_rust_analyzer.py: define host crates
Quoted 52 lines · Tamir Duberstein
Define host crates used by the
macroscrate separately from target crates, now that we can uniquely identify crates with the same name.This avoids rust-analyzer thinking the host
corecrate has our target configs applied to it.Suggested-by Miguel Ojeda <[email protected]>Link https://lore.kernel.org/all/CANiq72mw83RmLYeFFoJW6mUUygoyiA_f1ievSC2pmBESsQew+w@mail.gmail.com/ open hereReviewed-by Fiona Behrens <[email protected]>Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 10 ++++++----1 file changed, 6 insertions(+), 4 deletions(-)scripts/generate_rust_analyzer.py +6
-4@@ -143,10 +143,12 @@ def generate_crates( # NB: sysroot crates reexport items from one another so setting up our transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbols. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" for crate in crates)`. + host_core = append_sysroot_crate("core", []) + host_alloc = append_sysroot_crate("alloc", [host_core]) + host_std = append_sysroot_crate("std", [host_alloc, host_core]) + host_proc_macro = append_sysroot_crate("proc_macro", [host_core, host_std]) + core = append_sysroot_crate("core", [], cfg=crates_cfgs.get("core", [])) - alloc = append_sysroot_crate("alloc", [core]) - std = append_sysroot_crate("std", [alloc, core]) - proc_macro = append_sysroot_crate("proc_macro", [core, std]) compiler_builtins = append_crate( "compiler_builtins",@@ -157,7 +159,7 @@ def generate_crates( macros = append_proc_macro_crate( "macros", srctree / "rust" / "macros" / "lib.rs", - [std, proc_macro], + [host_std, host_proc_macro], ) build_error = append_crate(Signature · 2 lines
-- 2.48.1
PATCH v4 9/11 scripts: generate_rust_analyzer.py: avoid FD leak
In reply to Tamir DubersteinUse a context manager to avoid leaking file descriptors.
--- scripts/generate_rust_analyzer.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
scripts/generate_rust_analyzer.py +2 -1
@@ -211,7 +211,8 @@ def generate_crates( def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: - return f"{target}.o" in open(build_file).read() + with open(build_file) as f: + return f"{target}.o" in f.read() except FileNotFoundError: return False
Signature · 1 line
2.48.1
Re: [PATCH v4 09/11] scripts: generate_rust_analyzer.py: avoid FD leak
Quoted 30 lines · Tamir Duberstein
Use a context manager to avoid leaking file descriptors.
Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 3 ++-1 file changed, 2 insertions(+), 1 deletion(-)scripts/generate_rust_analyzer.py +2
-1@@ -211,7 +211,8 @@ def generate_crates( def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: - return f"{target}.o" in open(build_file).read() + with open(build_file) as f: + return f"{target}.o" in f.read() except FileNotFoundError: return FalseSignature · 2 lines
-- 2.48.1
Oh, nice!
Re: [PATCH v4 09/11] scripts: generate_rust_analyzer.py: avoid FD leak
In reply to Tamir DubersteinQuoted 3 lines · Tamir Duberstein
Use a context manager to avoid leaking file descriptors.
Signed-off-by Tamir Duberstein <[email protected]>
Quoted 17 lines
--- scripts/generate_rust_analyzer.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)scripts/generate_rust_analyzer.py +2
-1@@ -211,7 +211,8 @@ def generate_crates( def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: - return f"{target}.o" in open(build_file).read() + with open(build_file) as f: + return f"{target}.o" in f.read() except FileNotFoundError: return False
PATCH v4 10/11 scripts: generate_rust_analyzer.py: define scripts
In reply to Tamir DubersteinGenerate rust-project.json entries for scripts written in Rust. This is possible now that we have a definition for std built for the host.
--- scripts/generate_rust_analyzer.py | 13 +++++++++++++ 1 file changed, 13 insertions(+)
scripts/generate_rust_analyzer.py +13 -0
@@ -209,6 +209,19 @@ def generate_crates( uapi = append_crate_with_generated("uapi", [core]) kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi]) + scripts = srctree / "scripts" + with open(scripts / "Makefile") as f: + makefile = f.read() + for path in scripts.glob("*.rs"): + name = path.name.replace(".rs", "") + if f"{name}-rust" not in makefile: + continue + _script = append_crate( + name, + path, + [host_std], + ) + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: with open(build_file) as f:
Signature · 1 line
2.48.1
Re: [PATCH v4 10/11] scripts: generate_rust_analyzer.py: define scripts
Quoted 41 lines · Tamir Duberstein
Generate rust-project.json entries for scripts written in Rust. This is possible now that we have a definition for
stdbuilt for the host.Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 13 +++++++++++++1 file changed, 13 insertions(+)scripts/generate_rust_analyzer.py +13
-0@@ -209,6 +209,19 @@ def generate_crates( uapi = append_crate_with_generated("uapi", [core]) kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi]) + scripts = srctree / "scripts" + with open(scripts / "Makefile") as f: + makefile = f.read() + for path in scripts.glob("*.rs"): + name = path.name.replace(".rs", "") + if f"{name}-rust" not in makefile: + continue + _script = append_crate( + name, + path, + [host_std], + ) + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: with open(build_file) as f:Signature · 2 lines
-- 2.48.1
Re: [PATCH v4 10/11] scripts: generate_rust_analyzer.py: define scripts
In reply to Tamir DubersteinQuoted 4 lines · Tamir Duberstein
Generate rust-project.json entries for scripts written in Rust. This is possible now that we have a definition for
stdbuilt for the host.Signed-off-by Tamir Duberstein <[email protected]>
Quoted 28 lines
--- scripts/generate_rust_analyzer.py | 13 +++++++++++++ 1 file changed, 13 insertions(+)scripts/generate_rust_analyzer.py +13
-0@@ -209,6 +209,19 @@ def generate_crates( uapi = append_crate_with_generated("uapi", [core]) kernel = append_crate_with_generated("kernel", [core, macros, build_error, bindings, pin_init, uapi]) + scripts = srctree / "scripts" + with open(scripts / "Makefile") as f: + makefile = f.read() + for path in scripts.glob("*.rs"): + name = path.name.replace(".rs", "") + if f"{name}-rust" not in makefile: + continue + _script = append_crate( + name, + path, + [host_std], + ) + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: with open(build_file) as f:
PATCH v4 11/11 scripts: generate_rust_analyzer.py: use `cfg_groups`
In reply to Tamir DubersteinDeclare common cfgs just once to reduce the size of rust-analyzer.json from 30619 to 2624 lines.
--- scripts/generate_rust_analyzer.py | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)
scripts/generate_rust_analyzer.py +23 -12
@@ -37,6 +37,7 @@ class Crate(TypedDict): root_module: str is_workspace_member: bool deps: List[Dependency] + cfg_groups: List[str] cfg: List[str] edition: Literal["2021"] env: Dict[str, str]
@@ -59,15 +60,8 @@ def generate_crates( sysroot_src: pathlib.Path, external_src: pathlib.Path, cfgs: List[str], + cfg_groups: List[str], ) -> List[Crate]: - # Generate the configuration list. - cfg = [] - with open(objtree / "include" / "generated" / "rustc_cfg") as fd: - for line in fd: - line = line.replace("--cfg=", "") - line = line.replace("\n", "") - cfg.append(line) - # Now fill the crates list. crates: List[Crate] = [] crates_cfgs = args_crates_cfgs(cfgs)
@@ -77,6 +71,7 @@ def generate_crates( root_module: pathlib.Path, deps: List[Dependency], cfg: List[str] = [], + cfg_groups: List[str] = [], is_workspace_member: bool = True, ) -> Crate: return {
@@ -85,6 +80,7 @@ def generate_crates( "is_workspace_member": is_workspace_member, "deps": deps, "cfg": cfg, + "cfg_groups": cfg_groups, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer"
@@ -101,10 +97,13 @@ def generate_crates( root_module: pathlib.Path, deps: List[Dependency], cfg: List[str] = [], + cfg_groups: List[str] = [], is_workspace_member: bool = True, ) -> Dependency: return register_crate( - build_crate(display_name, root_module, deps, cfg, is_workspace_member) + build_crate( + display_name, root_module, deps, cfg, cfg_groups, is_workspace_member + ) ) def append_proc_macro_crate(
@@ -190,7 +189,7 @@ def generate_crates( display_name, srctree / "rust" / display_name / "lib.rs", deps, - cfg=cfg, + cfg_groups=cfg_groups, ) crate["env"]["OBJTREE"] = str(objtree.resolve(True)) crate_with_generated: CrateWithGenerated = {
@@ -252,7 +251,7 @@ def generate_crates( name, path, [core, kernel], - cfg=cfg, + cfg_groups=cfg_groups, ) return crates
@@ -277,9 +276,21 @@ def main() -> None: # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain. assert args.sysroot in args.sysroot_src.parents + # Generate the configuration list. + with open(args.objtree / "include" / "generated" / "rustc_cfg") as fd: + cfg_groups = {"rustc_cfg": [line.lstrip("--cfg=").rstrip("\n") for line in fd]} + rust_project = { - "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs), + "crates": generate_crates( + args.srctree, + args.objtree, + args.sysroot_src, + args.exttree, + args.cfgs, + list(cfg_groups.keys()), + ), "sysroot": str(args.sysroot), + "cfg_groups": cfg_groups, } json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)
Signature · 1 line
2.48.1
Re: [PATCH v4 11/11] scripts: generate_rust_analyzer.py: use `cfg_groups`
Quoted 6 lines · Tamir Duberstein
Declare common
cfgs just once to reduce the size of rust-analyzer.json from 30619 to 2624 lines.
A note in case it matters for someone out there: that commit appeared in tag 2024-08-26, but our minimum Rust version is dated 2024-05-02.
I think many developers are using a rust-analyzer that is newer (i.e. installed independently), but if others install the component that gets distributed alongside Rust, then I think it appeared in Rust 1.82.0.
So I wonder if we should wait for the MSRV bump to start using this.
Thanks!
Cheers,
Miguel
Re: [PATCH v4 11/11] scripts: generate_rust_analyzer.py: use `cfg_groups`
In reply to Tamir DubersteinQuoted 119 lines · Tamir Duberstein
Declare common
cfgs just once to reduce the size of rust-analyzer.json from 30619 to 2624 lines.Signed-off-by Tamir Duberstein <[email protected]>---
scripts/generate_rust_analyzer.py| 35 +++++++++++++++++++++++------------1 file changed, 23 insertions(+), 12 deletions(-)scripts/generate_rust_analyzer.py +2
-0@@ -37,6 +37,7 @@ class Crate(TypedDict): root_module: str is_workspace_member: bool deps: List[Dependency] + cfg_groups: List[str] cfg: List[str] edition: Literal["2021"] env: Dict[str, str]@@ -59,15 +60,8 @@ def generate_crates( sysroot_src: pathlib.Path, external_src: pathlib.Path, cfgs: List[str], + cfg_groups: List[str],) -> List[Crate]:
scripts/generate_rust_analyzer.py +21
-12- # Generate the configuration list. - cfg = [] - with open(objtree / "include" / "generated" / "rustc_cfg") as fd: - for line in fd: - line = line.replace("--cfg=", "") - line = line.replace("\n", "") - cfg.append(line) - # Now fill the crates list. crates: List[Crate] = [] crates_cfgs = args_crates_cfgs(cfgs)@@ -77,6 +71,7 @@ def generate_crates( root_module: pathlib.Path, deps: List[Dependency], cfg: List[str] = [], + cfg_groups: List[str] = [], is_workspace_member: bool = True, ) -> Crate: return {@@ -85,6 +80,7 @@ def generate_crates( "is_workspace_member": is_workspace_member, "deps": deps, "cfg": cfg, + "cfg_groups": cfg_groups, "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer"@@ -101,10 +97,13 @@ def generate_crates( root_module: pathlib.Path, deps: List[Dependency], cfg: List[str] = [], + cfg_groups: List[str] = [], is_workspace_member: bool = True, ) -> Dependency: return register_crate( - build_crate(display_name, root_module, deps, cfg, is_workspace_member) + build_crate( + display_name, root_module, deps, cfg, cfg_groups, is_workspace_member + ) ) def append_proc_macro_crate(@@ -190,7 +189,7 @@ def generate_crates( display_name, srctree / "rust" / display_name / "lib.rs", deps, - cfg=cfg, + cfg_groups=cfg_groups, ) crate["env"]["OBJTREE"] = str(objtree.resolve(True)) crate_with_generated: CrateWithGenerated = {@@ -252,7 +251,7 @@ def generate_crates( name, path, [core, kernel], - cfg=cfg, + cfg_groups=cfg_groups, ) return crates@@ -277,9 +276,21 @@ def main() -> None: # Making sure that the `sysroot` and `sysroot_src` belong to the same toolchain. assert args.sysroot in args.sysroot_src.parents + # Generate the configuration list. + with open(args.objtree / "include" / "generated" / "rustc_cfg") as fd: + cfg_groups = {"rustc_cfg": [line.lstrip("--cfg=").rstrip("\n") for line in fd]} + rust_project = { - "crates": generate_crates(args.srctree, args.objtree, args.sysroot_src, args.exttree, args.cfgs), + "crates": generate_crates( + args.srctree, + args.objtree, + args.sysroot_src, + args.exttree, + args.cfgs, + list(cfg_groups.keys()), + ), "sysroot": str(args.sysroot), + "cfg_groups": cfg_groups, } json.dump(rust_project, sys.stdout, sort_keys=True, indent=4)Signature · 2 lines
-- 2.48.1