dt-bindings: Improve yamllint performance
cover dt-bindings: Improve yamllint performance
Hi Rob,
This series improves yamllint performance by parallelizing the operation, and by restricting the checked files to those specified using DT_SCHEMA_FILES.
Changes compared to v1:
- New patch to parallelize yamllint,
- Introduce find_all_cmd,
- Only use the restricted set for yamllint.
I've been using this for the past 6 months, as it helps a lot when writing or updating DT bindings. Combined, this reduces the execution time of
make dt_binding_check DT_SCHEMA_FILES=Documentation/devicetree/bindings/your/binding/file.yaml
from ca. 30 to 10 s on i7-8700K.
Thanks for your comments!
Geert Uytterhoeven (2):
dt-bindings: Parallelize yamllint
dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema
Documentation/devicetree/bindings/Makefile | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
Signature · 12 lines
2.25.1
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus TorvaldsPATCH 1/2 dt-bindings: Parallelize yamllint
Use xargs sharding like "chk_bindings" does, to parallelize the execution of yamllint.
This reduces the yamllint execution time from ca. 21 to 5 seconds on i7-8700K.
---
v2:
- New.
--- Documentation/devicetree/bindings/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/devicetree/bindings/Makefile +1 -1
@@ -28,7 +28,7 @@ find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \ quiet_cmd_yamllint = LINT $(src) cmd_yamllint = ($(find_cmd) | \ - xargs $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true + xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true quiet_cmd_chk_bindings = CHKDT $@ cmd_chk_bindings = ($(find_cmd) | \
Signature · 1 line
2.25.1
Re: [PATCH 1/2] dt-bindings: Parallelize yamllint
Quoted 6 lines · Geert Uytterhoeven
Use xargs sharding like "
chk_bindings" does, to parallelize the execution of yamllint.This reduces the yamllint execution time from ca. 21 to 5 seconds on i7-8700K.
Quoted 9 lines
Suggested-by Rob Herring <[email protected]>Signed-off-by Geert Uytterhoeven <[email protected]>---
v2:
- New.--- Documentation/devicetree/bindings/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied, thanks!
PATCH 2/2 dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema
In reply to Geert UytterhoevenSetting DT_SCHEMA_FILES allows the user to restrict the "dt_binding_check" make target to a specified set of DT binding files. However, yamllint is still run on all available files, which not only takes time, but also outputs warnings for other binding files the developer is not interested in.
Fix this by renaming "find_cmd" to "find_all_cmd", introducing a new "find_cmd" to only return the files specified by DT_SCHEMA_FILES (if present), and using the latter for yamllint.
---
v2:
- Introduce find_all_cmd,
- Only use the restricted set for yamllint.
--- Documentation/devicetree/bindings/Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/Makefile +10 -4
@@ -22,23 +22,29 @@ $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE # Use full schemas when checking %.example.dts DT_TMP_SCHEMA := $(obj)/processed-schema-examples.json -find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \ +find_all_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \ -name 'processed-schema*' ! \ -name '*.example.dt.yaml' \) +ifeq ($(DT_SCHEMA_FILES),) +find_cmd = $(find_all_cmd) +else +find_cmd = echo $(addprefix $(srctree)/, $(DT_SCHEMA_FILES)) +endif + quiet_cmd_yamllint = LINT $(src) cmd_yamllint = ($(find_cmd) | \ xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true quiet_cmd_chk_bindings = CHKDT $@ - cmd_chk_bindings = ($(find_cmd) | \ + cmd_chk_bindings = ($(find_all_cmd) | \ xargs -n200 -P$$(nproc) $(DT_DOC_CHECKER) -u $(srctree)/$(src)) || true quiet_cmd_mk_schema = SCHEMA $@ cmd_mk_schema = f=$$(mktemp) ; \ $(if $(DT_MK_SCHEMA_FLAGS), \ printf '%s\n' $(real-prereqs), \ - $(find_cmd)) > $$f ; \ + $(find_all_cmd)) > $$f ; \ $(DT_MK_SCHEMA) -j $(DT_MK_SCHEMA_FLAGS) @$$f > $@ ; \ rm -f $$f
@@ -48,7 +54,7 @@ define rule_chkdt $(call cmd,mk_schema) endef -DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_cmd))) +DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_all_cmd))) override DTC_FLAGS := \ -Wno-avoid_unnecessary_addr_size \
Signature · 1 line
2.25.1
Rob Herring
maintainerRe: [PATCH 2/2] dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema
Quoted 9 lines · Geert Uytterhoeven
Setting
DT_SCHEMA_FILESallows the user to restrict the "dt_binding_check" make target to a specified set of DT binding files. However, yamllint is still run on all available files, which not only takes time, but also outputs warnings for other binding files the developer is not interested in.Fix this by renaming "
find_cmd" to "find_all_cmd", introducing a new "find_cmd" to only return the files specified byDT_SCHEMA_FILES(if present), and using the latter for yamllint.
We can also do the same thing on dt-doc-validate since checking and preprocessing schemas are separate steps, so I did while applying.
Quoted 10 lines
Signed-off-by Geert Uytterhoeven <[email protected]>---
v2:
- Introducefind_all_cmd,
- Only use the restricted set for yamllint.--- Documentation/devicetree/bindings/Makefile | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)