Skip to contentSkip to conversation

dt-bindings: Improve yamllint performance

PATCH Applied Geert Uytterhoeven 2 participants 5 messages Oct 18, 2021 – Oct 20, 2021 Device tree bindings

lore
Share & tools
lore
#0 lore raw

cover dt-bindings: Improve yamllint performance

Hi Rob,

This series improves yamllint performance by parallelizing the operation, and by restricting the checked files to those specified using DT_SCHEMA_FILES.

Changes compared to v1:
- New patch to parallelize yamllint,
- Introduce find_all_cmd,
- Only use the restricted set for yamllint.

I've been using this for the past 6 months, as it helps a lot when writing or updating DT bindings. Combined, this reduces the execution time of

make dt_binding_check DT_SCHEMA_FILES=Documentation/devicetree/bindings/your/binding/file.yaml

from ca. 30 to 10 s on i7-8700K.

Thanks for your comments!

Geert Uytterhoeven (2):
dt-bindings: Parallelize yamllint
dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

 Documentation/devicetree/bindings/Makefile | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
Signature · 12 lines
2.25.1

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds
#1 lore raw

PATCH 1/2 dt-bindings: Parallelize yamllint

Use xargs sharding like "chk_bindings" does, to parallelize the execution of yamllint.

This reduces the yamllint execution time from ca. 21 to 5 seconds on i7-8700K.

---
v2:
- New.

---
 Documentation/devicetree/bindings/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/devicetree/bindings/Makefile +1 -1
index a072e95de626c83b..6305cfa9495ed676 100644
@@ -28,7 +28,7 @@ find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
 
 quiet_cmd_yamllint = LINT    $(src)
       cmd_yamllint = ($(find_cmd) | \
-                     xargs $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
+                     xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
 
 quiet_cmd_chk_bindings = CHKDT   $@
       cmd_chk_bindings = ($(find_cmd) | \
Signature · 1 line
2.25.1
maintainer Applied
#2 lore raw

Re: [PATCH 1/2] dt-bindings: Parallelize yamllint

Quoted 6 lines · Geert Uytterhoeven

Use xargs sharding like "chk_bindings" does, to parallelize the execution of yamllint.

This reduces the yamllint execution time from ca. 21 to 5 seconds on i7-8700K.

Quoted 9 lines

---
v2:
- New.

---
 Documentation/devicetree/bindings/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Applied, thanks!

#3 lore raw

PATCH 2/2 dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

In reply to Geert Uytterhoeven

Setting DT_SCHEMA_FILES allows the user to restrict the "dt_binding_check" make target to a specified set of DT binding files. However, yamllint is still run on all available files, which not only takes time, but also outputs warnings for other binding files the developer is not interested in.

Fix this by renaming "find_cmd" to "find_all_cmd", introducing a new "find_cmd" to only return the files specified by DT_SCHEMA_FILES (if present), and using the latter for yamllint.

---
v2:
- Introduce find_all_cmd,
- Only use the restricted set for yamllint.

---
 Documentation/devicetree/bindings/Makefile | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
Documentation/devicetree/bindings/Makefile +10 -4
index 6305cfa9495ed676..1a8c3ab9a538837a 100644
@@ -22,23 +22,29 @@ $(obj)/%.example.dts: $(src)/%.yaml check_dtschema_version FORCE
 # Use full schemas when checking %.example.dts
 DT_TMP_SCHEMA := $(obj)/processed-schema-examples.json
 
-find_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
+find_all_cmd = find $(srctree)/$(src) \( -name '*.yaml' ! \
 		-name 'processed-schema*' ! \
 		-name '*.example.dt.yaml' \)
 
+ifeq ($(DT_SCHEMA_FILES),)
+find_cmd = $(find_all_cmd)
+else
+find_cmd = echo $(addprefix $(srctree)/, $(DT_SCHEMA_FILES))
+endif
+
 quiet_cmd_yamllint = LINT    $(src)
       cmd_yamllint = ($(find_cmd) | \
                      xargs -n200 -P$$(nproc) $(DT_SCHEMA_LINT) -f parsable -c $(srctree)/$(src)/.yamllint >&2) || true
 
 quiet_cmd_chk_bindings = CHKDT   $@
-      cmd_chk_bindings = ($(find_cmd) | \
+      cmd_chk_bindings = ($(find_all_cmd) | \
                          xargs -n200 -P$$(nproc) $(DT_DOC_CHECKER) -u $(srctree)/$(src)) || true
 
 quiet_cmd_mk_schema = SCHEMA  $@
       cmd_mk_schema = f=$$(mktemp) ; \
                       $(if $(DT_MK_SCHEMA_FLAGS), \
                            printf '%s\n' $(real-prereqs), \
-                           $(find_cmd)) > $$f ; \
+                           $(find_all_cmd)) > $$f ; \
                       $(DT_MK_SCHEMA) -j $(DT_MK_SCHEMA_FLAGS) @$$f > $@ ; \
 		      rm -f $$f
 
@@ -48,7 +54,7 @@ define rule_chkdt
 	$(call cmd,mk_schema)
 endef
 
-DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_cmd)))
+DT_DOCS = $(patsubst $(srctree)/%,%,$(shell $(find_all_cmd)))
 
 override DTC_FLAGS := \
 	-Wno-avoid_unnecessary_addr_size \
Signature · 1 line
2.25.1
maintainer
#4 lore raw

Re: [PATCH 2/2] dt-bindings: Consider DT_SCHEMA_FILES when finding all json-schema

Quoted 9 lines · Geert Uytterhoeven

Setting DT_SCHEMA_FILES allows the user to restrict the "dt_binding_check" make target to a specified set of DT binding files. However, yamllint is still run on all available files, which not only takes time, but also outputs warnings for other binding files the developer is not interested in.

Fix this by renaming "find_cmd" to "find_all_cmd", introducing a new "find_cmd" to only return the files specified by DT_SCHEMA_FILES (if present), and using the latter for yamllint.

We can also do the same thing on dt-doc-validate since checking and preprocessing schemas are separate steps, so I did while applying.

Quoted 10 lines

---
v2:
- Introduce find_all_cmd,
- Only use the restricted set for yamllint.

---
 Documentation/devicetree/bindings/Makefile | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)