dt-bindings: fix 'make clean' and improve dt-schema check rules
Subsystems (2)
cover dt-bindings: fix 'make clean' and improve dt-schema check rules
Masahiro Yamada (4):
dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
dt-bindings: do not build processed-schema.yaml for 'make
dt_binding_check'
dt-bindings: copy process-schema-examples.yaml to process-schema.yaml
dt-bindings: split DT schema check rules
.gitignore | 1 + Documentation/devicetree/bindings/Makefile | 35 ++++++++++++++++------ Makefile | 2 +- scripts/Makefile.build | 2 ++ scripts/Makefile.lib | 27 +++++++++-------- 5 files changed, 44 insertions(+), 23 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
We are having more and more schema files.
Commit 8b6b80218b01 ("dt-bindings: Fix command line length limit calling dt-mk-schema") fixed the 'Argument list too long' error of the schema checks, but the same error happens while cleaning too.
'make clean' after 'make dt_binding_check' fails as follows:
$ make dt_binding_check
[ snip ]
$ make clean
make[2]: execvp: /bin/sh: Argument list too long
make[2]: *** [scripts/Makefile.clean:52: __clean] Error 127
make[1]: *** [scripts/Makefile.clean:66: Documentation/devicetree/bindings] Error 2
make: *** [Makefile:1763: _clean_Documentation] Error 2'make dt_binding_check' generates so many .example.dts, .dt.yaml files, which are passed to the 'rm' command when you run 'make clean'.
I added a small hack to use the 'find' command to clean up most of the build artifacts before they are processed by scripts/Makefile.clean
---
Documentation/devicetree/bindings/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Documentation/devicetree/bindings/Makefile +5 -0
@@ -53,3 +53,8 @@ $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE $(call if_changed,mk_schema) extra-y += processed-schema.yaml + +# Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of +# build artifacts here before they are processed by scripts/Makefile.clean +clean-files = $(shell find $(obj) \( -name '*.example.dts' -o \ + -name '*.example.dt.yaml' \) -delete 2>/dev/null)
Signature · 1 line
2.25.1
Re: [PATCH 1/4] dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
Quoted 19 lines · Masahiro Yamada
We are having more and more schema files.
Commit
8b6b80218b01("dt-bindings: Fix command line length limit calling dt-mk-schema") fixed the 'Argument list too long' error of the schema checks, but the same error happens while cleaning too.'make clean' after 'make
dt_binding_check' fails as follows:$ make dt_binding_check [ snip ] $ make clean make[2]: execvp: /bin/sh: Argument list too long make[2]: *** [scripts/Makefile.clean:52: __clean] Error 127 make[1]: *** [scripts/Makefile.clean:66: Documentation/devicetree/bindings] Error 2 make: *** [Makefile:1763: _clean_Documentation] Error 2'make
dt_binding_check' generates so many.example.dts,.dt.yamlfiles, which are passed to the 'rm' command when you run 'make clean'.
Quoted 9 lines
I added a small hack to use the 'find' command to clean up most of the build artifacts before they are processed by
scripts/Makefile.cleanSigned-off-by Masahiro Yamada <[email protected]>---
Documentation/devicetree/bindings/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
Applied, thanks!
Re: [PATCH 1/4] dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
In reply to Masahiro YamadaDear Yamada-san,
Dear Kbuild experts,
Quoted 33 lines · Masahiro Yamada
We are having more and more schema files.
Commit
8b6b80218b01("dt-bindings: Fix command line length limit calling dt-mk-schema") fixed the 'Argument list too long' error of the schema checks, but the same error happens while cleaning too.'make clean' after 'make
dt_binding_check' fails as follows:$ make dt_binding_check [ snip ] $ make clean make[2]: execvp: /bin/sh: Argument list too long make[2]: *** [scripts/Makefile.clean:52: __clean] Error 127 make[1]: *** [scripts/Makefile.clean:66: Documentation/devicetree/bindings] Error 2 make: *** [Makefile:1763: _clean_Documentation] Error 2'make
dt_binding_check' generates so many.example.dts,.dt.yamlfiles, which are passed to the 'rm' command when you run 'make clean'.I added a small hack to use the 'find' command to clean up most of the build artifacts before they are processed by
scripts/Makefile.cleanSigned-off-by Masahiro Yamada <[email protected]>---
Documentation/devicetree/bindings/Makefile | 5 +++++ 1 file changed, 5 insertions(+)Documentation/devicetree/bindings/Makefile +0
-0
Documentation/devicetree/bindings/Makefile $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE · line 60
Quoted 8 lines
Documentation/devicetree/bindings/Makefile +5
-0@@ -53,3 +53,8 @@ $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE $(call if_changed,mk_schema) extra-y += processed-schema.yaml + +# Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of +# build artifacts here before they are processed by scripts/Makefile.clean +clean-files = $(shell find $(obj) \( -name '*.example.dts' -o \ + -name '*.example.dt.yaml' \) -delete 2>/dev/null)
JFYI, we are running into the "Argument list too long" simply by trying to clean an out-of-tree module (with lots of artifacts though), as below. Seems to be happening on vanilla (v6.4-rc6-199-gb73056e9f82ebd) too. Just in case there are any known hacks (or similar reports), please let us know.
$ make -C /data/linux-src M=my_module clean
make: Entering directory '/data/linux-src'
make -f ./scripts/Makefile.clean obj=my_module
set -e; echo ' CLEAN my_module'; rm -rf [~1k filenames / 200k char]
make[1]: /bin/sh: Argument list too long
make[1]: *** [scripts/Makefile.clean:45: __clean] Error 127
Signature · 2 lines
Best regards, Eugeniu Rosca
Re: [PATCH 1/4] dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
Quoted 52 lines · Eugeniu Rosca
Dear Yamada-san,
DearKbuildexperts,Quoted 33 lines · Masahiro Yamada
We are having more and more schema files.
Commit
8b6b80218b01("dt-bindings: Fix command line length limit calling dt-mk-schema") fixed the 'Argument list too long' error of the schema checks, but the same error happens while cleaning too.'make clean' after 'make
dt_binding_check' fails as follows:$ make dt_binding_check [ snip ] $ make clean make[2]: execvp: /bin/sh: Argument list too long make[2]: *** [scripts/Makefile.clean:52: __clean] Error 127 make[1]: *** [scripts/Makefile.clean:66: Documentation/devicetree/bindings] Error 2 make: *** [Makefile:1763: _clean_Documentation] Error 2'make
dt_binding_check' generates so many.example.dts,.dt.yamlfiles, which are passed to the 'rm' command when you run 'make clean'.I added a small hack to use the 'find' command to clean up most of the build artifacts before they are processed by
scripts/Makefile.cleanSigned-off-by Masahiro Yamada <[email protected]>---
Documentation/devicetree/bindings/Makefile | 5 +++++ 1 file changed, 5 insertions(+)Documentation/devicetree/bindings/Makefile +0
-0Documentation/devicetree/bindings/Makefile $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE · line 60
Quoted 8 lines
Documentation/devicetree/bindings/Makefile +5
-0@@ -53,3 +53,8 @@ $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE $(call if_changed,mk_schema) extra-y += processed-schema.yaml + +# Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of +# build artifacts here before they are processed by scripts/Makefile.clean +clean-files = $(shell find $(obj) \( -name '*.example.dts' -o \ + -name '*.example.dt.yaml' \) -delete 2>/dev/null)JFYI, we are running into the "Argument list too long" simply by trying to clean an out-of-tree module (with lots of artifacts though), as below. Seems to be happening on vanilla (v6.4-rc6-199-gb73056e9f82ebd) too. Just in case there are any known hacks (or similar reports), please let us know.
Quoted 7 lines
$ make -C /data/linux-src M=my_module cleanmake: Entering directory '/data/linux-src'
make -f./scripts/Makefile.cleanobj=my_module
set -e; echo ' CLEANmy_module'; rm -rf [~1k filenames / 200k char]
make[1]:/bin/sh: Argument list too long
make[1]: *** [scripts/Makefile.clean:45:__clean] Error 127
I sumitted a patch
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/
I do not know whether it will solve your problem or not.
Your module Makefile has ~1k files to remove.
It may still exceed ARG_MAX limit.
Quoted 3 lines
--
Best regards,
Eugeniu Rosca
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [PATCH 1/4] dt-bindings: fix error in 'make clean' after 'make dt_binding_check'
Hello Yamada-san,
On Sun, Jun 18, 2023 at 03:09:56AM +0900, Masahiro Yamada wrote:
[snip]
Thank you very much. Provided my comments in the new thread.
Signature · 2 lines
Best regards, Eugeniu Rosca
PATCH 2/4 dt-bindings: do not build processed-schema.yaml for 'make dt_binding_check'
In reply to Masahiro YamadaCurrently, processed-schema.yaml is always built, but it is actually used only for 'make dtbs_check'.
'make dt_binding_check' uses processed-schema-example.yaml instead.
Build processed-schema.yaml only for 'make dtbs_check'.
---
Documentation/devicetree/bindings/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Documentation/devicetree/bindings/Makefile +1 -1
@@ -52,7 +52,7 @@ $(obj)/processed-schema.yaml: DT_MK_SCHEMA_FLAGS := $(DT_MK_SCHEMA_USERONLY_FLAG $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE $(call if_changed,mk_schema) -extra-y += processed-schema.yaml +extra-$(CHECK_DTBS) += processed-schema.yaml # Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of # build artifacts here before they are processed by scripts/Makefile.clean
Signature · 1 line
2.25.1
Re: [PATCH 2/4] dt-bindings: do not build processed-schema.yaml for 'make dt_binding_check'
Quoted 5 lines · Masahiro Yamada
Currently,
processed-schema.yamlis always built, but it is actually used only for 'makedtbs_check'.'make
dt_binding_check' usesprocessed-schema-example.yamlinstead.
Quoted 8 lines
Build
processed-schema.yamlonly for 'makedtbs_check'.Signed-off-by Masahiro Yamada <[email protected]>---
Documentation/devicetree/bindings/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied, thanks!
PATCH 3/4 dt-bindings: copy process-schema-examples.yaml to process-schema.yaml
In reply to Masahiro YamadaThere are two processed schema files:
- processed-schema-examples.yaml
Used for 'make dt_binding_check'. This is always a full schema.
- processed-schema.yaml
Used for 'make dtbs_check'. This may be a full schema, or a smaller subset if DT_SCHEMA_FILES is given by a user.
If DT_SCHEMA_FILES is not specified, they are the same. You can copy the former to the latter instead of running dt-mk-schema twice. This saves the cpu time a lot when you do 'make dt_binding_check dtbs_check' because building the full schema takes a couple of seconds.
If DT_SCHEMA_FILES is specified, processed-schema.yaml is generated based on the specified yaml files.
---
Documentation/devicetree/bindings/Makefile | 28 +++++++++++++++------- scripts/Makefile.lib | 3 +++ 2 files changed, 23 insertions(+), 8 deletions(-)
Documentation/devicetree/bindings/Makefile +20 -8
@@ -2,7 +2,6 @@ DT_DOC_CHECKER ?= dt-doc-validate DT_EXTRACT_EX ?= dt-extract-example DT_MK_SCHEMA ?= dt-mk-schema -DT_MK_SCHEMA_USERONLY_FLAG := $(if $(DT_SCHEMA_FILES), -u) DT_SCHEMA_MIN_VERSION = 2020.5
@@ -35,12 +34,6 @@ quiet_cmd_mk_schema = SCHEMA $@ DT_DOCS = $(shell $(find_cmd) | sed -e 's|^$(srctree)/||') -DT_SCHEMA_FILES ?= $(DT_DOCS) - -extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dts, $(DT_SCHEMA_FILES)) -extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dt.yaml, $(DT_SCHEMA_FILES)) -extra-$(CHECK_DT_BINDING) += processed-schema-examples.yaml - override DTC_FLAGS := \ -Wno-avoid_unnecessary_addr_size \ -Wno-graph_child_address
@@ -48,10 +41,29 @@ override DTC_FLAGS := \ $(obj)/processed-schema-examples.yaml: $(DT_DOCS) check_dtschema_version FORCE $(call if_changed,mk_schema) -$(obj)/processed-schema.yaml: DT_MK_SCHEMA_FLAGS := $(DT_MK_SCHEMA_USERONLY_FLAG) +ifeq ($(DT_SCHEMA_FILES),) + +# Unless DT_SCHEMA_FILES is specified, use the full schema for dtbs_check too. +# Just copy processed-schema-examples.yaml + +$(obj)/processed-schema.yaml: $(obj)/processed-schema-examples.yaml FORCE + $(call if_changed,copy) + +DT_SCHEMA_FILES = $(DT_DOCS) + +else + +# If DT_SCHEMA_FILES is specified, use it for processed-schema.yaml + +$(obj)/processed-schema.yaml: DT_MK_SCHEMA_FLAGS := -u $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE $(call if_changed,mk_schema) +endif + +extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dts, $(DT_SCHEMA_FILES)) +extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dt.yaml, $(DT_SCHEMA_FILES)) +extra-$(CHECK_DT_BINDING) += processed-schema-examples.yaml extra-$(CHECK_DTBS) += processed-schema.yaml # Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of
scripts/Makefile.lib +3 -0
@@ -212,6 +212,9 @@ $(foreach m, $(notdir $1), \ $(addprefix $(obj)/, $(foreach s, $3, $($(m:%$(strip $2)=%$(s))))))) endef +quiet_cmd_copy = COPY $@ + cmd_copy = cp $< $@ + # Shipped files # ===========================================================================
Signature · 1 line
2.25.1
Re: [PATCH 3/4] dt-bindings: copy process-schema-examples.yaml to process-schema.yaml
Quoted 19 lines · Masahiro Yamada
There are two processed schema files:
-
processed-schema-examples.yamlUsed for 'make
dt_binding_check'. This is always a full schema.-
processed-schema.yamlUsed for 'make
dtbs_check'. This may be a full schema, or a smaller subset ifDT_SCHEMA_FILESis given by a user.If
DT_SCHEMA_FILESis not specified, they are the same. You can copy the former to the latter instead of running dt-mk-schema twice. This saves the cpu time a lot when you do 'makedt_binding_checkdtbs_check' because building the full schema takes a couple of seconds.If
DT_SCHEMA_FILESis specified,processed-schema.yamlis generated based on the specified yaml files.
Quoted 7 lines
Signed-off-by Masahiro Yamada <[email protected]>---
Documentation/devicetree/bindings/Makefile | 28 +++++++++++++++------- scripts/Makefile.lib | 3 +++ 2 files changed, 23 insertions(+), 8 deletions(-)
Applied, thanks!
PATCH 4/4 dt-bindings: split DT schema check rules
In reply to Masahiro YamadaWhen building %.dt.yaml from %.dts, two things happen in a row:
[1] Run DTC to convert %.dts into %.dt.yaml
[2] Run dt-validate against %.dt.yaml
Currently, when any .yaml schema file is updated, processed-schema.yaml is regenerated, then both [1] and [2] are rerun for all .dts files.
Actually, we do not need to rerun [1] since the original .dts is not updated.
Split the rule to avoid unneeded .dts/.dt.yaml conversion.
---
.gitignore | 1 + Documentation/devicetree/bindings/Makefile | 4 ++-- Makefile | 2 +- scripts/Makefile.build | 2 ++ scripts/Makefile.lib | 24 ++++++++++------------ 5 files changed, 17 insertions(+), 16 deletions(-)
.gitignore +1 -0
@@ -16,6 +16,7 @@ *.bin *.bz2 *.c.[012]*.* +*.dt.checked *.dt.yaml *.dtb *.dtb.S
Documentation/devicetree/bindings/Makefile +2 -2
@@ -62,11 +62,11 @@ $(obj)/processed-schema.yaml: $(DT_SCHEMA_FILES) check_dtschema_version FORCE endif extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dts, $(DT_SCHEMA_FILES)) -extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dt.yaml, $(DT_SCHEMA_FILES)) +extra-$(CHECK_DT_BINDING) += $(patsubst $(src)/%.yaml,%.example.dt.checked, $(DT_SCHEMA_FILES)) extra-$(CHECK_DT_BINDING) += processed-schema-examples.yaml extra-$(CHECK_DTBS) += processed-schema.yaml # Hack: avoid 'Argument list too long' error for 'make clean'. Remove most of # build artifacts here before they are processed by scripts/Makefile.clean clean-files = $(shell find $(obj) \( -name '*.example.dts' -o \ - -name '*.example.dt.yaml' \) -delete 2>/dev/null) + -name '*.example.dt.checked' \) -delete 2>/dev/null)
Makefile +1 -1
@@ -1774,7 +1774,7 @@ clean: $(clean-dirs) -o -name '*.lex.c' -o -name '*.tab.[ch]' \ -o -name '*.asn1.[ch]' \ -o -name '*.symtypes' -o -name 'modules.order' \ - -o -name '.tmp_*.o.*' \ + -o -name '.tmp_*.o.*' -o -name '*.dt.checked' \ -o -name '*.c.[012]*.*' \ -o -name '*.ll' \ -o -name '*.gcno' \) -type f -print | xargs rm -f
scripts/Makefile.build +2 -0
@@ -442,10 +442,12 @@ intermediate_targets = $(foreach sfx, $(2), \ $(filter %$(strip $(1)), $(targets)))) # %.asn1.o <- %.asn1.[ch] <- %.asn1 # %.dtb.o <- %.dtb.S <- %.dtb <- %.dts +# %.dt.checked <- %.dt.yaml <- %.dts # %.lex.o <- %.lex.c <- %.l # %.tab.o <- %.tab.[ch] <- %.y targets += $(call intermediate_targets, .asn1.o, .asn1.c .asn1.h) \ $(call intermediate_targets, .dtb.o, .dtb.S .dtb) \ + $(call intermediate_targets, .dt.checked, .dt.yaml) \ $(call intermediate_targets, .lex.o, .lex.c) \ $(call intermediate_targets, .tab.o, .tab.c .tab.h)
scripts/Makefile.lib +11 -13
@@ -74,8 +74,8 @@ extra-y += $(dtb-y) extra-$(CONFIG_OF_ALL_DTBS) += $(dtb-) ifneq ($(CHECK_DTBS),) -extra-y += $(patsubst %.dtb,%.dt.yaml, $(dtb-y)) -extra-$(CONFIG_OF_ALL_DTBS) += $(patsubst %.dtb,%.dt.yaml, $(dtb-)) +extra-y += $(patsubst %.dtb,%.dt.checked, $(dtb-y)) +extra-$(CONFIG_OF_ALL_DTBS) += $(patsubst %.dtb,%.dt.checked, $(dtb-)) endif # Add subdir path
@@ -308,26 +308,24 @@ cmd_dtc = mkdir -p $(dir ${dtc-tmp}) ; \ -d $(depfile).dtc.tmp $(dtc-tmp) ; \ cat $(depfile).pre.tmp $(depfile).dtc.tmp > $(depfile) +dtc-tmp = $(subst $(comma),_,$(dot-target).dts.tmp) + $(obj)/%.dtb: $(src)/%.dts $(DTC) FORCE $(call if_changed_dep,dtc) +$(obj)/%.dt.yaml: $(src)/%.dts $(DTC) FORCE + $(call if_changed_dep,dtc) + DT_CHECKER ?= dt-validate DT_BINDING_DIR := Documentation/devicetree/bindings # DT_TMP_SCHEMA may be overridden from Documentation/devicetree/bindings/Makefile DT_TMP_SCHEMA ?= $(objtree)/$(DT_BINDING_DIR)/processed-schema.yaml -quiet_cmd_dtb_check = CHECK $@ - cmd_dtb_check = $(DT_CHECKER) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $@ +quiet_cmd_dt_check = CHECK $< + cmd_dt_check = $(DT_CHECKER) -u $(srctree)/$(DT_BINDING_DIR) -p $(DT_TMP_SCHEMA) $<; touch $@ -define rule_dtc - $(call cmd_and_fixdep,dtc) - $(call cmd,dtb_check) -endef - -$(obj)/%.dt.yaml: $(src)/%.dts $(DTC) $(DT_TMP_SCHEMA) FORCE - $(call if_changed_rule,dtc,yaml) - -dtc-tmp = $(subst $(comma),_,$(dot-target).dts.tmp) +$(obj)/%.dt.checked: $(obj)/%.dt.yaml $(DT_TMP_SCHEMA) FORCE + $(call if_changed,dt_check) # Bzip2 # ---------------------------------------------------------------------------
Signature · 1 line
2.25.1
Re: [PATCH 4/4] dt-bindings: split DT schema check rules
Quoted 5 lines · Masahiro Yamada
When building %.dt.yaml from %.dts, two things happen in a row:
[1] Run DTC to convert %.dts into %.dt.yaml
Quoted 7 lines
[2] Run dt-validate against %.dt.yaml
Currently, when any .yaml schema file is updated,
processed-schema.yamlis regenerated, then both [1] and [2] are rerun for all .dts files.Actually, we do not need to rerun [1] since the original .dts is not updated.
I have plans (and an intern working on it) to integrate the schema checks into dtc. That's going to make turning on the schema checks just a flag to dtc. I'm not sure if adding the complexity here is worth it as I'd expect much of this patch to go away again.
Is there any negative impact on the absolute clean build time? I'm more concerned about that than optimizing rerunning.
Rob
Re: [PATCH 4/4] dt-bindings: split DT schema check rules
Quoted 15 lines · Rob Herring
Quoted 5 lines · Masahiro Yamada
When building %.dt.yaml from %.dts, two things happen in a row:
[1] Run DTC to convert %.dts into %.dt.yaml
Quoted 7 lines
[2] Run dt-validate against %.dt.yaml
Currently, when any .yaml schema file is updated,
processed-schema.yamlis regenerated, then both [1] and [2] are rerun for all .dts files.Actually, we do not need to rerun [1] since the original .dts is not updated.
Quoted 7 lines
I have plans (and an intern working on it) to integrate the schema checks into dtc. That's going to make turning on the schema checks just a flag to dtc. I'm not sure if adding the complexity here is worth it as I'd expect much of this patch to go away again.
Is there any negative impact on the absolute clean build time? I'm more concerned about that than optimizing rerunning.
No benefit on the absolute clean build time.
OK, then please skip this patch.
Signature · 2 lines
Best Regards Masahiro Yamada