Makefile: Deal with missing blobs consistently
v2 cover Makefile: Deal with missing blobs consistently
Missing blobs should cause the build (with make) to fail, but at present success is returned. This is because binman currently produces an exit code of 0 in this case.
Of course this is not correct, since the images cannot actually be used.
This series fixes that and adjusts buildman to deal sensibly with the situation.
Changes in v2:
- Fix missing usage of the msg variable
- Simplify code to remove the 'else'
Simon Glass (5):
buildman: Handle the MAINTAINERS 'N' tag
Makefile: Correct a missing FORCE on the binman rule
doc: Correct the path to the Makefile documentation
binman: Use an exit code when blobs are missing
buildman: Detect binman reporting missing blobs
Makefile | 2 +- scripts/Kbuild.include | 2 +- tools/binman/binman.rst | 4 ++++ tools/binman/cmdline.py | 3 +++ tools/binman/control.py | 6 +++++- tools/buildman/boards.py | 11 +++++++++++ tools/buildman/builderthread.py | 6 +++++- 7 files changed, 30 insertions(+), 4 deletions(-)
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
PATCH v2 1/5 buildman: Handle the MAINTAINERS 'N' tag
This is needed for some soon-to-be-applied patches. Scan the configs/ directory to see if any of the files match.
---
(no changes since v1)
tools/buildman/boards.py | 11 +++++++++++ 1 file changed, 11 insertions(+)
tools/buildman/boards.py +11 -0
@@ -368,6 +368,17 @@ class MaintainersDatabase: targets.append(front) elif tag == 'S:': status = rest + elif tag == 'N:': + # Just scan the configs directory since that's all we care + # about + for dirpath, _, fnames in os.walk('configs'): + for fname in fnames: + path = os.path.join(dirpath, fname) + front, match, rear = path.partition('configs/') + if not front and match: + front, match, rear = rear.rpartition('_defconfig') + if match and not rear: + targets.append(front) elif line == '\n': for target in targets: self.database[target] = (status, maintainers)
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Re: [PATCH v2 1/5] buildman: Handle the MAINTAINERS 'N' tag
Quoted 6 lines · Simon Glass
This is needed for some soon-to-be-applied patches. Scan the configs/ directory to see if any of the files match.
Signed-off-by Simon Glass <[email protected]>Tested-by Tom Rini <[email protected]>Suggested-by Tom Rini <[email protected]>
Applied to u-boot/master, thanks!
Signature · 1 line
Tom
PATCH v2 2/5 Makefile: Correct a missing FORCE on the binman rule
In reply to Simon GlassThis is required for if_changed to work correctly. Add it.
---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman) endif
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0
Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)
'all' is usually used as a phony target.
I think this went wrong.
if_changed should never be used for a phony target. In other words, if_changed should produce a build artifact.
FORCE is never used for a phony target because
it is added to 'PHONY'.
PHONY += all
Quoted 4 lines
Makefile +0
-1endif --2.38.0.rc1.362.ged0d419d3c-goog
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Quoted 38 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.
Quoted 6 lines
FORCE is never used for a phony target because
it is added to 'PHONY'.PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
And binman call should be moved to different non-phony target with the correct output file name.
Quoted 7 lines
Quoted 7 lines
Makefile +0
-1endif --2.38.0.rc1.362.ged0d419d3c-goog
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Hi Pali,
Quoted 43 lines · Pali Rohár
Quoted 41 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.FORCE is never used for a phony target because
it is added to 'PHONY'.
Quoted 8 lines
Quoted 3 lines
PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
OK, will need to figure out where to put the binman thing then.
Quoted 3 lines
And binman call should be moved to different non-phony target with the correct output file name.
Binman can generate all sorts of files. I will see if I can use .binman-stamp since that is always generated.
Regards,
Simon
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Quoted 58 lines · On Wednesday 19 October 2022 19:55:43 Simon Glass wrote:
Hi Pali,
Quoted 43 lines · Pali Rohár
Quoted 41 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.FORCE is never used for a phony target because
it is added to 'PHONY'.Quoted 8 lines
Quoted 3 lines
PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
OK, will need to figure out where to put the binman thing then.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
And binman call should be moved to different non-phony target with the correct output file name.
Binman can generate all sorts of files. I will see if I can use .binman-stamp since that is always generated.
Regards,
Simon
Would not something like this work?
BINMAN_INPUTS := .... # fill thisBINMAN_OUTPUTS := .... # fill this
INPUTS-$(CONFIG_BINMAN) += .binman_stamp
.binman_stamp: $(BINMAN_INPUTS) FORCE
$(call if_changed,binman)
$(BINMAN_OUTPUTS): .binman_stamp
all: inputs
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Hi Pali,
Quoted 74 lines · Pali Rohár
Quoted 58 lines · On Wednesday 19 October 2022 19:55:43 Simon Glass wrote:
Hi Pali,
Quoted 43 lines · Pali Rohár
Quoted 41 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.FORCE is never used for a phony target because
it is added to 'PHONY'.Quoted 8 lines
Quoted 3 lines
PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
OK, will need to figure out where to put the binman thing then.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
And binman call should be moved to different non-phony target with the correct output file name.
Binman can generate all sorts of files. I will see if I can use .binman-stamp since that is always generated.
Regards,
SimonWould not something like this work?
BINMAN_INPUTS:= .... # fill thisBINMAN_OUTPUTS:= .... # fill this
Quoted 8 lines
INPUTS-$(CONFIG_BINMAN) += .binman_stamp
.binman_stamp: $(BINMAN_INPUTS) FORCE
$(call if_changed,binman)$(BINMAN_OUTPUTS): .binman_stamp
all: inputs
Sadly we don't know the outputs without asking binman.
I'll send out another rev of this patch by the end of the week.
Regards,
Simon
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Quoted 79 lines · On Friday 21 October 2022 14:17:11 Simon Glass wrote:
Hi Pali,
Quoted 76 lines · Pali Rohár
Quoted 58 lines · On Wednesday 19 October 2022 19:55:43 Simon Glass wrote:
Hi Pali,
Quoted 43 lines · Pali Rohár
Quoted 41 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.FORCE is never used for a phony target because
it is added to 'PHONY'.Quoted 8 lines
Quoted 3 lines
PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
OK, will need to figure out where to put the binman thing then.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
And binman call should be moved to different non-phony target with the correct output file name.
Binman can generate all sorts of files. I will see if I can use .binman-stamp since that is always generated.
Regards,
SimonWould not something like this work?
BINMAN_INPUTS:= .... # fill thisBINMAN_OUTPUTS:= .... # fill thisINPUTS-$(CONFIG_BINMAN) += .binman_stamp
Quoted 8 lines
Quoted 6 lines
.binman_stamp: $(BINMAN_INPUTS) FORCE
$(call if_changed,binman)$(BINMAN_OUTPUTS): .binman_stamp
all: inputs
Sadly we don't know the outputs without asking binman.
So what about extending binman via some cmdline arg to just prints outputs?
Quoted 4 lines
I'll send out another rev of this patch by the end of the week.
Regards,
Simon
Re: [PATCH v2 2/5] Makefile: Correct a missing FORCE on the binman rule
Hi Pali,
Quoted 84 lines · Pali Rohár
Quoted 82 lines · On Friday 21 October 2022 14:17:11 Simon Glass wrote:
Hi Pali,
Quoted 79 lines · Pali Rohár
Quoted 58 lines · On Wednesday 19 October 2022 19:55:43 Simon Glass wrote:
Hi Pali,
Quoted 43 lines · Pali Rohár
Quoted 41 lines · On Tuesday 11 October 2022 23:35:22 Masahiro Yamada wrote:
Quoted 17 lines · Simon Glass
This is required for
if_changedto work correctly. Add it.Signed-off-by Simon Glass <[email protected]>Reviewed-by Pali Rohár <[email protected]>---
(no changes since v1)
Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +0
-0Makefile endef · line 1116
Quoted 7 lines
Makefile +1
-1@@ -1111,7 +1111,7 @@ endef PHONY += inputs inputs: $(INPUTS-y) -all: .binman_stamp inputs +all: .binman_stamp inputs FORCE ifeq ($(CONFIG_BINMAN),y) $(call if_changed,binman)'all' is usually used as a phony target.
I think this went wrong.
if_changedshould never be used for a phony target. In other words,if_changedshould produce a build artifact.FORCE is never used for a phony target because
it is added to 'PHONY'.Quoted 8 lines
Quoted 3 lines
PHONY += all
I agree, this is really written in the wrong way.
Target "all:" should be really phony target and should only depends on other target, it should not have any body / commands.
OK, will need to figure out where to put the binman thing then.
Quoted 1 line
Quoted 8 lines
Quoted 2 lines
And binman call should be moved to different non-phony target with the correct output file name.
Binman can generate all sorts of files. I will see if I can use .binman-stamp since that is always generated.
Regards,
SimonWould not something like this work?
BINMAN_INPUTS:= .... # fill thisBINMAN_OUTPUTS:= .... # fill thisINPUTS-$(CONFIG_BINMAN) += .binman_stamp
.binman_stamp: $(BINMAN_INPUTS) FORCE
$(call if_changed,binman)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
$(BINMAN_OUTPUTS): .binman_stamp
all: inputs
Sadly we don't know the outputs without asking binman.
So what about extending binman via some cmdline arg to just prints outputs?
Yes I've thought about that but not done it. Things to do:
- put intermediate files in a separate output subdir from the final output
- have a way to determine the output filenames without actually building the output
Quoted 2 lines
Quoted 2 lines
I'll send out another rev of this patch by the end of the week.
Regards,
Simon
PATCH v2 3/5 doc: Correct the path to the Makefile documentation
In reply to Simon GlassThis is out-of-date now. Fix it.
---
(no changes since v1)
scripts/Kbuild.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Kbuild.include +1 -1
@@ -229,7 +229,7 @@ objectify = $(foreach o,$(1),$(if $(filter /%,$(o)),$(o),$(obj)/$(o))) # if_changed_dep - as if_changed, but uses fixdep to reveal dependencies # including used config symbols # if_changed_rule - as if_changed but execute rule instead -# See Documentation/kbuild/makefiles.txt for more info +# See doc/develop/makefiles.rst for more info ifneq ($(KBUILD_NOCMDDEP),1) # Check if both arguments are the same including their order. Result is empty
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
PATCH v2 4/5 binman: Use an exit code when blobs are missing
In reply to Simon GlassAt present binman returns success when told to handle missing blobs. This is confusing this in fact the resulting image cannot work.
Use exit code 103 to signal this problem, with a -W option to convert it to a warning.
---
Changes in v2:
- Fix missing usage of the msg variable
- Simplify code to remove the 'else'
tools/binman/binman.rst | 4 ++++ tools/binman/cmdline.py | 3 +++ tools/binman/control.py | 6 +++++- 3 files changed, 12 insertions(+), 1 deletion(-)
tools/binman/binman.rst +4 -0
@@ -1458,6 +1458,10 @@ space-separated list of directories to search for binary blobs:: odroid-c4/build/board/hardkernel/odroidc4/firmware \ odroid-c4/build/scp_task" binman ... +Note that binman fails with error code 103 when there are missing blobs. If you +wish to continue anyway, you can pass `-W` to binman. + + Code coverage -------------
tools/binman/cmdline.py +3 -0
@@ -128,6 +128,9 @@ controlled by a description in the board device tree.''' default=False, help='Update the binman node with offset/size info') build_parser.add_argument('--update-fdt-in-elf', type=str, help='Update an ELF file with the output dtb: infile,outfile,begin_sym,end_sym') + build_parser.add_argument( + '-W', '--ignore-missing-blobs', action='store_true', default=False, + help='Return success even if there are missing blobs') subparsers.add_parser( 'bintool-docs', help='Write out bintool documentation (see bintool.rst)')
tools/binman/control.py +5 -1
@@ -742,7 +742,11 @@ def Binman(args): elf.UpdateFile(*elf_params, data) if invalid: - tout.warning("\nSome images are invalid") + msg = '\nSome images are invalid' + if not args.ignore_missing_blobs: + tout.error(msg) + return 103 + tout.warning(msg) # Use this to debug the time take to pack the image #state.TimingShow()
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog
PATCH v2 5/5 buildman: Detect binman reporting missing blobs
In reply to Simon GlassBuildman should consider a build as a success (with warnings) if missing blobs have been dealt with by binman, even though buildman itself returns and error code overall. This is how other warnings are dealt with.
We cannot easily access the 103 exit code, so detect the problem in the output.
With this change, missing blobs result in an exit code of 101, although they still indicate failure.
---
(no changes since v1)
tools/buildman/builderthread.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
tools/buildman/builderthread.py +5 -1
@@ -288,10 +288,14 @@ class BuilderThread(threading.Thread): args.append('cfg') result = self.Make(commit, brd, 'build', cwd, *args, env=env) + if (result.return_code == 2 and + ('Some images are invalid' in result.stderr)): + # This is handled later by the check for output in + # stderr + result.return_code = 0 if adjust_cfg: errs = cfgutil.check_cfg_file(cfg_file, adjust_cfg) if errs: - print('errs', errs) result.stderr += errs result.return_code = 1 result.stderr = result.stderr.replace(src_dir + '/', '')
Signature · 1 line
2.38.0.rc1.362.ged0d419d3c-goog