Makefile: With BINMAN_ALLOW_MISSING=1 don't error on missing
PATCH 1/1 Makefile: With BINMAN_ALLOW_MISSING=1 don't error on missing
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \ $(foreach f,$(BINMAN_INDIRS),-I $(f)) \
Signature · 1 line
2.25.1
Hi Tom,
Quoted 30 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-2@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \ $(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --
Quoted 2 lines
2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
Yes I know this passes on CI, but it will cause breakages when people use mkimage or other things which have missing inputs. It will be quite confusing too!
As to the logic, I thought you had wanted the build to fail if there are missing blobs, regardless of whether they were allowed or not. There is currently code in buildman to detect this failure and either report it or ignore it:
if (result.return_code == 2 and
('Some images are invalid' in result.stderr)):
# This is handled later by the check for output in # stderr
result.return_code = 0
Since buildman sets BINMAN_ALLOW_MISSING=1 if -M is given, we will not be able to detect missing binaries at all when built from buildman. I think a suitable fix would be to update the code above to detect the 'Some images are invalid' regardless of the return code.
Regards,
Simon
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \
Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in the binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?
Quoted 15 lines
Yes I know this passes on CI, but it will cause breakages when people use mkimage or other things which have missing inputs. It will be quite confusing too!
As to the logic, I thought you had wanted the build to fail if there are missing blobs, regardless of whether they were allowed or not. There is currently code in buildman to detect this failure and either report it or ignore it:
if (result.return_code == 2 and
('Some images are invalid' in result.stderr)):
# This is handled later by the check for output in # stderr
result.return_code = 0
Quoted 4 lines
Since buildman sets BINMAN_ALLOW_MISSING=1 if -M is given, we will not be able to detect missing binaries at all when built from buildman. I think a suitable fix would be to update the code above to detect the 'Some images are invalid' regardless of the return code.
Well, what we designed here first missed how the allwinner case is used by Fedora. That needs both a zero exit code and allowing what turns out to be an optional blob to be missing.
Signature · 1 line
Tom
Quoted 54 lines · Tom Rini
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in the
Quoted 8 lines
binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?
Simon, can you test this last case by chance? I think that's the one that's unclear about if this patch breaks or not and I want to know if I need to respin to still include the fake blobs flag still too or not. Or does anyone else use this case and can test?
Signature · 1 line
Tom
Hi Tom,
Quoted 65 lines · Tom Rini
Quoted 54 lines · Tom Rini
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in theQuoted 8 lines
binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?
Quoted 4 lines
Simon, can you test this last case by chance? I think that's the one that's unclear about if this patch breaks or not and I want to know if I need to respin to still include the fake blobs flag still too or not. Or does anyone else use this case and can test?
Yes I have your email printed out but travel and family things have made it hard to get to it. I do think you need the flag, but the extra case you mention need another look and probably a test, as you say.
Regards,
Simon
Quoted 68 lines · Simon Glass
Hi Tom,
Quoted 65 lines · Tom Rini
Quoted 62 lines · Tom Rini
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in the binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?
Quoted 8 lines
Quoted 4 lines
Simon, can you test this last case by chance? I think that's the one that's unclear about if this patch breaks or not and I want to know if I need to respin to still include the fake blobs flag still too or not. Or does anyone else use this case and can test?
Yes I have your email printed out but travel and family things have made it hard to get to it. I do think you need the flag, but the extra case you mention need another look and probably a test, as you say.
Yeah, understandable. My plan is to take this patch before the next -rc, so Fedora won't be broken and if we need the other flag (and I'll check in before release) we can follow up there and have a fixes tag on top of this one here.
Signature · 1 line
Tom
Hi Tom,
Quoted 75 lines · Tom Rini
Quoted 73 lines · Simon Glass
Hi Tom,
Quoted 69 lines · Tom Rini
Quoted 54 lines · Tom Rini
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in theQuoted 8 lines
binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?Simon, can you test this last case by chance? I think that's the one that's unclear about if this patch breaks or not and I want to know if I need to respin to still include the fake blobs flag still too or not. Or does anyone else use this case and can test?
Quoted 8 lines
Quoted 3 lines
Yes I have your email printed out but travel and family things have made it hard to get to it. I do think you need the flag, but the extra case you mention need another look and probably a test, as you say.
Yeah, understandable. My plan is to take this patch before the next -rc, so Fedora won't be broken and if we need the other flag (and I'll check in before release) we can follow up there and have a fixes tag on top of this one here.
I just sent a patch that allows entries (with external blobs) to be marked as optional. I believe this covers the remaining case.
You still need to keep --fake-ext-blobs (in this case) since otherwise boards that rely on faked blobs will not build - e.g. will get an error in the mkimage step if an input file is missing.
Regards,
Simon
Quoted 87 lines · Simon Glass
Hi Tom,
Quoted 75 lines · Tom Rini
Quoted 73 lines · Simon Glass
Hi Tom,
Quoted 69 lines · Tom Rini
Quoted 54 lines · Tom Rini
Quoted 31 lines · Simon Glass
Hi Tom,
Quoted 28 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Cc Simon Glass <[email protected]>Reported-by Peter Robinson <[email protected]>Signed-off-by Tom Rini <[email protected]>---
This passes CI as-is:
https://source.denx.de/u-boot/u-boot/-/pipelines/14340--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Makefile +1
-1@@ -1334,7 +1334,7 @@ cmd_binman = $(srctree)/tools/binman/binman $(if $(BINMAN_DEBUG),-D) \ --toolpath $(objtree)/tools \ $(if $(BINMAN_VERBOSE),-v$(BINMAN_VERBOSE)) \ build -u -d u-boot.dtb -O . -m \ - $(if $(BINMAN_ALLOW_MISSING),--allow-missing --fake-ext-blobs) \ + $(if $(BINMAN_ALLOW_MISSING),--allow-missing --ignore-missing) \ -I . -I $(srctree) -I $(srctree)/board/$(BOARDDIR) \ -I arch/$(ARCH)/dts -a of-list=$(CONFIG_OF_LIST) \Quoted 8 lines
Quoted 4 lines
Makefile +0
-1$(foreach f,$(BINMAN_INDIRS),-I $(f)) \ --2.25.1
I believe we need the --fake-ext-blobs flag as well, since otherwise boards which use tools (like mkimage) on things that don't exist will not work.
So, we need to list out all the use cases perhaps, as we had missed one before. In my mind we have:
- Board requires 1 or more blobs, developer will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. Blobs must be present or non-zero exist status by default. We didn't have this before, and we do now.
- Board requires 1 or more blobs AND has optional blobs, will be running this on hardware, expects output from U-Boot 'make' (or buildman) to boot. This is the case we had missed before as allwinner requires bl31 and has optional PMIC firmware. This is the case Peter reported and we had missed before, which this patch allows for, with the caveat that if you forget BL31 you're not going to boot and make won't complain exit-code wise. Another way to resolve this would be a property in theQuoted 8 lines
binman node to mark a blob as optional and warn if missing, rather than error if missing.
- Board requires 1 or more blobs, output will NOT be run. This is the CI case and the compile testing lots of board cases. This is what CI passes still, with the above.
- Board requires 1 or more blobs, developer will be running this on hardware, BUT will be injecting the blobs later on. I think this is the use case you're talking about?Simon, can you test this last case by chance? I think that's the one that's unclear about if this patch breaks or not and I want to know if I need to respin to still include the fake blobs flag still too or not. Or does anyone else use this case and can test?
Quoted 8 lines
Quoted 3 lines
Yes I have your email printed out but travel and family things have made it hard to get to it. I do think you need the flag, but the extra case you mention need another look and probably a test, as you say.
Yeah, understandable. My plan is to take this patch before the next -rc, so Fedora won't be broken and if we need the other flag (and I'll check in before release) we can follow up there and have a fixes tag on top of this one here.
Quoted 6 lines
I just sent a patch that allows entries (with external blobs) to be marked as optional. I believe this covers the remaining case.
You still need to keep --fake-ext-blobs (in this case) since otherwise boards that rely on faked blobs will not build - e.g. will get an error in the mkimage step if an input file is missing.
OK thanks, we'll pick that up after the next rc.
Signature · 1 line
Tom
Quoted 6 lines · Tom Rini
When the user builds with BINMAN_ALLOW_MISSING=1 they're explicitly setting the flag to allow for additional binaries to be missing and so have acknowledged the output might not work. In this case we want to default to not passing a non-zero exit code.
Applied to u-boot/master, thanks!
Signature · 1 line
Tom