fetch2: local files only in DL_DIR becomes fatal error
PATCH 1/2 fetch2: local files only in DL_DIR becomes fatal error
When trying to checksum local files, if a given file is not found anywhere else than the DL_DIR then this means that the the build is inconsistent, and unreproducible.
This also means that if the DL_DIR is removed or not available the build does not know how to fetch the file and will fail. With this commit we fail earlier and consistently on this condition.
--- lib/bb/fetch2/__init__.py | 4 +++- lib/bb/tests/fetch.py | 7 +++++++ 2 files changed, 10 insertions(+), 1 deletion(-)
lib/bb/fetch2/__init__.py +3 -1
@@ -1233,7 +1233,9 @@ def get_checksum_file_list(d): if f.startswith(dl_dir): # The local fetcher's behaviour is to return a path under DL_DIR if it couldn't find the file anywhere else if os.path.exists(f): - bb.warn("Getting checksum for %s SRC_URI entry %s: file not found except in DL_DIR" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal(("Getting checksum for %s SRC_URI entry %s: file not found except in DL_DIR." + " This means there is no way to get the file except for an orphaned copy" + " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) filelist.append(f + ":" + str(os.path.exists(f)))
lib/bb/tests/fetch.py +7 -0
@@ -693,6 +693,13 @@ class FetcherLocalTest(FetcherTest): flst.sort() return flst + def test_local_checksum_fails_if_only_in_dldir(self): + with open(os.path.join(self.dldir, "on_dl_dir"), "wb"): + pass + self.d.setVar("SRC_URI", "file://on_dl_dir") + with self.assertRaises(bb.BBHandledException): + bb.fetch.get_checksum_file_list(self.d) + def test_local(self): tree = self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c'])
Signature · 1 line
2.25.1
PATCH 2/2 fetch: bb.fatal when trying to checksum non-existing files.
If the local fetcher was not able to find the file anywhere but it was included in the SRC_URI for checksumming just make it a fatal error.
--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)
lib/bb/fetch2/__init__.py +1 -1
@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get the file except for an orphaned copy" " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: - bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) filelist.append(f + ":" + str(os.path.exists(f))) return " ".join(filelist)
lib/bb/tests/fetch.py +5 -0
@@ -700,6 +700,11 @@ class FetcherLocalTest(FetcherTest): with self.assertRaises(bb.BBHandledException): bb.fetch.get_checksum_file_list(self.d) + def test_local_checksum_fails_no_file(self): + self.d.setVar("SRC_URI", "file://404") + with self.assertRaises(bb.BBHandledException): + bb.fetch.get_checksum_file_list(self.d) + def test_local(self): tree = self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c'])
Signature · 1 line
2.25.1
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Hello,
I believe this is the cause of the following failure:
2022-07-13 08:32:51,511 - oe-selftest - INFO - bbtests.BitbakeTests.test_invalid_recipe_src_uri (subunit.RemotedTestCase)
2022-07-13 08:32:51,512 - oe-selftest - INFO - ... FAIL
2022-07-13 08:32:51,512 - oe-selftest - INFO - 3: 18/58 119/493 (3.15s) (bbtests.BitbakeTests.test_invalid_recipe_src_uri)
2022-07-13 08:32:51,512 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bbtests.py", line 148, in test_invalid_recipe_src_uri
bitbake('-ccleanall man-db')
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 229, in bitbake
return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 207, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))AssertionError: Command 'bitbake -ccleanall man-db' returned non-zero exit status 1:/usr/lib64/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
return f(*args, **kwds)
NOTE: Reconnecting to bitbake server...
Loading cache...done.
Loaded 1718 entries from dependency cache.
Parsing recipes...ERROR: /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/recipes-extended/man-db/man-db_2.10.2.bb: Unable to get checksum for man-db SRC_URI entry invalid: file could not be found
ERROR: Parsing halted due to errors, see error messages above
I guess you have to update bbtests.BitbakeTests.test_invalid_recipe_src_uri too
Quoted 50 lines · Paulo Neves
If the local fetcher was not able to find the file anywhere but it was included in the
SRC_URIfor checksumming just make it a fatal error.--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +1
-1@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get the file except for an orphaned copy" " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: - bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) filelist.append(f + ":" + str(os.path.exists(f))) return " ".join(filelist)lib/bb/tests/fetch.py +5
-0@@ -700,6 +700,11 @@ class FetcherLocalTest(FetcherTest): with self.assertRaises(bb.BBHandledException): bb.fetch.get_checksum_file_list(self.d) + def test_local_checksum_fails_no_file(self): + self.d.setVar("SRC_URI", "file://404") + with self.assertRaises(bb.BBHandledException): + bb.fetch.get_checksum_file_list(self.d) + def test_local(self): tree = self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c'])Signature · 11 lines
2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13813): https://lists.openembedded.org/g/bitbake-devel/message/13813 Mute This Topic: https://lists.openembedded.org/mt/92260795/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Hello,
Well this illustrates the kind of issue I wanted to catch, but this test's condition will now not be reachable at all. Previously, the checksum failed with only a warning, while the failure occurred in the download stage of the local fetcher. Now as the checksum fails and it is immediately fatal there is no way to reach a download state at all.
Should this test be removed, then? In my patch i added the appropriate test and it is in the bitbake self test which does not depend on poky or openembedded core. Let me know what you think.
Paulo Neves
Quoted 76 lines · Alexandre Belloni
Hello,
I believe this is the cause of the following failure:
2022-07-13 08:32:51,511 - oe-selftest - INFO - bbtests.BitbakeTests.test_invalid_recipe_src_uri (subunit.RemotedTestCase) 2022-07-13 08:32:51,512 - oe-selftest - INFO - ... FAIL 2022-07-13 08:32:51,512 - oe-selftest - INFO - 3: 18/58 119/493 (3.15s) (bbtests.BitbakeTests.test_invalid_recipe_src_uri) 2022-07-13 08:32:51,512 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/bbtests.py", line 148, in test_invalid_recipe_src_uri bitbake('-ccleanall man-db') File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 229, in bitbake return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options) File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 207, in runCmd raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))AssertionError: Command 'bitbake -ccleanall man-db' returned non-zero exit status 1:
/usr/lib64/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from__spec__or__package__, falling back on__name__and__path__
returnf(*args, **kwds)
NOTE: Reconnecting to bitbake server...
Loading cache...done.
Loaded 1718 entries from dependency cache.
Parsing recipes...ERROR:/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/recipes-extended/man-db/man-db_2.10.2.bb: Unable to get checksum for man-dbSRC_URIentry invalid: file could not be found
ERROR: Parsing halted due to errors, see error messages aboveI guess you have to update bbtests.BitbakeTests.test_invalid_recipe_src_uri too
Quoted 48 lines · Paulo Neves
If the local fetcher was not able to find the file anywhere but it was included in the
SRC_URIfor checksumming just make it a fatal error.--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +1
-1@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get the file except for an orphaned copy" " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: - bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) filelist.append(f + ":" + str(os.path.exists(f))) return " ".join(filelist)lib/bb/tests/fetch.py +5
-0@@ -700,6 +700,11 @@ class FetcherLocalTest(FetcherTest): with self.assertRaises(bb.BBHandledException): bb.fetch.get_checksum_file_list(self.d) + def test_local_checksum_fails_no_file(self): + self.d.setVar("SRC_URI", "file://404") + with self.assertRaises(bb.BBHandledException): + bb.fetch.get_checksum_file_list(self.d) + def test_local(self): tree = self.fetchUnpack(['file://a', 'file://dir/c']) self.assertEqual(tree, ['a', 'dir/c'])Signature · 9 lines
2.25.1 -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13813): https://lists.openembedded.org/g/bitbake-devel/message/13813 Mute This Topic: https://lists.openembedded.org/mt/92260795/3617179 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 9 lines · Paulo Neves
Hello,
Well this illustrates the kind of issue I wanted to catch, but this test's condition will now not be reachable at all. Previously, the checksum failed with only a warning, while the failure occurred in the download stage of the local fetcher.
Now as the checksum fails and it is immediately fatal there is no way to reach a download state at all.
Quoted 3 lines
Should this test be removed, then? In my patch i added the appropriate test and it is in the bitbake self test which does not depend on poky or openembedded core. Let me know what you think.
The test makes sense, we probably just need to remove the bitbake('-ccleanall man-db') lines since the issue changed from runtime to parsing.
Cheers,
Richard
Re: [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
In reply to Paulo NevesQuoted 13 lines · Paulo Neves
If the local fetcher was not able to find the file anywhere but it was included in the
SRC_URIfor checksumming just make it a fatal error.--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +0
-0
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1240
Quoted 5 lines
lib/bb/fetch2/__init__.py +1
-1@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get the file except for an orphaned copy" " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: - bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f)))
Now that we've picked up this change, it seems to have caused a bunch of irritating hard failures where we use to just get irritating warnings. Our recipes probably aren't 100% ideal, but we had a number of recipes which are not pulled into all our machine configs, and could end up with broken SRC_URIs on machine configs they are not intended to be used on.
There are some more complex examples, but one easy example is a recipe which had file://${MACHINE}/eeprom.h in its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.
I know we've been ignoring the warning(s) for a while on these kinds of failures, so it is our own fault, but it is kind of annoying the new behavior. We now have to make sure every recipe not only parses validly on all machine configs but it also has to have fully populated SRC_URIs even when the recipe is never used on that machine config.
Signature · 1 line
Patrick Williams
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 4 lines · Patrick Williams
There are some more complex examples, but one easy example is a recipe which had
file://${MACHINE}/eeprom.hin its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.
This is not how machine-specific versions of the same file are supposed to be listed. Just use file://eeprom.h, and bitbake will look for it in subdirectories that include $MACHINE, and if not found there, a default version will be picked (which you can make bogus, so builds will fail at recipe build stage instead of parsing stage).
Quoted 7 lines
I know we've been ignoring the warning(s) for a while on these kinds of failures, so it is our own fault, but it is kind of annoying the new behavior. We now have to make sure every recipe not only parses validly on all machine configs but it also has to have fully populated
SRC_URIseven when the recipe is never used on that machine config.
Look, this is bleeding edge. And bleeding edge sometimes bleeds and breaks your builds. You signed up for it; if you don't like it there, use stable branches, or use controlled transitions to milestone tags.
Alex
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 6 lines · Alexander Kanavin
Quoted 4 lines · Patrick Williams
There are some more complex examples, but one easy example is a recipe which had
file://${MACHINE}/eeprom.hin its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.
Quoted 8 lines
This is not how machine-specific versions of the same file are supposed to be listed. Just use
file://eeprom.h, and bitbake will look for it in subdirectories that include $MACHINE, and if not found there, a default version will be picked (which you can make bogus, so builds will fail at recipe build stage instead of parsing stage).
I agree. The person who originally wrote the recipe didn't know about the automatic addition of overrides with the FILESEXTRAPATHS. As I wrote this was just one easy example. I fixed this one to both use the override implicitly (as you also suggested) and put a default 'eeprom.h' with a #error[1].
A seriously more complex example we have is that we have our own u-boot recipes and used the same u-boot-foo.inc file names as are present in meta/. It happens that the recipes in meta/recipes-bsp/u-boot end up, even though we'd never use them, using our u-boot-foo.inc files and ending up with totally bogus SRC_URIs in the parsing phase (and thus failing). Essentially, whenever there is a new version of u-boot in meta we're going to end up having to make some bogus empty files[2].
Quoted 7 lines
Quoted 7 lines
I know we've been ignoring the warning(s) for a while on these kinds of failures, so it is our own fault, but it is kind of annoying the new behavior. We now have to make sure every recipe not only parses validly on all machine configs but it also has to have fully populated
SRC_URIseven when the recipe is never used on that machine config.
Quoted 5 lines
Quoted 1 line
Look, this is bleeding edge. And bleeding edge sometimes bleeds and breaks your builds. You signed up for it; if you don't like it there, use stable branches, or use controlled transitions to milestone tags.
I agree we signed up for the bleeding edge, partially to be more aware of changes like this. Hopefully this wasn't worded too poorly, but I am not complaining that "things are currently broken for me". I was trying to raise awareness that this change isn't as pleasant to downstream layer maintainers as it might seem on the surface. If there is an improvement or revert, great; if there isn't, we'll deal with it.
1. https://github.com/facebook/openbmc/commit/c4e6ffacb73a567c83458f6a1b7e5bb0a032b770
2. https://github.com/facebook/openbmc/tree/c4e6ffacb73a567c83458f6a1b7e5bb0a032b770/meta-aspeed/recipes-bsp/u-boot/files/openbmc-fb
Signature · 1 line
Patrick Williams
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 8 lines · Patrick Williams
A seriously more complex example we have is that we have our own u-boot recipes and used the same
u-boot-foo.incfile names as are present inmeta/. It happens that the recipes inmeta/recipes-bsp/u-bootend up, even though we'd never use them, using ouru-boot-foo.incfiles and ending up with totally bogusSRC_URIsin the parsing phase (and thus failing). Essentially, whenever there is a new version of u-boot inmetawe're going to end up having to make some bogus empty files[2].
Reusing the same filenames to me seems like asking for trouble. It's really better to name your own u-boot items in a way that is guaranteed not to get in the way with what items in meta/ look for. u-boot-openbmc-* or similar would do.
Alex
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
In reply to Patrick WilliamsHi Patrick,
Quoted 22 lines · Patrick Williams
Quoted 6 lines · Alexander Kanavin
Quoted 4 lines · Patrick Williams
There are some more complex examples, but one easy example is a recipe which had
file://${MACHINE}/eeprom.hin its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.Quoted 8 lines
This is not how machine-specific versions of the same file are supposed to be listed. Just use
file://eeprom.h, and bitbake will look for it in subdirectories that include $MACHINE, and if not found there, a default version will be picked (which you can make bogus, so builds will fail at recipe build stage instead of parsing stage).I agree. The person who originally wrote the recipe didn't know about the automatic addition of overrides with the FILESEXTRAPATHS. As I wrote this was just one easy example. I fixed this one to both use the override implicitly (as you also suggested) and put a default '
eeprom.h' with a#error[1].
Quoted 9 lines
A seriously more complex example we have is that we have our own u-boot recipes and used the same
u-boot-foo.incfile names as are present inmeta/. It happens that the recipes inmeta/recipes-bsp/u-bootend up, even though we'd never use them, using ouru-boot-foo.incfiles and ending up with totally bogusSRC_URIsin the parsing phase (and thus failing). Essentially, whenever there is a new version of u-boot inmetawe're going to end up having to make some bogus empty files[2].
Alexander already suggested to use a slightly different recipe name (and then you can pick which recipe you want from a configuration file with PREFERRED_PROVIDER_u-boot = "" IIRC), but I'm actually surprised by the behavior you mention seeing.
The original u-boot recipe uses "require" with just the filename, c.f. https://cgit.openembedded.org/openembedded-core/tree/meta/recipes-bsp/u-boot/u-boot_2022.07.bb?h=master, etc... there's not a single recipes-bsp/u-boot returned by git grep in openembedded-core... My understanding of the require mechanism so far is that if you pass it a filename, it'll only look into the current directory (where the recipe is located) and if it has a path, it'll look for that path in all layers (well, BBPATH, which in the majority of cases should be LAYERDIR, the root of the layer). Since there is no path in require for the u-boot recipes in openmebedded-core, I don't understand what you're saying happens happens. c.f. https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#require-directive for the doc on require directive.
This highlights one of the following:
- a rogue bbappend or some weird configuration in one the layers in use,
- a bug/corner-case in require directive,
Maybe there's something to fix or document, hence why I'm writing this mail, it'd be good to actually be able to reproduce this and understand why it happens.
Cheers,
Quentin
Re: [bitbake-devel] [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 17 lines · Quentin Schulz
Quoted 2 lines · Patrick Williams
Quoted 1 line · Alexander Kanavin
On Tue, 26 Jul 2022 at 06:09, Patrick Williams <[email protected]> wrote:
The original u-boot recipe uses "require" with just the filename, c.f.
https://cgit.openembedded.org/openembedded-core/tree/meta/recipes-bsp/u-boot/u-boot_2022.07.bb?h=master,
etc... there's not a single recipes-bsp/u-boot returned by git grep in openembedded-core... My understanding of the require mechanism so far is that if you pass it a filename, it'll only look into the current directory (where the recipe is located) and if it has a path, it'll look for that path in all layers (well, BBPATH, which in the majority of cases should be LAYERDIR, the root of the layer). Since there is no path in require for the u-boot recipes in openmebedded-core, I don't understand what you're saying happens happens. c.f.
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#require-directive
for the doc on require directive.
Quoted 2 lines
This highlights one of the following:
- a rogue bbappend or some weird configuration in one the layers in use,
I dug into it a little more. It is a rogue bbappend in the u-boot case and not an issue with the require.
We have an internal tree that contains all the u-boot code along side the recipes, along with a u-boot%.bbappend to replace the external SRC_URI with an internal ("file://"-based) one. This gets applied to the meta/recipes-bsp versions even though we don't have copies of their code.
Signature · 1 line
Patrick Williams
Re: [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
In reply to Patrick WilliamsQuoted 31 lines · Patrick Williams
Quoted 13 lines · Paulo Neves
If the local fetcher was not able to find the file anywhere but it was included in the
SRC_URIfor checksumming just make it a fatal error.--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +0
-0lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1240
Quoted 5 lines
lib/bb/fetch2/__init__.py +1
-1@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get the file except for an orphaned copy" " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: - bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f)))Now that we've picked up this change, it seems to have caused a bunch of irritating hard failures where we use to just get irritating warnings. Our recipes probably aren't 100% ideal, but we had a number of recipes which are not pulled into all our machine configs, and could end up with broken
SRC_URIson machine configs they are not intended to be used on.There are some more complex examples, but one easy example is a recipe which had
file://${MACHINE}/eeprom.hin its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.
Quoted 7 lines
I know we've been ignoring the warning(s) for a while on these kinds of failures, so it is our own fault, but it is kind of annoying the new behavior. We now have to make sure every recipe not only parses validly on all machine configs but it also has to have fully populated
SRC_URIseven when the recipe is never used on that machine config.
I'm just going from memory but it might help to set COMPATIBLE_MACHINE in the recipe so that it isn't fully parsed for the machines it isn't intended for.
Cheers,
Richard
Re: [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
That is what I do. If the recipe is machine specific it should be marked as such.
Quoted 57 lines · Richard Purdie
Quoted 9 lines · Patrick Williams
Quoted 8 lines · Paulo Neves
If the local fetcher was not able to find the file anywhere but it was included in the
SRC_URIfor checksumming just make it a fatal error.--- lib/bb/fetch2/__init__.py | 2 +- lib/bb/tests/fetch.py | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-)Quoted 6 lines
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1237
Quoted 6 lines
lib/bb/fetch2/__init__.py +0
-0@@ -1237,7 +1237,7 @@ def get_checksum_file_list(d): " This means there is no way to get thefile except for an orphaned copy"
Quoted 1 line
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1238
Quoted 1 line
lib/bb/fetch2/__init__.py +0
-0" in DL_DIR.") % (d.getVar('PN'),
os.path.basename(f)))Quoted 2 lines
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1240
Quoted 2 lines
lib/bb/fetch2/__init__.py +0
-1else: - bb.warn("Unable to get checksum for %s
SRC_URIentry %s: file could not be found" % (d.getVar('PN'),os.path.basename(f)))Quoted 1 line
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1240
Quoted 1 line
lib/bb/fetch2/__init__.py +1
-0+ bb.fatal("Unable to get checksum for %s
SRC_URIentry %s: file could not be found" % (d.getVar('PN'),os.path.basename(f)))Quoted 12 lines
Now that we've picked up this change, it seems to have caused a bunch of irritating hard failures where we use to just get irritating warnings. Our recipes probably aren't 100% ideal, but we had a number of recipes which are not pulled into all our machine configs, and could end up with broken
SRC_URIson machine configs they are not intended to be used on.There are some more complex examples, but one easy example is a recipe which had
file://${MACHINE}/eeprom.hin its SRC_URI[1]. Any machine which didn't provide this file, even if it never intended to use the recipe, now fails when we picked up this change.Quoted 7 lines
I know we've been ignoring the warning(s) for a while on these kinds of failures, so it is our own fault, but it is kind of annoying the new behavior. We now have to make sure every recipe not only parses validly on all machine configs but it also has to have fully populated
SRC_URIseven when the recipe is never used on that machine config.1.
I'm just going from memory but it might help to set
COMPATIBLE_MACHINEin the recipe so that it isn't fully parsed for the machines it isn't intended for.Cheers,
Richard
Re: [PATCH 2/2] fetch: bb.fatal when trying to checksum non-existing files.
Quoted 6 lines · Paulo Neves
Quoted 3 lines
I'm just going from memory but it might help to set
COMPATIBLE_MACHINEin the recipe so that it isn't fully parsed for the machines it isn't intended for.That is what I do. If the recipe is machine specific it should be marked as such.
Thanks for the info. At first blush it doesn't sound pleasant to maintain a list like this for each recipe (since we have over 30 different machines), but we might be able to take a tactical approach on specific problematic recipes and/or leverage some overrides to better classify the machine feature that causes the recipe to be necessary.
Signature · 1 line
Patrick Williams
Quoted 19 lines · Paulo Neves
When trying to checksum local files, if a given file is not found anywhere else than the
DL_DIRthen this means that the the build is inconsistent, and unreproducible.This also means that if the
DL_DIRis removed or not available the build does not know how to fetch the file and will fail. With this commit we fail earlier and consistently on this condition.Signed-off-by Paulo Neves <[email protected]>--- lib/bb/fetch2/__init__.py | 4 +++- lib/bb/tests/fetch.py | 7 +++++++ 2 files changed, 10 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +0
-0
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1241
Quoted 10 lines
lib/bb/fetch2/__init__.py +3
-1@@ -1233,7 +1233,9 @@ def get_checksum_file_list(d): if f.startswith(dl_dir): # The local fetcher's behaviour is to return a path under DL_DIR if it couldn't find the file anywhere else if os.path.exists(f): - bb.warn("Getting checksum for %s SRC_URI entry %s: file not found except in DL_DIR" % (d.getVar('PN'), os.path.basename(f))) + bb.fatal(("Getting checksum for %s SRC_URI entry %s: file not found except in DL_DIR." + " This means there is no way to get the file except for an orphaned copy" + " in DL_DIR.") % (d.getVar('PN'), os.path.basename(f))) else: bb.warn("Unable to get checksum for %s SRC_URI entry %s: file could not be found" % (d.getVar('PN'), os.path.basename(f))) filelist.append(f + ":" + str(os.path.exists(f)))
Did you manage to trigger that error in a real world use case?
I've just been looking at this code and it is horribly old and outdated. I can't help wonder if we shouldn't do something a bit more invasive to clean it up a bit more. I suspect it does some of these things for old/obsolete reasons...
I ask since I'm wondering if anyone ever hits these code paths in a valid usecase.
Thanks for working on it, we do need to improve some of these things.
Cheers,
Richard
Quoted 45 lines · On Sat, Jul 9, 2022, 08:52 Richard Purdie <…
Quoted 19 lines · Paulo Neves
When trying to checksum local files, if a given file is not found anywhere else than the
DL_DIRthen this means that the the build is inconsistent, and unreproducible.This also means that if the
DL_DIRis removed or not available the build does not know how to fetch the file and will fail. With this commit we fail earlier and consistently on this condition.Signed-off-by Paulo Neves <[email protected]>--- lib/bb/fetch2/__init__.py | 4 +++- lib/bb/tests/fetch.py | 7 +++++++ 2 files changed, 10 insertions(+), 1 deletion(-)lib/bb/fetch2/__init__.py +0
-0lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1234
Quoted 2 lines
lib/bb/fetch2/__init__.py +0
-0@@ -1233,7 +1233,9 @@ def get_checksum_file_list(d): if f.startswith(dl_dir): # The local fetcher's behaviour is to return a pathunder
DL_DIRif it couldn't find the file anywhere elselib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1236
Quoted 2 lines
lib/bb/fetch2/__init__.py +0
-1if os.path.exists(f): - bb.warn("Getting checksum for %s SRC_URI entry%s: file not found except in
DL_DIR" % (d.getVar('PN'),os.path.basename(f)))lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1236
Quoted 1 line
lib/bb/fetch2/__init__.py +1
-0+ bb.fatal(("Getting checksum for %s SRC_URIentry %s: file not found except in
DL_DIR."lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1237
Quoted 1 line
lib/bb/fetch2/__init__.py +1
-0+ " This means there is no way to get thefile except for an orphaned copy"
lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1238
Quoted 1 line
lib/bb/fetch2/__init__.py +1
-0+ " in DL_DIR.") % (d.getVar('PN'),
os.path.basename(f)))lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1240
Quoted 2 lines
lib/bb/fetch2/__init__.py +0
-0else: bb.warn("Unable to get checksum for %s SRC_URIentry %s: file could not be found" % (
d.getVar('PN'),os.path.basename(f)))lib/bb/fetch2/__init__.py def get_checksum_file_list(d): · line 1241
Quoted 1 line
lib/bb/fetch2/__init__.py +0
-0filelist.append(f + ":" + str(os.path.exists(f)))Did you manage to trigger that error in a real world use case?
I've just been looking at this code and it is horribly old and outdated. I can't help wonder if we shouldn't do something a bit more invasive to clean it up a bit more. I suspect it does some of these things for old/obsolete reasons...
Quoted 9 lines
I ask since I'm wondering if anyone ever hits these code paths in a valid usecase.
Thanks for working on it, we do need to improve some of these things.
Cheers,
Richard
Hey Richard,
I think these codepaths are hit on mostly on bugs(bad uri parsing) or bad files(bad permissions).
The use case i added on the tests is also clearly bad, but I have seen builds going on for very long in the state of warning, which becomes fatal with the patch, because the dl_dir was stable and used by all users. So expect this patch to increase the reports of latent issues people just ignored.
Yesterday I also hit this warning organically exactly on the situation of the test: a directory mentioned on the SRC_URI which did not exist in any FILESPATHS but somehow existed on the dl_dir. So it is real.
I will investigate the whole default to dl_dir if not found and remove it if I cannot find a good reason.
Paulo Neves
Quoted 1 line
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 97 lines · Paulo Neves
On 7/9/22 09:19, Paulo Neves wrote:Quoted 89 lines
On Sat, Jul 9, 2022, 08:52 Richard Purdie
<[email protected]> wrote:Quoted 57 lines · Paulo Neves
> When trying to checksum local files, if a given file is not found
> anywhere else than theDL_DIRthen this means that the the build is
> inconsistent, and unreproducible.
>
> This also means that if theDL_DIRis removed or not available the
> build does not know how to fetch the file and will fail. With this
> commit we fail earlier and consistently on this condition. >
> Signed-off-by: Paulo Neves <[email protected]>
> ---
>lib/bb/fetch2/__init__.py| 4 +++-
> lib/bb/tests/fetch.py | 7 +++++++
> 2 files changed, 10 insertions(+), 1 deletion(-)
>
> diff --gita/lib/bb/fetch2/__init__.pyb/lib/bb/fetch2/__init__.py
> indexac557176..5f05278a100644
> ---a/lib/bb/fetch2/__init__.py
> +++b/lib/bb/fetch2/__init__.py
> @@ -1233,7 +1233,9 @@ defget_checksum_file_list(d):
> iff.startswith(dl_dir):
> # The local fetcher's behaviour is to return a path underDL_DIRif it couldn't find the file anywhere else
> ifos.path.exists(f):
> - bb.warn("Getting checksum for %sSRC_URIentry %s: file not found except inDL_DIR" % (d.getVar('PN'),os.path.basename(f)))
> + bb.fatal(("Getting checksum for %sSRC_URIentry %s: file not found except inDL_DIR."
> + " This means there is no way to get the file except for an orphaned copy"
> + " inDL_DIR.") % (d.getVar('PN'),os.path.basename(f)))
> else:
> bb.warn("Unable to get checksum for %sSRC_URIentry %s: file could not be found" % (d.getVar('PN'),os.path.basename(f)))
>filelist.append(f + ":" + str(os.path.exists(f)))
Did you manage to trigger that error in a real world use case?
I've just been looking at this code and it is horribly old and outdated. I can't help wonder if we shouldn't do something a bit more
invasive to clean it up a bit more. I suspect it does some of these
things for old/obsolete reasons...
I ask since I'm wondering if anyone ever hits these code paths in aQuoted 9 lines
valid usecase.
Thanks for working on it, we do need to improve some of these things.
Cheers,
RichardHey Richard,
I think these codepaths are hit on mostly on bugs(bad uri parsing) or bad files(bad permissions).
The use case i added on the tests is also clearly bad, but I have seen builds going on for very long in the state of warning, which becomes fatal with the patch, because the
dl_dirwas stable and used by all users. So expect this patch to increase the reports of latent issues people just ignored.Yesterday I also hit this warning organically exactly on the situation of the test: a directory mentioned on the
SRC_URIwhich did not exist in any FILESPATHS but somehow existed on thedl_dir. So it is real.Quoted 4 lines
I will investigate the whole default to
dl_dirif not found and remove it if I cannot find a good reason.Paulo Neves
Quoted 13 lines
I did some archeology trying to find whyDL_DIRhas anything to do with the local fetcher and the reason probably comes from this[1] and this[2]. The commit is about sstate cache and i think it considersDL_DIRas valid mirror directory in agreement with lib/bb/fetch2/README[2].
The problem is that [3] from the same README "spec", is either contradictory with with [2], senseless, or accurate depending on the usage of the local file:// fetcher. For aSRC_URIfile, what it is senseless to searchDL_DIRif we have the FILEPATHS? If it is not at hand and only in theDL_DIRthen it is irreproducible and contradicts
[3] (the case this patch argues for).
sstate.bbclass sets DL_DIR to SSTATE_DIR in the context of sstate downloads. It also puts SSTATE_DIR in FILESPATH which I suspect it did not back in 2011. I therefore suspect we now have a better fix for the issue as FILESPATH is set and the change from 2011 is now obsolete for sstate. Or I hope so anyway!
Cheers,
Richard