base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS
PATCH 1/1 base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.
--- meta/classes-global/base.bbclass | 1 + 1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -0
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() { src_uri = (d.getVar('SRC_URI') or "").split()
Signature · 1 line
2.25.1
How does this work when sharing sstate between builders with different PREMIRRORS/MIRRORS? Doesn't this cause all signatures to be different as do_unpack does depend on do_fetch?
Quoted 41 lines · Changqing Li via lists.openembedded.org
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://,
do_fetchwill
create symlink underDL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li <[email protected]>--- meta/classes-global/base.bbclass | 1 + 1 file changed, 1 insertion(+)meta/classes-global/base.bbclass +1
-1@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() { src_uri = (d.getVar('SRC_URI') or "").split() --2.25.1
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198222): https://lists.openembedded.org/g/openembedded-core/message/198222
Mute This Topic: https://lists.openembedded.org/mt/105532159/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 21 lines · Changqing Li via lists.openembedded.org
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://, do_fetch will
create symlink underDL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li <[email protected]>---
meta/classes-global/base.bbclass | 1 +1 file changed, 1 insertion(+)meta/classes-global/base.bbclass +0
-0@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
Quoted 8 lines
do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "
fetcher_hashes_dummyfunc" do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
pythonbase_do_fetch(){
src_uri= (d.getVar('SRC_URI')or "").split()
Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.
Cheers,
Richard
Quoted 31 lines · Richard Purdie
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 27 lines · Changqing Li via lists.openembedded.org
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://,
do_fetchwill
create symlink underDL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li <[email protected]>--- meta/classes-global/base.bbclass | 1 + 1 file changed, 1 insertion(+)meta/classes-global/base.bbclass +1
-0@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() {
Quoted 4 lines
meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145
Quoted 2 lines
meta/classes-global/base.bbclass +0
-0src_uri = (d.getVar('SRC_URI') or "").split()Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so
do_fetchshouldn't depend upon them.
The failure case I met is like this:
I have a project with PREMIRRORS setting, and the tarballs are under it.
PREMIRRORS:append = "file:///project/path/to/localmirrorA"
after first bitbake pseudo-native, there will a symlink under DL_DIR:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
Then, Since all the tarballs are moved to /project/path/to/localmirrorB
I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"
Then bitbake pseudo-native again, do_fetch will not rerun since no signature change.
do_unpack rerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,
but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist
since localmirrorB replace it. Then will cause do_unpck failure:
NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory
So this is why I want to add this depends, and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB
Regards
Changqing
Quoted 5 lines
Cheers,
Richard
I'm not sure why files in DL_DIR would symlink to the local (pre)mirrors instead of making an independent copy? Mirrors can indeed move or disappear altogether.
Can you look into where the symlinking happens, and whether that can be modified or configured differently (maybe also the git history for that code)?
Alex
Quoted 90 lines · Changqing Li via lists.openembedded.org
Quoted 31 lines · Richard Purdie
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 27 lines · Changqing Li via lists.openembedded.org
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://,
do_fetchwill
create symlink underDL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li <[email protected]>--- meta/classes-global/base.bbclass | 1 + 1 file changed, 1 insertion(+)meta/classes-global/base.bbclass +1
-0@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() {Quoted 4 lines
meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145
Quoted 2 lines
meta/classes-global/base.bbclass +0
-0src_uri = (d.getVar('SRC_URI') or "").split()Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so
do_fetchshouldn't depend upon them.The failure case I met is like this:
I have a project with PREMIRRORS setting, and the tarballs are under it.
PREMIRRORS:append = "file:///project/path/to/localmirrorA"
after first bitbake pseudo-native, there will a symlink under
DL_DIR:pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
Then, Since all the tarballs are moved to /project/path/to/localmirrorB
I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"
Then bitbake pseudo-native again,
do_fetchwill not rerun since no
signature change.
do_unpackrerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,but since
do_fetchnot rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not existsince localmirrorB replace it. Then will cause
do_unpckfailure:NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directorySo this is why I want to add this depends, and want to make
do_fetchrerun and resymlink to /project/path/to/localmirrorBRegards
Changqing
Quoted 5 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198246): https://lists.openembedded.org/g/openembedded-core/message/198246
Mute This Topic: https://lists.openembedded.org/mt/105532159/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 10 lines · Alexander Kanavin
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.I'm not sure why files in
DL_DIRwould symlink to the local (pre)mirrors instead of making an independent copy? Mirrors can indeed move or disappear altogether.Can you look into where the symlinking happens, and whether that can be modified or configured differently (maybe also the git history for that code)?
Current code is here: https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/__init__.py#n1096
It is created during handle the (pre)mirrors in do_fetch. And the history long time ago:
https://git.openembedded.org/bitbake/commit/?id=83f9445081595630488a5d3e170d282a13836c98
About if we can change to independent copy, I need to do more research, welcome to provide me some
background and notes.
Regards
Changqing
Quoted 91 lines
Alex
Quoted 86 lines · Changqing Li via lists.openembedded.org
Quoted 31 lines · Richard Purdie
CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.Quoted 27 lines · Changqing Li via lists.openembedded.org
From: Changqing Li<[email protected]>
When there are tarballs in local mirrorfile://,
do_fetchwill
create symlink underDL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li<[email protected]>--- meta/classes-global/base.bbclass | 1 + 1 file changed, 1 insertion(+)meta/classes-global/base.bbclass +1
-0@@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() {Quoted 4 lines
meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145
Quoted 2 lines
meta/classes-global/base.bbclass +0
-0src_uri = (d.getVar('SRC_URI') or "").split()Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so
do_fetchshouldn't depend upon them.The failure case I met is like this:
I have a project with PREMIRRORS setting, and the tarballs are under it.
PREMIRRORS:append ="file:///project/path/to/localmirrorA"
after first bitbake pseudo-native, there will a symlink under
DL_DIR:pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
Then, Since all the tarballs are moved to /project/path/to/localmirrorB
I changed PREMIRRORS:append ="file:///project/path/to/localmirrorB"
Then bitbake pseudo-native again,
do_fetchwill not rerun since no
signature change.
do_unpackrerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,but since
do_fetchnot rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not existsince localmirrorB replace it. Then will cause
do_unpckfailure:NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directorySo this is why I want to add this depends, and want to make
do_fetchrerun and resymlink to /project/path/to/localmirrorBRegards
Changqing
Quoted 4 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198246):https://lists.openembedded.org/g/openembedded-core/message/198246 Mute This Topic:https://lists.openembedded.org/mt/105532159/1686489
Group Owner:[email protected] Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 72 lines · Changqing Li
Quoted 38 lines · Richard Purdie
CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe.
Quoted 32 lines · Changqing Li via lists.openembedded.org
From: Changqing Li <[email protected]>
When there are tarballs in local mirror file://, do_fetch will create symlink under
DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, becausedo_fetchdon't depends on PREMIRRORS/MIRRORS,do_fetchwill not be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then,do_upackwill failed since the tarball cannot be found.Signed-off-by Changqing Li <[email protected]>---
meta/classes-global/base.bbclass | 1 +1 file changed, 1 insertion(+)meta/classes- renamed from meta/classes-global/base.bbclass +0
-0global/base.bbclass
index0999b42daa..5094a1540e100644meta/classes-global/base.bbclass +0
-0@@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "
fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" pythonbase_do_fetch(){Quoted 4 lines
Quoted 2 lines
src_uri= (d.getVar('SRC_URI')or "").split()Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so
do_fetchshouldn't depend upon them.The failure case I met is like this:
I have a project with PREMIRRORS setting, and the tarballs are under it.
PREMIRRORS:append = "file:///project/path/to/localmirrorA"
after first bitbake pseudo-native, there will a symlink under
DL_DIR:pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz
Then, Since all the tarballs are moved to
/project/path/to/localmirrorBI changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"
Then bitbake pseudo-native again, do_fetch will not rerun since no signature change.
do_unpackrerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,but since
do_fetchnot rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not existsince localmirrorB replace it. Then will cause
do_unpckfailure:
Quoted 7 lines
NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directorySo this is why I want to add this depends, and want to make
do_fetchrerun and resymlink to /project/path/to/localmirrorB
The question is if the fetch reruns, does it correct the path?
do_fetch should definitely not depend on PREMIRRORS or MIRRORS as those can be set by local users for local setups. The idea is that result would be the same regardless, else the fetch checksums (or git revision) would not match. I will therefore not take any patch changing that.
Also, despite what Alex said, the fetcher does only make symlinks for local files, it does not copy every file into DL_DIR. It does this for efficiency, both time and disk usage.
If you start moving files around locally, you're expected to fix up the links too. I believe that if you manually run the fetch task, it will update correctly so the only issue is it not automatically triggering but I don't believe it should trigger.
So whilst I can see how you have an unfortunate build failure, it was of your own making by moving mirror directories around.
Cheers,
Richard