Skip to contentSkip to conversation

base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

PATCH In review [email protected] 4 participants 7 messages Apr 15, 2024 – Apr 17, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 base.bbclass: make do_fetch depends on PREMIRRORS MIRRORS

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
 meta/classes-global/base.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -0
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
 do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
 do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
 python base_do_fetch() {
 
     src_uri = (d.getVar('SRC_URI') or "").split()
Signature · 1 line
2.25.1
#1 lore raw

How does this work when sharing sstate between builders with different PREMIRRORS/MIRRORS? Doesn't this cause all signatures to be different as do_unpack does depend on do_fetch?

Quoted 41 lines · Changqing Li via lists.openembedded.org

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
 meta/classes-global/base.bbclass | 1 +
 1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -1
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
 do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
 do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
 do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
 python base_do_fetch() {
 
     src_uri = (d.getVar('SRC_URI') or "").split()
--

2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198222): https://lists.openembedded.org/g/openembedded-core/message/198222
Mute This Topic: https://lists.openembedded.org/mt/105532159/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#2 lore raw
In reply to [email protected]
Quoted 21 lines · Changqing Li via lists.openembedded.org

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
 meta/classes-global/base.bbclass | 1 +

 1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +0 -0
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
Quoted 8 lines

 do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
 python base_do_fetch() {

     src_uri = (d.getVar('SRC_URI') or "").split()

Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.

Cheers,

Richard

#3 lore raw
Quoted 31 lines · Richard Purdie

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 27 lines · Changqing Li via lists.openembedded.org

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
  meta/classes-global/base.bbclass | 1 +
  1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -0
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
  do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
  do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
  do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
  python base_do_fetch() {
 
Quoted 4 lines

meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145

Quoted 2 lines
meta/classes-global/base.bbclass +0 -0
      src_uri = (d.getVar('SRC_URI') or "").split()

Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.

The failure case I met is like this:

I have a project with PREMIRRORS setting, and the tarballs are under it.

PREMIRRORS:append = "file:///project/path/to/localmirrorA"

after first bitbake pseudo-native, there will a symlink under DL_DIR:

pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz

Then, Since all the tarballs are moved to /project/path/to/localmirrorB

I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"

Then bitbake pseudo-native again,   do_fetch  will not rerun since no signature change.

do_unpack rerun since one dependency  change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,

but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist

since localmirrorB replace it.  Then will cause do_unpck failure:

NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz: /build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory

So this is why I want to add this depends,  and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB

Regards

Changqing

Quoted 5 lines

Cheers,

Richard

#4 lore raw

I'm not sure why files in DL_DIR would symlink to the local (pre)mirrors instead of making an independent copy? Mirrors can indeed move or disappear altogether.

Can you look into where the symlinking happens, and whether that can be modified or configured differently (maybe also the git history for that code)?

Alex

Quoted 90 lines · Changqing Li via lists.openembedded.org
Quoted 31 lines · Richard Purdie

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 27 lines · Changqing Li via lists.openembedded.org

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
  meta/classes-global/base.bbclass | 1 +
  1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -0
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
  do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
  do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
  do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
  python base_do_fetch() {
 
Quoted 4 lines

meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145

Quoted 2 lines
meta/classes-global/base.bbclass +0 -0
      src_uri = (d.getVar('SRC_URI') or "").split()

Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.

The failure case I met is like this:

I have a project with PREMIRRORS setting, and the tarballs are under it.

PREMIRRORS:append = "file:///project/path/to/localmirrorA"

after first bitbake pseudo-native, there will a symlink under DL_DIR:

pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz

Then, Since all the tarballs are moved to /project/path/to/localmirrorB

I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"

Then bitbake pseudo-native again, do_fetch will not rerun since no
signature change.

do_unpack rerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,

but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist

since localmirrorB replace it. Then will cause do_unpck failure:

NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory

So this is why I want to add this depends, and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB

Regards

Changqing

Quoted 5 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198246): https://lists.openembedded.org/g/openembedded-core/message/198246
Mute This Topic: https://lists.openembedded.org/mt/105532159/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw
Quoted 10 lines · Alexander Kanavin

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

I'm not sure why files in DL_DIR would symlink to the local (pre)mirrors instead of making an independent copy? Mirrors can indeed move or disappear altogether.

Can you look into where the symlinking happens, and whether that can be modified or configured differently (maybe also the git history for that code)?

Current code is here: https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/__init__.py#n1096

It is created during handle the (pre)mirrors in do_fetch.  And the history long time ago:

https://git.openembedded.org/bitbake/commit/?id=83f9445081595630488a5d3e170d282a13836c98

About if we can change to independent copy, I need to do more research, welcome to provide me some

background and notes.

Regards

Changqing

Quoted 91 lines

Alex

Quoted 86 lines · Changqing Li via lists.openembedded.org
Quoted 31 lines · Richard Purdie

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 27 lines · Changqing Li via lists.openembedded.org

From: Changqing Li<[email protected]>

When there are tarballs in local mirrorfile://, do_fetch will
create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because
do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not
be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
   meta/classes-global/base.bbclass | 1 +
   1 file changed, 1 insertion(+)
meta/classes-global/base.bbclass +1 -0
index 0999b42daa..5094a1540e 100644
@@ -139,6 +139,7 @@ do_fetch[file-checksums] ="${@bb.fetch.get_checksum_file_list(d)}"
   do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}"
   do_fetch[prefuncs] += "fetcher_hashes_dummyfunc"
   do_fetch[network] = "1"
+do_fetch[vardeps] += "PREMIRRORS MIRRORS"
   python base_do_fetch() {
 
Quoted 4 lines

meta/classes-global/base.bbclass do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}" · line 145

Quoted 2 lines
meta/classes-global/base.bbclass +0 -0
       src_uri = (d.getVar('SRC_URI') or "").split()

Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.

The failure case I met is like this:

I have a project with PREMIRRORS setting, and the tarballs are under it.

PREMIRRORS:append ="file:///project/path/to/localmirrorA"

after first bitbake pseudo-native, there will a symlink under DL_DIR:

pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz

Then, Since all the tarballs are moved to /project/path/to/localmirrorB

I changed PREMIRRORS:append ="file:///project/path/to/localmirrorB"

Then bitbake pseudo-native again, do_fetch will not rerun since no
signature change.

do_unpack rerun since one dependency change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,

but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist

since localmirrorB replace it. Then will cause do_unpck failure:

NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory

So this is why I want to add this depends, and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB

Regards

Changqing

Quoted 4 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#198246):https://lists.openembedded.org/g/openembedded-core/message/198246 Mute This Topic:https://lists.openembedded.org/mt/105532159/1686489
Group Owner:[email protected] Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
In reply to Changqing Li
Quoted 72 lines · Changqing Li
Quoted 38 lines · Richard Purdie

CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe.

Quoted 32 lines · Changqing Li via lists.openembedded.org

From: Changqing Li <[email protected]>

When there are tarballs in local mirror file://, do_fetch will create symlink under DL_DIR, eg:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz after user update path from localmirrorA to localmirrorB, because do_fetch don't depends on PREMIRRORS/MIRRORS, do_fetch will not be rerun, so the symlink will not update to:
pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorB/pseudo-prebuilt-2.33.tar.xz then, do_upack will failed since the tarball cannot be found.

---
  meta/classes-global/base.bbclass | 1 +

  1 file changed, 1 insertion(+)
meta/classes- renamed from meta/classes-global/base.bbclass +0 -0

global/base.bbclass
index 0999b42daa..5094a1540e 100644

meta/classes-global/base.bbclass +0 -0
@@ -139,6 +139,7 @@ do_fetch[file-checksums] =

"${@bb.fetch.get_checksum_file_list(d)}" do_fetch[file-checksums] += " ${@get_lic_checksum_file_list(d)}" do_fetch[prefuncs] += "fetcher_hashes_dummyfunc" do_fetch[network] = "1" +do_fetch[vardeps] += "PREMIRRORS MIRRORS" python base_do_fetch() {

Quoted 4 lines
Quoted 2 lines

      src_uri = (d.getVar('SRC_URI') or "").split()

Changing PREMIRRORS or MIRRORS should not change the output of the task, ever, so do_fetch shouldn't depend upon them.

The failure case I met is like this:

I have a project with PREMIRRORS setting, and the tarballs are under it.

PREMIRRORS:append = "file:///project/path/to/localmirrorA"

after first bitbake pseudo-native, there will a symlink under DL_DIR:

pseudo-prebuilt-2.33.tar.xz -> /project/path/to/localmirrorA/pseudo-prebuilt-2.33.tar.xz

Then, Since all the tarballs are moved to
/project/path/to/localmirrorB

I changed PREMIRRORS:append = "file:///project/path/to/localmirrorB"

Then bitbake pseudo-native again,   do_fetch  will not rerun since no signature change.

do_unpack rerun since one dependency  change, it will try to unpack ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz,

but since do_fetch not rerun, ${DL_DIR}/pseudo-prebuilt-2.33.tar.xz still point to localmirrorA, but localmirrorA not exist

since localmirrorB replace it.  Then will cause do_unpck failure:

Quoted 7 lines

NOTE: Unpacking /build/downloads/pseudo-prebuilt-2.33.tar.xz to /build/tmp-glibc/work/x86_64-linux/pseudo-native/1.9.0+git xz:
/build/downloads/pseudo-prebuilt-2.33.tar.xz: No such file or directory

So this is why I want to add this depends,  and want to make do_fetch rerun and resymlink to /project/path/to/localmirrorB

The question is if the fetch reruns, does it correct the path?

do_fetch should definitely not depend on PREMIRRORS or MIRRORS as those can be set by local users for local setups. The idea is that result would be the same regardless, else the fetch checksums (or git revision) would not match. I will therefore not take any patch changing that.

Also, despite what Alex said, the fetcher does only make symlinks for local files, it does not copy every file into DL_DIR. It does this for efficiency, both time and disk usage.

If you start moving files around locally, you're expected to fix up the links too. I believe that if you manually run the fetch task, it will update correctly so the only issue is it not automatically triggering but I don't believe it should trigger.

So whilst I can see how you have an unfortunate build failure, it was of your own making by moving mirror directories around.

Cheers,

Richard