Skip to contentSkip to conversation

Is http://downloads.yoctoproject.org/ down?

DISCUSSION [email protected] 5 participants 10 messages May 12, 2020 – Aug 7, 2020

lore
Share & tools
lore
#2 lore raw
Quoted 11 lines · Paul Barker
Quoted 6 lines · Zwicker Severin

I can’t reach http://downloads.yoctoproject.org/ to download http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz .

Did somebody know anything?

Yes, it currently appears to be down. The relevant folks will be looking at it but it's still early in the US timezones.

Could it be Europe-specific problem?
Seems slow, but still working here in the US:

$ wget http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz
--2020-05-12 09:08:15--  http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz
Resolving downloads.yoctoproject.org... 198.145.29.63
Connecting to downloads.yoctoproject.org|198.145.29.63|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5450224 (5.2M) [application/octet-stream]
Saving to: ‘x86_64-nativesdk-libc.tar.xz’

x86_64-nativesdk-libc.tar.xz 100%[==========================================================================>] 5.20M 540KB/s in 59s

2020-05-12 09:09:27 (89.5 KB/s) - ‘x86_64-nativesdk-libc.tar.xz’ saved [5450224/5450224]
#3 lore raw

With downloads.yoctoproject.org down again this begs the question:

Why does uninative tarball get fetched by default from the internet every time?
Why is it in not cached like every other tarball that is downloaded by recipes?
The default behaviour for the yocto project should be to have the tarball stored in the yocto cache such that it is not fetched every time?

#4 lore raw
Quoted 9 lines · Scott Branden via lists.yoctoproject.org

With downloads.yoctoproject.org down again this begs the question:

Why does uninative tarball get fetched by default from the internet every time?
Why is it in not cached like every other tarball that is downloaded by recipes?
The default behaviour for the yocto project should be to have the tarball stored in the yocto cache such that it is not fetched every time?

Those are good questions.

As far as I know on my own local builds, the uninative tarball *is* cached locally in DL_DIR. We don't see repeated downloads on the autobuilder either so it appears to work there too.

If you're not seeing that it would definitely be something to investigate.

Cheers,

Richard

#5 lore raw

Hi Richard,

Quoted 11 lines · Richard Purdie
Quoted 9 lines · Scott Branden via lists.yoctoproject.org

With downloads.yoctoproject.org down again this begs the question:

Why does uninative tarball get fetched by default from the internet every time?
Why is it in not cached like every other tarball that is downloaded by recipes?
The default behaviour for the yocto project should be to have the tarball stored in the yocto cache such that it is not fetched every time?

Quoted 5 lines

Those are good questions.

As far as I know on my own local builds, the uninative tarball *is* cached locally in DL_DIR. We don't see repeated downloads on the autobuilder either so it appears to work there too.

Yes, the uninative tarball *is* cached but I see repeated downloads.

Quoted 3 lines

If you're not seeing that it would definitely be something to investigate.

I just did a clone of poky.git and the repeated downloads do NOT happen. So I now need to investigate what is different when adding layers and conf settings that is causing the issue.

Quoted 4 lines

Cheers,

Richard

Thanks for confirming the uninative caching should work, but it is not in my environment and I will have to investigate.

Scott

#6 lore raw

We found the source of the uninative fetch problem:

SOURCE_MIRROR_URL needs to be set in order for uninative to be picked up from local mirror.

The standard PREMIRRORS mechanism doesn't work for this single tarball.
So every time a clean poky build took place it always went to the internet to fetch the uninative tarball the first time and store it in the local build download.

I think this should be clearly documented somewhere in the yocto manual?

Quoted 33 lines · Scott Branden

Hi Richard,

Quoted 11 lines · Richard Purdie
Quoted 9 lines · Scott Branden via lists.yoctoproject.org

With downloads.yoctoproject.org down again this begs the question:

Why does uninative tarball get fetched by default from the internet every time?
Why is it in not cached like every other tarball that is downloaded by recipes?
The default behaviour for the yocto project should be to have the tarball stored in the yocto cache such that it is not fetched every time?

Quoted 5 lines

Those are good questions.

As far as I know on my own local builds, the uninative tarball *is* cached locally in DL_DIR. We don't see repeated downloads on the autobuilder either so it appears to work there too.

Yes, the uninative tarball *is* cached but I see repeated downloads.

Quoted 2 lines

If you're not seeing that it would definitely be something to investigate.

I just did a clone of poky.git and the repeated downloads do NOT happen. So I now need to investigate what is different when adding layers and conf settings that is causing the issue.

Quoted 4 lines

Cheers,

Richard

Thanks for confirming the uninative caching should work, but it is not in my environment and I will have to investigate.

Scott

#7 lore raw
Quoted 5 lines · Scott Branden

We found the source of the uninative fetch problem:

SOURCE_MIRROR_URL needs to be set in order for uninative to be picked up from local mirror.

Quoted 8 lines

The standard PREMIRRORS mechanism doesn't work for this single tarball.
So every time a clean poky build took place it always went to the internet to fetch the uninative tarball
the first time and store it in the local build download.

I think this should be clearly documented somewhere in the yocto manual?

That definitely explains things. I think its assumed that people would set appropriate PREMIRRORS to cover uninative in this case but that isn't documented. I'm not even sure we want for people to have to care about this :/.

Can you try this change:

meta/classes/uninative.bbclass +8 -3
index 70799bbf6d..99072e1326 100644
@@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
             # Our games with path manipulation of DL_DIR mean standard PREMIRRORS don't work
             # and we can't easily put 'chksum' into the url path from a url parameter with
             # the current fetcher url handling
-            ownmirror = d.getVar('SOURCE_MIRROR_URL')
-            if ownmirror:
-                localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
+            premirrors = bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))
+            for line in premirrors:
+                try:
+                    (find, replace) = line
+                except ValueError:
+                    continue
+                if find.startswith("http"):
+                    localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))
 
             srcuri = d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" % chksum)
             bb.note("Fetching uninative binary shim from %s" % srcuri)

Hopefully that handles this case more generically.

Cheers,
Richard

#8 lore raw

Hi Richard,

Quoted 33 lines · Richard Purdie
Quoted 5 lines · Scott Branden

We found the source of the uninative fetch problem:

SOURCE_MIRROR_URL needs to be set in order for uninative to be picked up from local mirror.

Quoted 8 lines

The standard PREMIRRORS mechanism doesn't work for this single tarball.
So every time a clean poky build took place it always went to the internet to fetch the uninative tarball
the first time and store it in the local build download.

I think this should be clearly documented somewhere in the yocto manual?

That definitely explains things. I think its assumed that people would set appropriate PREMIRRORS to cover uninative in this case but that isn't documented. I'm not even sure we want for people to have to care about this :/.

Can you try this change:

meta/classes/uninative.bbclass +1 -3
index 70799bbf6d..99072e1326 100644
@@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
             # Our games with path manipulation of DL_DIR mean standard PREMIRRORS don't work
             # and we can't easily put 'chksum' into the url path from a url parameter with
             # the current fetcher url handling
-            ownmirror = d.getVar('SOURCE_MIRROR_URL')
-            if ownmirror:
-                localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
+            premirrors = bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))

meta/classes/uninative.bbclass python uninative_event_fetchloader() { · line 69

Quoted 10 lines
meta/classes/uninative.bbclass +7 -0
+            for line in premirrors:
+                try:
+                    (find, replace) = line
+                except ValueError:
+                    continue
+                if find.startswith("http"):
+                    localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))
 
             srcuri = d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" % chksum)
             bb.note("Fetching uninative binary shim from %s" % srcuri)

Note is missleading as it is going to fetch from PREMIRRORS.

Quoted 2 lines

Hopefully that handles this case more generically.

Yes, this works.  uninative is now fetched from the PREMIRRORS like all the other downloads even though it prints the note about Fetching uninative binary shim from the URL.

Quoted 7 lines

Cheers,
Richard

#9 lore raw
Quoted 36 lines · Scott Branden
Quoted 28 lines
meta/classes/uninative.bbclass +0 -0
index 70799bbf6d..99072e1326 100644
@@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
             # Our games with path manipulation of DL_DIR mean

standard PREMIRRORS don't work

             # and we can't easily put 'chksum' into the url path
from a url parameter with
             # the current fetcher url handling
meta/classes/uninative.bbclass +0 -3
-            ownmirror = d.getVar('SOURCE_MIRROR_URL')
-            if ownmirror:
-                localdata.appendVar("PREMIRRORS", "

${UNINATIVE_URL}${UNINATIVE_TARBALL} ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
+ premirrors =
bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))

meta/classes/uninative.bbclass +7 -0
+            for line in premirrors:
+                try:
+                    (find, replace) = line
+                except ValueError:
+                    continue
+                if find.startswith("http"):
+                    localdata.appendVar("PREMIRRORS", "

${UNINATIVE_URL}${UNINATIVE_TARBALL} %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))

Quoted 5 lines

srcuri =
d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" % chksum)
bb.note("Fetching uninative binary shim from %s" %
srcuri)

Note is missleading as it is going to fetch from PREMIRRORS.

Quoted 4 lines
Quoted 1 line

Hopefully that handles this case more generically.

Yes, this works. uninative is now fetched from the PREMIRRORS like all the other downloads even though it prints the note about Fetching uninative binary shim from the URL.

Thanks, I've tweaked the patch to improve the note and submitted it. Good to get to the bottom of this!

Cheers,

Richard