DISTRO_MIRRORS / DISTRO_PREMIRRORS

RFC In review Saul Wold 4 participants 7 messages Jun 15, 2011 – Jun 16, 2011

lore

Share & tools

lore

Saul Wold

author

2011-06-15 16:08 UTC #0 lore raw

RFC 1/1 DISTRO_MIRRORS / DISTRO_PREMIRRORS

Folks,

Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).

The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.

As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.

I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.

meta/classes/mirrors.bbclass +18 -0

index 2146087..a52c6d5 100644
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.*   http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.*      http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.*     http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.*    http://autobuilder.yoctoproject.org/sources/ \n"
+
 MIRRORS += "\
+${DISTRO_MIRRORS} \
 ${DEBIAN_MIRROR}/main  http://snapshot.debian.net/archive/pool \n \
 ${DEBIAN_MIRROR}       ftp://ftp.de.debian.org/debian/pool \n \
 ${DEBIAN_MIRROR}       ftp://ftp.au.debian.org/debian/pool \n \

Thoughts, ACKs, NAKs??

--
Sau!

Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System

Mark Hatle

2011-06-15 16:14 UTC #1 lore raw

Quoted 21 lines · Saul Wold

Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.

...

Quoted 2 lines

Thoughts, ACKs, NAKs??

The only concern I have is that we may miss when an upstream source location changes, thus not fixing a recipe when a package moves, disappears and otherwise can't be downloaded.

--Mark

Quoted 12 lines

--
Sau!
Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Khem Raj

2011-06-15 16:16 UTC #3 lore raw

In reply to Saul Wold

Quoted 49 lines · Saul Wold

Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.
meta/classes/mirrors.bbclass +18 -0
index 2146087..a52c6d5 100644
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
 MIRRORS += "\
+${DISTRO_MIRRORS} \
 ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \

Quoted 2 lines

Thoughts, ACKs, NAKs??

FWIW http://mirrors.openembedded.org/ also exists.

Quoted 13 lines

--
Sau!
Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Khem Raj

2011-06-15 16:26 UTC #2 lore raw

In reply to Mark Hatle

Quoted 23 lines · Mark Hatle

Quoted 15 lines · Saul Wold
Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
Quoted 6 lines
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.

Quoted 9 lines

...
Quoted 2 lines
Thoughts, ACKs, NAKs??
The only concern I have is that we may miss when an upstream source location changes, thus not fixing a recipe when a package moves, disappears and otherwise can't be downloaded.

yeah thats why mirror.openembedded.org usually contains the sources that are not found anywhere.

Quoted 20 lines

--Mark
Quoted 8 lines
--
Sau!
Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Saul Wold

author

2011-06-15 16:31 UTC #4 lore raw

In reply to Khem Raj

Quoted 50 lines · Khem Raj

Quoted 49 lines · Saul Wold
Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.
meta/classes/mirrors.bbclass +18 -0
index 2146087..a52c6d5 100644
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
 MIRRORS += "\
+${DISTRO_MIRRORS} \
 ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \

Quoted 5 lines

Quoted 2 lines
Thoughts, ACKs, NAKs??
FWIW http://mirrors.openembedded.org/ also exists.
Quoted 1 line

How up to date is it and how is it maintained? Is it possible for this mirror site to fall behind oe-core?

Sau!

Quoted 19 lines

Quoted 8 lines
--
Sau!
Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Khem Raj

2011-06-15 16:33 UTC #5 lore raw

Quoted 51 lines · Saul Wold

Quoted 50 lines · Khem Raj
Quoted 49 lines · Saul Wold
Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.
meta/classes/mirrors.bbclass +18 -0
index 2146087..a52c6d5 100644
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
 MIRRORS += "\
+${DISTRO_MIRRORS} \
 ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \

Quoted 8 lines

Quoted 5 lines
Quoted 2 lines
Thoughts, ACKs, NAKs??
FWIW http://mirrors.openembedded.org/ also exists.
Quoted 1 line
How up to date is it and how is it maintained? Is it possible for this mirror site to fall behind oe-core?

oe.dev falls back to it by default and whenever there is a report of non existing tars they are put up there.

Quoted 21 lines

Sau!
Quoted 19 lines
Quoted 8 lines
--
Sau!
Saul Wold
Yocto Component Wrangler @ Intel
Yocto Project / Poky Build System
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Tom Rini

2011-06-16 00:35 UTC #6 lore raw

In reply to Khem Raj

Quoted 50 lines · Khem Raj

Quoted 49 lines · Saul Wold
Folks,
Recently we have seen a number of fetch failures, due to upstream sites being down or deprecated (tinylogin, pscmic and others).
The issue is that oe-core as a non-distro does not have a default set of mirrors, which causes the fetcher to fail as there is no fall-back mirror to distrubte the source code.
As has been pointed out, different projects and distributions (Yocto Project and Angstrom) both have upstream source mirrors which could be used. Currently the Yocto Project defines it's MIRRORS and PREMIRRORS in meta-yocto/conf/distro/poky.conf.
I propose that we add a list of DISTO_MIRRORS and DISTRO_PREMIRRORS with default settings pointing to the current autobuilder.yoctoproject.org source site. These would be added to the mirrors.bbclass, so that oe-core will have a default set of mirrors that can be overridden by a distro or layer.
meta/classes/mirrors.bbclass +18 -0
index 2146087..a52c6d5 100644
@@ -1,4 +1,22 @@
+DISTRO_PREMIRRORS ?= "\
+bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
+PREMIRRORS += "${DISTRO_PREMIRRORS}"
+
+DISTRO_MIRRORS ?= "\
+ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
+https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
+
 MIRRORS += "\
+${DISTRO_MIRRORS} \
 ${DEBIAN_MIRROR}/main http://snapshot.debian.net/archive/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.de.debian.org/debian/pool \n \
 ${DEBIAN_MIRROR} ftp://ftp.au.debian.org/debian/pool \n \

Quoted 4 lines

Quoted 2 lines
Thoughts, ACKs, NAKs??
FWIW http://mirrors.openembedded.org/ also exists.

To be clear, mirrors.openembedded.org is a "last resort" type of mirror (eg the tinylogin case) whereas autobuilder.yoctoproject.org and the angstrom mirrors are complete (modulo fetch2 git issues, on the angstrom side) mirrors.

Both have their uses and I think for oe-core we want just the former (so that when tzcode/tzdata get their weekly upgrade, we know) while poky, angstrom, etc, etc, will continue to want what they have now.

Signature · 2 lines

Tom Rini
Mentor Graphics Corporation