Fetch failure for source at kernel.org
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
Thanks,
Philip
Quoted 5 lines · Philip Balister
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
Quoted 7 lines
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
Myself, I use:
PREMIRRORS ?= "\
bzr://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ cvs://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ git://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ hg://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ osc://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ p4://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ svk://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \ svn://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
MIRRORS =+ "\
ftp://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
http://.*/.* http://autobuilder.yoctoproject.org/sources/ \n \
https://.*/.* http://autobuilder.yoctoproject.org/sources/ \n"
PREMIRRORS are tried before the upstream source, MIRRORS are tried only after a failure. (Please correct me someone if I have that wrong.)
--Mark
Quoted 8 lines
Thanks,
Philip
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 10 lines · Philip Balister
Quoted 9 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Quoted 5 lines
Quoted 3 lines
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
regards,
Koen
Quoted 16 lines · Koen Kooi
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 14 lines · Philip Balister
Quoted 12 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
Quoted 6 lines
Quoted 1 line
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
Are you setting BB_GENERATE_MIRROR_TARBALLS?
Cheers,
Richard
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Koen Kooi
Quoted 18 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 15 lines · Philip Balister
Quoted 5 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
Quoted 7 lines
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
Quoted 7 lines
Quoted 4 lines
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
Quoted 26 lines · Koen Kooi
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 24 lines · Koen Kooi
Quoted 22 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 10 lines · Philip Balister
Quoted 9 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Quoted 5 lines
Quoted 3 lines
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Quoted 6 lines
Quoted 2 lines
Is this a versioned tarball problem or just the tarballs not being generated at all?
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting since the fetcher is efficient and will incrementally update something that is incomplete.
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
Cheers,
Richard
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
Quoted 30 lines · Koen Kooi
Quoted 29 lines
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 26 lines · Koen Kooi
Quoted 16 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 14 lines · Philip Balister
Quoted 12 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
Quoted 6 lines
Quoted 1 line
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
Quoted 7 lines
Quoted 3 lines
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads.
Quoted 2 lines
since the fetcher is efficient and
will incrementally update something that is incomplete.
Which won't work currently since kernel.org is down.
Quoted 4 lines
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
regards,
Koen
Quoted 48 lines · Koen Kooi
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
Quoted 30 lines · Koen Kooi
Quoted 29 lines
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 26 lines · Koen Kooi
Quoted 16 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 14 lines · Philip Balister
Quoted 12 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
Quoted 6 lines
Quoted 1 line
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
Quoted 7 lines
Quoted 3 lines
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads.
Quoted 2 lines
since the fetcher is efficient and
will incrementally update something that is incomplete.
Quoted 8 lines
Which won't work currently since kernel.org is down.
Quoted 4 lines
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
FWIW: I would also like to have versioned git checkouts in fetcher2 as it was in old fetcher. It would be more consitent with svn versioned checkouts which are still created.
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Op 22 sep. 2011, om 10:45 heeft Martin Jansa het volgende geschreven:
Quoted 56 lines · Koen Kooi
Quoted 55 lines
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
Quoted 30 lines · Koen Kooi
Quoted 29 lines
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 26 lines · Koen Kooi
Quoted 16 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 14 lines · Philip Balister
Quoted 12 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
Quoted 6 lines
Quoted 1 line
This is a feature, as people did not want oe-core tied to a given mirror.
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
Quoted 7 lines
Quoted 3 lines
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads.
Quoted 2 lines
since the fetcher is efficient and
will incrementally update something that is incomplete.Which won't work currently since kernel.org is down.
Quoted 4 lines
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
Quoted 5 lines
Quoted 1 line
But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
FWIW: I would also like to have versioned git checkouts in fetcher2 as it was in old fetcher. It would be more consitent with svn versioned checkouts which are still created.
http://patches.openembedded.org/patch/11971/ adds a global flag to generate such tarballs, but I don't know if the mirroring logic will actually try to fetch those or if fetch2 will actually try to unpack them when present, but at least we can generate the tarballs now :)
regards,
Koen
Quoted 34 lines · Koen Kooi
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
Quoted 33 lines · Koen Kooi
Quoted 31 lines
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Koen Kooi
Quoted 18 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 15 lines · Philip Balister
Quoted 5 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
Quoted 7 lines
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
Quoted 7 lines
Quoted 4 lines
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
Quoted 8 lines
Quoted 3 lines
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
Well, it depends on your perspective, I was looking at this from a user's point of view. The server's bandwidth is nothing compared to the users pulling the data.
Quoted 3 lines
The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads.
Fair enough but you could make an exception for the git scm tarballs (which follow a fairly identifiable pattern).
Quoted 4 lines
Quoted 2 lines
since the fetcher is efficient and
will incrementally update something that is incomplete.Which won't work currently since kernel.org is down.
This just means there aren't appropriate git:// -> git:// mappings in people's mirrors files (including Yocto). If git:// -> git:// mappings don't work, we should fix that but they should.
Quoted 10 lines
Quoted 4 lines
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
Now someone has explained the issue, yes, I can see why it is causing problems. I do think some of them are of people's own choosing but I'm trying to be flexible and not make judgement on that.
So we can do something along the lines of what you after but there are the following considerations:
a) The versioned tarballs do need to be different to what we had before in that the tarball will be of a .git directory containing the git metadata, not just a checkout. Why? This is so we can incrementally update a git repo if the user changes the source revision as one example. We're dealing with the git fetcher and limiting ourselves to a flat one dimension doesn't make sense.
b) We need to agree what order the tarballs on the server should be searched for. The "obvious" order would be:
1. Specific Revision X tarball
2. generic mirror tarball
3. git clone
The problems come about when we have an existing git checkout which doesn't contain the revision we want. Currently we can just update and assume that will bring in the revision. If we now support people doing rebases and other things, we now at least in theory have to:
1. See if the revision exists locally
2. Try pulling
3. If that doesn't work look for a versioned tarball
4. Blow away all fetcher git status from DL_DIR
5. Extract the tarball
My big concern with this is rather than having some general archictecture and standard way of doing this, the fetcher once again becomes a set of special case if A do B but if C do D first then do E but don't forget to do A if Z happens type code.
So what I'm saying is I don't want the fetcher to become the unmaintainable mess the that the code paths in fetch1 were. I don't honestly know how to add a code path like the above and avoid this :(
The patch on the bitbake list doesn't consider any of this complexity or the corner cases :(
Cheers,
Richard
Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:
Quoted 48 lines · Koen Kooi
Quoted 34 lines
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende geschreven:
Quoted 33 lines · Koen Kooi
Quoted 31 lines
Op 22 sep. 2011, om 02:26 heeft Richard Purdie het volgende geschreven:
Quoted 19 lines · Koen Kooi
Quoted 18 lines
Op 21 sep. 2011, om 21:43 heeft Mark Hatle het volgende geschreven:
Quoted 15 lines · Philip Balister
Quoted 5 lines
I'm finally trying to get started with oe-core. I'm using a setup created with the setup-script created by Koen. Everything looks ok until it tried to build cpufrequtils. The recipe tries to fetch the source from a git repo at kernel.org. Obviously, the fetch fails.
Quoted 7 lines
What is odd, in the "good old days" I would expect this config to fall back and use the Angstrom source mirror, but apparently this is not the case with oe-core.
Is this a bug or a feature? If this is a bug, is there a plan to fix it? If it is a feature, how are we supposed to do builds if servers go away? What do we do about GPL compliance?
This is a feature, as people did not want oe-core tied to a given mirror.
Quoted 7 lines
Quoted 4 lines
This is the "OE core doesn't generate versioned git tarballs" problem people keep talking about. I know I can edit each and every recipe to add ';rebasable=true', but I'd like a global solution before the GPL police starts looking at the angstrom source mirrors.
Is this a versioned tarball problem or just the tarballs not being generated at all?
versioned tarball problem, since the source mirror won't overwrite files with the same name, so unversioned ones are both useless and bandwidth wasting.
Quoted 8 lines
Quoted 3 lines
They seem to be working perfectly well for Yocto...
They're also not bandwidth wasting
They are, since for the kernel you need to re-upload the 300MB tarball to the source mirror for mirroring to work, even if there were no real changes but bitbake repacked it. I know that versioned tarballs aren't terribly efficient either, but at least they are deterministic.
Well, it depends on your perspective, I was looking at this from a user's point of view. The server's bandwidth is nothing compared to the users pulling the data.
Quoted 6 lines
Quoted 3 lines
The angstrom source mirror checks for duplicate filenames to avoid introducing checksum changes due to upstream stupidity or corrupted downloads.
Fair enough but you could make an exception for the git scm tarballs (which follow a fairly identifiable pattern).
That still opens the door for people to upload "old" tarballs with less revs in them. No solution is perfect, though.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
since the fetcher is efficient and
will incrementally update something that is incomplete.Which won't work currently since kernel.org is down.
This just means there aren't appropriate
git://->git://mappings in people's mirrors files (including Yocto). Ifgit://->git://mappings don't work, we should fix that but they should.
You mean something like git.kernel.org/scm/torvalds/linux-2.6.git -> github.com/torvalds/linux.git ?
Quoted 18 lines
Quoted 10 lines
Quoted 4 lines
Anyhow, this is the first time someone has explained that the mirror tarball improvements that were made to bitbake have not done everything they desire. If you want someone on the Yocto team to look at this I'd suggest a bug report or I'll take patches to bitbake (as always).
But you do acknowledge the problem? FWIW, I've been arguing for a global method of enabled versioned tarballs since I first started using oe-core (called differently back then). I know linux-yocto stretches the fetcher to its limits, but I'd like this change for all the recipes using a single git tree with a single srcrev.
Now someone has explained the issue, yes, I can see why it is causing problems. I do think some of them are of people's own choosing but I'm trying to be flexible and not make judgement on that.
So we can do something along the lines of what you after but there are the following considerations:
Quoted 6 lines
a) The versioned tarballs do need to be different to what we had before in that the tarball will be of a .git directory containing the git metadata, not just a checkout. Why? This is so we can incrementally update a git repo if the user changes the source revision as one example. We're dealing with the git fetcher and limiting ourselves to a flat one dimension doesn't make sense.
Agreed on that. The bitbake patch I sent does at least do that :)
Quoted 9 lines
b) We need to agree what order the tarballs on the server should be searched for. The "obvious" order would be:
1. Specific Revision X tarball
2. generic mirror tarball
3. git cloneThe problems come about when we have an existing git checkout which doesn't contain the revision we want.
There currently is a bug in the fetcher that makes it hard to use now that kernel.org is down. The situation:
I have a recipe that pulls from git (e.g. cpufrequtils.git from kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a -c fetch -f it will try to do an update and fail since kernel.org is down. I would have expected it to check the existing bare clone for deadbeef first and not try updating if it is present.
Anyway, I agree on the search order above.
Quoted 9 lines
Currently we can just update and
assume that will bring in the revision. If we now support people doing rebases and other things, we now at least in theory have to:1. See if the revision exists locally
2. Try pulling
3. If that doesn't work look for a versioned tarball
4. Blow away all fetcher git status fromDL_DIR
5. Extract the tarball
Since git supports local fetches the above can be:
1) see if rev exists
2) try pulling
3) try versioned tarball
4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repo
That would stop things from being blown away. It does allow the versioned snapshots to differ, but they will always have the rev they were versioned with.
Quoted 4 lines
My big concern with this is rather than having some general archictecture and standard way of doing this, the fetcher once again becomes a set of special case if A do B but if C do D first then do E but don't forget to do A if Z happens type code.
I know virtually nothing of the fetcher code, but I think treating every tarball as a local mirror that gets fetched into DL_DIR/git2/repodir would avoid a lot of special casing.
Quoted 6 lines
So what I'm saying is I don't want the fetcher to become the unmaintainable mess the that the code paths in fetch1 were. I don't honestly know how to add a code path like the above and avoid this :(
The patch on the bitbake list doesn't consider any of this complexity or the corner cases
Sadly it's the best thing I can accomplish with my python skills :( It does solve my #1 need, which is an easy way (ls | grep) to check for GPL compliance.
regards,
Koen
Quoted 10 lines · Koen Kooi
Op 22 sep 2011, om 12:20 heeft Richard Purdie het volgende geschreven:
Quoted 5 lines · Koen Kooi
Quoted 1 line
Op 22 sep. 2011, om 09:03 heeft Richard Purdie het volgende
This just means there aren't appropriate
git://->git://mappings in people's mirrors files (including Yocto). Ifgit://->git://mappings don't work, we should fix that but they should.You mean something like git.kernel.org/scm/torvalds/linux-2.6.git -> github.com/torvalds/linux.git ?
Yes. Thinking about this, the current fetcher might not be optimised in making this as efficient as possible but it should work and if it doesn't, we should make it.
Quoted 16 lines
Quoted 7 lines
Now someone has explained the issue, yes, I can see why it is causing problems. I do think some of them are of people's own choosing but I'm trying to be flexible and not make judgement on that.
So we can do something along the lines of what you after but there are the following considerations:
Quoted 8 lines
a) The versioned tarballs do need to be different to what we had before
in that the tarball will be of a .git directory containing the git metadata, not just a checkout. Why? This is so we can incrementally update a git repo if the user changes the source revision as one example. We're dealing with the git fetcher and limiting ourselves to a
flat one dimension doesn't make sense.
Quoted 1 line
Agreed on that. The bitbake patch I sent does at least do that :)
Right :)
I just want to be clear we're doing something different to what was there before (and why).
Quoted 13 lines
Quoted 9 lines
b) We need to agree what order the tarballs on the server should be searched for. The "obvious" order would be:
1. Specific Revision X tarball
2. generic mirror tarball
3. git cloneThe problems come about when we have an existing git checkout which doesn't contain the revision we want.
There currently is a bug in the fetcher that makes it hard to use now that kernel.org is down. The situation:
Quoted 6 lines
I have a recipe that pulls from git (e.g. cpufrequtils.git from kernel.org) with SRCREV fixed to e.g. deadbeef. DL_DIR/git/ git.kernel.org.utils.cpufrequtils has revision deadbeef. When I do a - c fetch -f it will try to do an update and fail since kernel.org is down. I would have expected it to check the existing bare clone for deadbeef first and not try updating if it is present.
Agreed, it shouldn't be hitting the network unless the revision is floating. That is bad :/.
Quoted 14 lines
Anyway, I agree on the search order above.
Quoted 9 lines
Currently we can just update and
assume that will bring in the revision. If we now support people doing rebases and other things, we now at least in theory have to:1. See if the revision exists locally
2. Try pulling
3. If that doesn't work look for a versioned tarball
4. Blow away all fetcher git status fromDL_DIR
5. Extract the tarballSince git supports local fetches the above can be:
Quoted 8 lines
1) see if rev exists
2) try pulling
3) try versioned tarball
4) extract tarball into tmpdir, fetch objects into DL_DIR/git2 repoThat would stop things from being blown away. It does allow the versioned snapshots to differ, but they will always have the rev they were versioned with.
This is a much better approach. Its quite a big difference from anything we've done before but the approach should fit in better. I need to think on it more...
Quoted 8 lines
Quoted 4 lines
My big concern with this is rather than having some general archictecture and standard way of doing this, the fetcher once again becomes a set of special case if A do B but if C do D first then do E but don't forget to do A if Z happens type code.
I know virtually nothing of the fetcher code, but I think treating every tarball as a local mirror that gets fetched into DL_DIR/git2/ repodir would avoid a lot of special casing.
Yes, its nicer. We keep falling into the trap of thinking about the non-distributed SCM cases but we have more capability with git and it makes sense to use it.
Quoted 11 lines
Quoted 7 lines
So what I'm saying is I don't want the fetcher to become the unmaintainable mess the that the code paths in fetch1 were. I don't honestly know how to add a code path like the above and avoid this :(
The patch on the bitbake list doesn't consider any of this complexity or
the corner casesSadly it's the best thing I can accomplish with my python skills :( It does solve my #1 need, which is an easy way (ls | grep) to check for GPL compliance.
Ok. Can you at least file a bug in the yocto bugzilla with a summary of this discussion please? :) That should ensure the issue gets more attention since with the best intentions, I can't remember everything.
Cheers,
Richard
Op 22 sep. 2011, om 13:42 heeft Richard Purdie het volgende geschreven:
[..]
Quoted 3 lines
Ok. Can you at least file a bug in the yocto bugzilla with a summary of this discussion please? :) That should ensure the issue gets more attention since with the best intentions, I can't remember everything.
I filed it as http://bugzilla.pokylinux.org/show_bug.cgi?id=1511