Skip to contentSkip to conversation

BB_NO_NETWORK broken for git

DISCUSSION Gary Thomas 2 participants 6 messages Dec 5, 2013

lore
Share & tools
lore
#0 lore raw

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

bitbake: git: Use merge-base instead of log for testing if a commit is present

The current use of git log to check if a given revision is present can be a little fragile.

For example if revision X was on branch A, and then later added to branch B, the update checks would not notice this since they just check for X being in the repository.

We also had some autobuilder corruption where an older packed-refs file was copied over a new repository containing newer pack files. There was no update to the refs file since the revision was present but not accessible in any branch.

The correct fix is to check that the required revisions are present on the specific branches. This patch does this using merge-base.

(Bitbake rev: 89abfbc1953e3711d6c90aff793ee622c22609b1)

To test this, just add these settings to local.conf
SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors"
BB_NO_NETWORK = "1" BB_GENERATE_MIRROR_TARBALLS ?= "1"

Make sure and put the appropriate git tarball into the SOURCE_MIRROR. I tested this using prelink, e.g.

  $ bitbake prelink -c fetch
the the tarball is git2_git.yoctoproject.org.prelink-cross.git.tar.gz
A copy of this is at
  http://www.mlbassoc.com/poky/git2_git.yoctoproject.org.prelink-cross.git.tar.gz

Filed as bug #5631 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=5631)

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#1 lore raw
Quoted 8 lines · Gary Thomas

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

Its actually the metadata that has the issue. The commit specified is not on the master branch, its on the cross_prelink branch so the fetcher is justifiably getting upset.

When the network is present it does a complete reset to upstream and then assumes the revision is present though which is something we might want to fix too.

For now we can at least correct the metadata to point at the right branch. I've sent out a patch accordingly.

Cheers,

Richard

#2 lore raw
Quoted 14 lines · Richard Purdie
Quoted 8 lines · Gary Thomas

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

Its actually the metadata that has the issue. The commit specified is not on the master branch, its on the cross_prelink branch so the fetcher is justifiably getting upset.

Quoted 6 lines

When the network is present it does a complete reset to upstream and then assumes the revision is present though which is something we might want to fix too.

For now we can at least correct the metadata to point at the right branch. I've sent out a patch accordingly.

Does this work for other git based repositories? My example was for prelink, but it seems to fail for me for any SRC_URI that uses git. For example, matchbox-keyboard also fails under the same test scenario.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#3 lore raw
Quoted 19 lines · Gary Thomas
Quoted 18 lines · Richard Purdie
Quoted 8 lines · Gary Thomas

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

Its actually the metadata that has the issue. The commit specified is not on the master branch, its on the cross_prelink branch so the fetcher is justifiably getting upset.

When the network is present it does a complete reset to upstream and then assumes the revision is present though which is something we might want to fix too.

Quoted 6 lines
Quoted 2 lines

For now we can at least correct the metadata to point at the right branch. I've sent out a patch accordingly.

Does this work for other git based repositories? My example was for prelink, but it seems to fail for me for any SRC_URI that uses git. For example, matchbox-keyboard also fails under the same test scenario.

Its the same problem. The commit referenced is not on master but on a different branch. The URL needs fixing to point at the right branch.

Cheers,

Richard

#4 lore raw
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Gary Thomas
Quoted 20 lines · Richard Purdie
Quoted 8 lines · Gary Thomas

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

Its actually the metadata that has the issue. The commit specified is not on the master branch, its on the cross_prelink branch so the fetcher is justifiably getting upset.

When the network is present it does a complete reset to upstream and then assumes the revision is present though which is something we might want to fix too.

For now we can at least correct the metadata to point at the right branch. I've sent out a patch accordingly.

Quoted 6 lines
Quoted 3 lines

Does this work for other git based repositories? My example was for prelink, but it seems to fail for me for any SRC_URI that uses git. For example, matchbox-keyboard also fails under the same test scenario.

Its the same problem. The commit referenced is not on master but on a different branch. The URL needs fixing to point at the right branch.

So I just [luckily] picked two recipes that have incorrect SRC_URI?? How many others will there be? I'd wager most all since very few of them actually mention a branch, just a revision.

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#5 lore raw
Quoted 28 lines · Gary Thomas
Quoted 27 lines · Richard Purdie
Quoted 25 lines · Gary Thomas
Quoted 14 lines · Richard Purdie
Quoted 8 lines · Gary Thomas

I try to use a local source mirror, fully populated with packages, so I can deliver a complete (no network required) bundle to my customers. This has recently been broken.

I can no longer use BB_NO_NETWORK=1 for git recipes, since this revision:
commit 2354250a95eab484459f41f8715ae112295c2174

Author: Richard Purdie <[email protected]>
Date:   Mon Nov 18 17:17:16 2013 +0000

Its actually the metadata that has the issue. The commit specified is not on the master branch, its on the cross_prelink branch so the fetcher is justifiably getting upset.

Quoted 6 lines

When the network is present it does a complete reset to upstream and then assumes the revision is present though which is something we might want to fix too.

For now we can at least correct the metadata to point at the right branch. I've sent out a patch accordingly.

Does this work for other git based repositories? My example was for prelink, but it seems to fail for me for any SRC_URI that uses git. For example, matchbox-keyboard also fails under the same test scenario.

Quoted 6 lines
Quoted 2 lines

Its the same problem. The commit referenced is not on master but on a different branch. The URL needs fixing to point at the right branch.

So I just [luckily] picked two recipes that have incorrect SRC_URI?? How many others will there be? I'd wager most all since very few of them actually mention a branch, just a revision.

Most git trees use the master branch which is the default so most urls are fine.

I just added something to the fetcher to detect this (which I will merge), then ran a world fetchall against OE-Core master.

We appear to have issues with lttng-{modules,ust,tools} and mx so its far from "all" git recipes.

Cheers,

Richard