How to use externalsrc with git workspace

DISCUSSION Cliff Brake 2 participants 8 messages Mar 25, 2016 – Mar 29, 2016

lore

Share & tools

lore

Cliff Brake

author

2016-03-25 20:18 UTC #0 lore raw

Hi,

In the past (say Dora era), I've used externalsrc like:

====================
DESCRIPTION = "MyApp"
LICENSE = "CLOSED"

inherit externalsrc

SRCREV = "${AUTOREV}"
PV = "1.4+gitr${SRCPV}"
PR = "r1"

SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD" S = "${FILE_DIRNAME}"

...
====================

This was very nice in that it set SRCPV to to the git version of the HEAD of my local workspace. When building from a local git workspace, this makes sense to me -- I want the Git version of the workspace, not a remote version.

Now with Jethro, I get errors like:

ERROR: ExpansionError during parsing myapp.bb: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'HEAD' in upstream git repository in git ls-remote output for /scratch/externalsrc/MyApp

Do you have any suggestions how to use externalsrc + a local git workspace and avoid the above error. I can set branch=master, etc and bitbake will then parse it, but the remote master version is not really what I'm building.

Thanks,
Cliff

Cliff Brake

author

2016-03-25 21:07 UTC #1 lore raw

This change gets things parsing again, but still testing to see if it does what I want it to do:

lib/bb/fetch2/git.py +1 -1

index 9bd87ad..fb44fdc 100644
@@ -358,7 +358,7 @@ class Git(FetchMethod):
         """
         output = self._lsremote(ud, d, "")
         # Tags of the form ^{} may not work, need to fallback to other form
-        if ud.unresolvedrev[name][:5] == "refs/":
+        if ud.unresolvedrev[name][:5] == "refs/" or ud.unresolvedrev[name] ==

"HEAD":
head = ud.unresolvedrev[name]
tag = ud.unresolvedrev[name]
else:

Cliff

Quoted 38 lines · Cliff Brake

Hi,
In the past (say Dora era), I've used externalsrc like:
====================
DESCRIPTION = "MyApp"
LICENSE = "CLOSED"
inherit externalsrc
SRCREV = "${AUTOREV}"
PV = "1.4+gitr${SRCPV}"
PR = "r1"
SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD" S = "${FILE_DIRNAME}"
...
====================
This was very nice in that it set SRCPV to to the git version of the HEAD of my local workspace. When building from a local git workspace, this makes sense to me -- I want the Git version of the workspace, not a remote version.
Now with Jethro, I get errors like:
ERROR: ExpansionError during parsing myapp.bb: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'HEAD' in upstream git repository in git ls-remote output for /scratch/externalsrc/MyApp
Do you have any suggestions how to use externalsrc + a local git workspace and avoid the above error. I can set branch=master, etc and bitbake will then parse it, but the remote master version is not really what I'm building.
Thanks,
Cliff

Richard Purdie

2016-03-26 07:41 UTC #2 lore raw

In reply to Cliff Brake

Quoted 36 lines · Cliff Brake

Hi,
In the past (say Dora era), I've used externalsrc like:
====================
DESCRIPTION = "MyApp"
LICENSE = "CLOSED"
inherit externalsrc
SRCREV = "${AUTOREV}"
PV = "1.4+gitr${SRCPV}"
PR = "r1"
SRC_URI = "git://${FILE_DIRNAME};protocol=file;branch=HEAD" S = "${FILE_DIRNAME}"
...
====================
This was very nice in that it set SRCPV to to the git version of the HEAD of my
local workspace. When building from a local git workspace, this makes sense to
me -- I want the Git version of the workspace, not a remote version.
Now with Jethro, I get errors like:
ERROR: ExpansionError during parsing myapp.bb: Failure expanding variable SRCPV,
expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError:
Fetcher failure: Unable to resolve 'HEAD' in upstream git repository in git
ls-remote output for /scratch/externalsrc/MyApp

Quoted 5 lines

Do you have any suggestions how to use externalsrc + a local git workspace and
avoid the above error. I can set branch=master, etc and bitbake will then parse
it, but the remote master version is not really what I'm building.

HEAD is a revision, not a branch and the fetcher became a bit more picky about that difference. Your local tree must have HEAD on some branch which would usually be master so setting branch=master is likely correct, or don't set that parameter at all? master isn't just a remote thing.

The piece which says "use the HEAD revision" is the SRCREV = "${AUTOREV}" line.

Cheers,

Richard

Cliff Brake

author

2016-03-28 19:58 UTC #3 lore raw

Quoted 6 lines · Richard Purdie

HEAD is a revision, not a branch and the fetcher became a bit more picky about that difference. Your local tree must have HEAD on some branch which would usually be master so setting branch=master is likely correct, or don't set that parameter at all? master isn't just a remote thing.

Quoted 7 lines

The piece which says "use the HEAD revision" is the SRCREV = "${AUTOREV}" line.
Cheers,
Richard

We typically set up our projects to use git submodules, such that the top level calls out a set of submodules which are the actively developed applications and other bits in the system. These submodules are externalsrc trees for bitbake, and include a bitbake recipe (bitbake recipe is embedded directly in source code). Sometimes these submodules/externalsrc trees track branches (typically release-xyz, or develop-xyz), but almost never master. Sometimes they may be at a tagged version that is not at the HEAD of a branch. In the end, I want to be able to build exactly what is in the externalsrc directory, and accurately include the git version in the PV. Branches/tags may change at any time and I don't want to have to edit the recipe every time this happens.

Repo might be used in a similar fashion to manage a tree of components that get built. What we are trying to avoid is having to edit the git version in the recipes to lock down versions for a set of components for a build -- git submodules or repo work much better for doing this.

I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.

The git fetcher could have an option like head=1 in the SRC_URI, that told it to use whatever is in the git workspace, and allow AUTOREV to figure out version.

Perhaps a better approach might be to figure out a function for PV that simply extracts the git version of ${S} -- that is really all I want. In this case, I might even need a SRC_URI, as I'm not really fetching anything -- its already there.

Thanks,
Cliff

Richard Purdie

2016-03-28 22:28 UTC #4 lore raw

Quoted 44 lines · Cliff Brake

Quoted 8 lines · Richard Purdie
HEAD is a revision, not a branch and the fetcher became a bit more picky about that difference. Your local tree must have HEAD on some branch which would usually be master so setting branch=master is likely
correct, or don't set that parameter at all? master isn't just a remote
thing.
Quoted 7 lines
The piece which says "use the HEAD revision" is the SRCREV = "${AUTOREV}" line.
Cheers,
Richard
We typically set up our projects to use git submodules, such that the top level
calls out a set of submodules which are the actively developed applications and
other bits in the system. These submodules are externalsrc trees for bitbake,
and include a bitbake recipe (bitbake recipe is embedded directly in source
code). Sometimes these submodules/externalsrc trees track branches (typically
release-xyz, or develop-xyz), but almost never master. Sometimes they may be at
a tagged version that is not at the HEAD of a branch. In the end, I want to be
able to build exactly what is in the externalsrc directory, and accurately
include the git version in the PV. Branches/tags may change at any time and I
don't want to have to edit the recipe every time this happens.
Repo might be used in a similar fashion to manage a tree of components that get
built. What we are trying to avoid is having to edit the git version in the
recipes to lock down versions for a set of components for a build -- git
submodules or repo work much better for doing this.

Quoted 1 line

I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.

Did you just try nobranch=1 and leave rev to get set from SRCREV?

Cheers,

Richard

Cliff Brake

author

2016-03-29 12:25 UTC #5 lore raw

Quoted 28 lines · Richard Purdie

Quoted 28 lines
We typically set up our projects to use git submodules, such that the top level
calls out a set of submodules which are the actively developed applications and
other bits in the system. These submodules are externalsrc trees for bitbake,
and include a bitbake recipe (bitbake recipe is embedded directly in source
code). Sometimes these submodules/externalsrc trees track branches (typically
release-xyz, or develop-xyz), but almost never master. Sometimes they may be at
a tagged version that is not at the HEAD of a branch. In the end, I want to be
able to build exactly what is in the externalsrc directory, and accurately
include the git version in the PV. Branches/tags may change at any time and I
don't want to have to edit the recipe every time this happens.
Repo might be used in a similar fashion to manage a tree of components that get
built. What we are trying to avoid is having to edit the git version in the
recipes to lock down versions for a set of components for a build -- git
submodules or repo work much better for doing this.

Quoted 2 lines

Quoted 1 line
I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.
Did you just try nobranch=1 and leave rev to get set from SRCREV?

That will parse, but it puts the wrong git rev in the PV:

The following package gets built:

blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk

But, the actual version of the source that gets built is 0edb656fc9a1a77d75eaa49a17586deb4c986bb1.

a97... is the version of the branch (default is master) in the workspace.

So this would work if we did not care what gitv got put in the PV. There is also the issue of bitbake not rebuilding the source if you change the git version of the workspace, but the branch happens to not change.

Does no one else use externalsrc with Git?

Thanks,
Cliff

Richard Purdie

2016-03-29 12:28 UTC #6 lore raw

Quoted 14 lines · Cliff Brake

Quoted 3 lines · Richard Purdie
Quoted 2 lines
I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.
Did you just try nobranch=1 and leave rev to get set from SRCREV?
That will parse, but it puts the wrong git rev in the PV:
The following package gets built:
blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk
But, the actual version of the source that gets built is 0edb656fc9a1a77d75eaa49a17586deb4c986bb1.

Quoted 8 lines

a97... is the version of the branch (default is master) in the workspace.
So this would work if we did not care what gitv got put in the PV.
There is
also the issue of bitbake not rebuilding the source if you change the git
version of the workspace, but the branch happens to not change.

At least from my perspective, this is what *should* work and the fact it doesn't is a bug somewhere. Adding extra things to the fetcher would likely just workaround it. Filing a bug report for it would be the likely next step.

Cheers,

Richard

Cliff Brake

author

2016-03-29 13:50 UTC #7 lore raw

Quoted 22 lines · Richard Purdie

Quoted 13 lines · Cliff Brake
Quoted 3 lines · Richard Purdie
Quoted 2 lines
I tried nobranch=1;rev=HEAD, but then that conflicts with ${AUTOREV}.
Did you just try nobranch=1 and leave rev to get set from SRCREV?
That will parse, but it puts the wrong git rev in the PV:
The following package gets built:
blkqcl-app-tune_1.4+gitr1+a97fcb58e6-r1.0_cortexa9hf-vfp-neon.ipk
But, the actual version of the source that gets built is 0edb656fc9a1a77d75eaa49a17586deb4c986bb1.
Quoted 8 lines
a97... is the version of the branch (default is master) in the workspace.
So this would work if we did not care what gitv got put in the PV.
There is
also the issue of bitbake not rebuilding the source if you change the git
version of the workspace, but the branch happens to not change.

Quoted 4 lines

At least from my perspective, this is what *should* work and the fact it doesn't is a bug somewhere. Adding extra things to the fetcher would likely just workaround it. Filing a bug report for it would be the likely next step.

Thanks, bug submitted: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9351

Cliff