git: fix handling of git repos using master as version

'+' is a valid character in a url. The current dl-wrapper gets the URI scheme by dropping everything after the last '+' character, with the intension of finding 'git' from e.g. 'git+https://uri'.

If a uri has a '+' anywhere in it, it ends up using too much of the string as a scheme, and fails to match the handler properly.

An example of where this form of URI is used is when using deploy tokens
in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
where username for deploy token is of the form 'gitlab+deploy-token-<number>'.

Use the %% operator to search backwards until the last '+' character when dropping the rest of the string as we know that the first '+' in the string should be the scheme.

Signed-off-by Robert Beckett <[email protected]>

---
 support/download/dl-wrapper | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

support/download/dl-wrapper +1 -1

index 8d6365e..4059c37 100755
@@ -88,7 +88,7 @@ main() {
     download_and_check=0
     rc=1
     for uri in "${uris[@]}"; do
-        backend=${uri%+*}
+        backend=${uri%%+*}
         case "${backend}" in
             git|svn|cvs|bzr|file|scp|hg) ;;
             *) backend="wget" ;;

Signature · 1 line

2.7.4

'+' is a valid character in a url. The current dl-wrapper gets the URI scheme by dropping everything after the last '+' character, with the intension of finding 'git' from e.g. 'git+https://uri'.

If a uri has a '+' anywhere in it, it ends up using too much of the string as a scheme, and fails to match the handler properly.

An example of where this form of URI is used is when using deploy tokens
in gitlab. It uses a form like https://<username>:<password>@gitlab.com/<group>/<repo.git>
where username for deploy token is of the form 'gitlab+deploy-token-<number>'.

Use the %% operator to search backwards until the last '+' character when dropping the rest of the string as we know that the first '+' in the string should be the scheme.

Signed-off-by Robert Beckett <[email protected]>

---
 support/download/dl-wrapper | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Since changing to git init followed by fetch to initialise git cache, when using master branch as the target version for a package, git refuses to fetch in to currently checkout out branch or master.

Hello,

Quoted 4 lines · Robert Beckett

Since changing to git init followed by fetch to initialise git cache, when using master branch as the target version for a package, git refuses to fetch in to currently checkout out branch or master.

I think we don't want to support using branches as Git versions, as it doesn't make sense, because it won't behave as the user expects.

If you use a branch as _VERSION for a package, Buildroot will only download once, and never update to newer commits in this branch, because it will already have a cached tarball in its DL_DIR.

So I'm not sure we want to add some code to support a use case that we don't want support :-)

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering
https://bootlin.com

Is this philosophy set in stone? I can conceive of useful situations where branches are preferable.

During large projects, where the rootfs is being developed at the same time as some custom packages, it is quite useful to have a branch as the target. Usually, I just do a dirclean and delete the tar when I know something has changed.

If you wanted to make it more predictable, then maybe name the tar based on the version (branch name in this case) plus the sha,

this way new tar packages would be created if the branch advances and also if a tag (or other supported refspec) changes via a rebase etc.

The particular thing I like to avoid is multiple version bumps in the package spec and tags in the package source repos during development. When many people are working on package project, expecting them all to take time to publish tags for every little change gets unrealistic.

The question I would ask is is there any harm in supporting that if it can be done reliably and predictably? Anyway, if this design is locked down, then I will just maintain this patch locally for us to use :)

Thanks

Bob
On Mon, Jun 4, 2018 at 3:33 PM, Thomas Petazzoni < [1][email protected]> wrote:

Hello,

Quoted 4 lines · Robert Beckett

Since changing to git init followed by fetch to initialise git cache, when using master branch as the target version for a package, git refuses to fetch in to currently checkout out branch or master.

I think we don't want to support using branches as Git versions, as it doesn't make sense, because it won't behave as the user expects.

If you use a branch as _VERSION for a package, Buildroot will only download once, and never update to newer commits in this branch, because it will already have a cached tarball in its DL_DIR.

So I'm not sure we want to add some code to support a use case that we don't want support :-)
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering [2]https://bootlin.com

Links:
1. mailto:thomas.petazzoni at bootlin.com
2. https://bootlin.com

On 2018-06-04 15:59 +0100, Bob Beckett spake thusly:

Quoted 1 line

Is this philosophy set in stone? I can conceive of useful situations

where branches are preferable.

The problem, as Thomas quickly explained, is that it does not behave as users expect.

First, Buildroot creates a local tarball of the git reository. We do that for various reasons, one beiong to be able to prepare the compliance archive (make legal-info).

So, if you run "make clean; make" then the local archive already exists and Buildroot will not download it again, i.e. it will nt update your git tree.

Now, if you run "make distclean", then yes, it removes the download directory that you get in "buildroot/dl/", and so you will re-download again, and that will be the new branch you'll get.

However, it is possible to override the download location, if you export BR2_DL_DIR in the environment, to tell Buildroot where to look for, and store new downloads, as a way to cache downloads for those users behind slow connections (not everyone has ADSL-class connection, even less so gigabit-class fibers). If BR2_DL_DIR is set, then Buildroot will never remove it, even with "make distclean". So, you're back to square one, where the local tarball still represent an old state of your branch.

Ergo, it does not behave as you would think it would.

The second problem, is that using a branch name makes the build non-reproducible.

Today, you'll get a set of source from your branch, but maybe tomorrow it will point to something else, or even have disapeared (which is the least problematic issue, because the build would obviously fail, while it could succeed and result in a flawed state with a new branch).

Ergo, it does not behave as you would think it would.

So, using a branch name is flawed in two ways. We can't add support for using branch names in any reliable and predictable manner.

Quoted 1 line

During large projects, where the rootfs is being developed at the same

time as some custom packages,

Quoted 1 line

it is quite useful to have a branch as the target. Usually, I just do a

dirclean and delete the tar when I know

Quoted 3 lines

something has changed.

If you wanted to make it more predictable, then maybe name the tar based

on the version (branch name in this case) plus the sha,

But we can't know the sha1 before we git checkout the branch, so we don;t know if we already have it localy, so we don;t know we can use a cached version, so this undermines the very reason to have a local cache to start with...

Quoted 1 line

this way new tar packages would be created if the branch advances and

also if a tag (or other supported refspec) changes via a

Quoted 1 line

rebase etc.

Changing a tag is bad. Repeat after me: changing a tag is bad! ;-)

Yes, git allows that. Guns also allow one to shoot in ones' own foot. That does not mean this is a good idea... ;-p

Quoted 1 line

The particular thing I like to avoid is multiple version bumps in the

package spec and tags in the package source repos during

Quoted 1 line

development. When many people are working on package project, expecting

them all to take time to publish tags for every little

Quoted 3 lines

change gets unrealistic.

The question I would ask is is there any harm in supporting that if it

can be done reliably and predictably?

Quoted 1 line

Anyway, if this design is locked down, then I will just maintain this

patch locally for us to use :)

And as I hope I clearly explained above, there is no wau to make it work predictably and reliably.

Regards,
Yann E. MORIN.

Quoted 4 lines

Thanks

Bob
On Mon, Jun 4, 2018 at 3:33 PM, Thomas Petazzoni < [1]

Quoted 23 lines · thomas.petazzoni at bootlin.com> wrote:

Hello,

Quoted 4 lines · Robert Beckett

Since changing to git init followed by fetch to initialise git cache, when using master branch as the target version for a package, git refuses to fetch in to currently checkout out branch or master.

I think we don't want to support using branches as Git versions, as it doesn't make sense, because it won't behave as the user expects.

If you use a branch as _VERSION for a package, Buildroot will only download once, and never update to newer commits in this branch, because it will already have a cached tarball in its DL_DIR.

So I'm not sure we want to add some code to support a use case that we don't want support :-)
Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering [2]https://bootlin.com

Quoted 3 lines

Links:
1. mailto:thomas.petazzoni at bootlin.com
2. https://bootlin.com

--
.-----------------.--------------------.------------------.-
-------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
     |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is
no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
 conspiracy.  |

'------------------------------^-------^------------------^- -------------------'

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that active development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep updating my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive seen this discussion come up multiple times before).

wrote:

Hi Bob

Quoted 3 lines · Bob Beckett

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely

a

Quoted 7 lines

reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that

active

Quoted 4 lines

development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep

updating

Quoted 7 lines

my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Quoted 6 lines

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

Yes 'ls-remote' is actually a good option. You could have the following package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Then it would pick up any change automatically until you finish your dev and change the FOO_VERSION to a proper ID.

Quoted 4 lines

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive

seen

Quoted 1 line

this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that should be a perfectly fine option for your custom package.

Regards,
Gary

De: "Bob Beckett" <[email protected]>
Para: "Gary Bisson" <[email protected]>
Cc: "Yann E. MORIN" <[email protected]>, "Thomas Petazzoni" <[email protected]>, "buildroot" <[email protected]>
Enviadas: Segunda-feira, 4 de junho de 2018 13:54:30
Assunto: Re: [Buildroot] [PATCH 1/2] git: fix handling of git repos using master as version

Quoted 31 lines · Gary Bisson

Hi Bob

Quoted 21 lines · Bob Beckett

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that active development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep updating my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Quoted 6 lines

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db refs/remotes/upstream/master

Quoted 6 lines

Yes 'ls-remote' is actually a good option. You could have the following package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Then it would pick up any change automatically until you finish your dev and change the FOO_VERSION to a proper ID.

Thats actually a cracking idea. I like it (and I'm not sure why I didnt think of it when suggesting it for detecting outdated named head versions :) )

Quoted 8 lines

Quoted 5 lines

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive seen this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that should be a perfectly fine option for your custom package.

Quoted 2 lines

Regards,
Gary

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 18 lines

Quoted 3 lines

Regarding not knowing the sha before a checkout, would git ls-

remote not

Quoted 3 lines

suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git

master

Quoted 2 lines

407fb2fe202aaeb273e4986dc88c30596a7fe8db refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db

refs/remotes/upstream/master

Yes 'ls-remote' is actually a good option. You could have the following
package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Quoted 5 lines

Then it would pick up any change automatically until you finish your dev
and change the FOO_VERSION to a proper ID.

Thats actually a cracking idea. I like it (and I'm not sure why I didnt think of it when suggesting it for detecting outdated named head versions :) )

Hi Bob

Quoted 21 lines · Bob Beckett

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that active development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep updating my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Quoted 6 lines

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

Yes 'ls-remote' is actually a good option.

package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Then it would pick up any change automatically until you finish your dev and change the FOO_VERSION to a proper ID.

Quoted 5 lines

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive seen this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that should be a perfectly fine option for your custom package.

Regards,
Gary
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 25 lines · Gary Bisson

Quoted 24 lines · Bob Beckett

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that active development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep updating my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

Quoted 5 lines

Quoted 3 lines

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

Yes 'ls-remote' is actually a good option.

It's a bit racy though. By the time you do the actual fetch, the remote branch
may already have been updated by someone else.

You could have the following

Quoted 17 lines

package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Then it would pick up any change automatically until you finish your dev and change the FOO_VERSION to a proper ID.

Quoted 5 lines

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive seen this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that should be a perfectly fine option for your custom package.

Regards,
Gary

Hi Arnout,

Quoted 30 lines · Arnout Vandecappelle

Quoted 29 lines · Gary Bisson

Quoted 27 lines · Bob Beckett

Thanks for the more in depth explanation.

I agree with all of the reasons outlined for the purpose of being purely a reproducible build manager, which buildroot only ever aimed to be.

However, people do use it during development, and with a reasonably large number of custom packages all in development at the same time, using the _OVERRIDE_SRCDIR method to test latest versions of multiple packages becomes very unwieldy.
My solution so far has been to point the version at the branch that active development is taking place on, and removing the build directories and packages for the specific projects each time.
When I am not the one doing the development on each package's source, but am developing the rootfs, this means I dont have to manually keep updating my local git repositories for each project.
Once each package has hit their first release version, they do get tagged and the release branch for the project's BR2_EXTERNAL directory gets updated with those versions, and each version thereafter, but the development branch for the external directory persistently stays on the development branch head for each package.

Regarding not knowing the sha before a checkout, would git ls-remote not suffice for this? e.g.

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master

Quoted 4 lines

Quoted 1 line

Yes 'ls-remote' is actually a good option.

It's a bit racy though. By the time you do the actual fetch, the remote branch
may already have been updated by someone else.

Someone might update the branch while you are fetching as well. As long as you fetch a valid commit ID that is known to have been at the branch tip at some point in the recent past, it should be fine from the user perspective.

baruch

Quoted 3 lines

You could have the following

Quoted 2 lines

package version during development:
FOO_VERSION = $(shell git ls-remote URL branch | awk '{ print $$1 }')

Quoted 15 lines

Quoted 15 lines

Then it would pick up any change automatically until you finish your dev and change the FOO_VERSION to a proper ID.

Quoted 5 lines

It would allow you to check for changes in sha without doing a new fetch.

Oh well, not to worry. Ill keep it as a local patch, as I suspect I am treading old ground of "buildroot is not a development platform" (Ive seen this discussion come up multiple times before).

Yes that will most likely not be included inside BR infrastructure but that should be a perfectly fine option for your custom package.

Regards,
Gary

Quoted 5 lines

Quoted 3 lines

$ git ls-remote git at gitlab.com:adhlinux/buildroot/buildroot.git master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/heads/master
407fb2fe202aaeb273e4986dc88c30596a7fe8db    refs/remotes/upstream/master  

Yes 'ls-remote' is actually a good option.

It's a bit racy though. By the time you do the actual fetch, the remote branch
may already have been updated by someone else.

If you wanted to make it more predictable, then maybe name the tar based on the version (branch name in this case) plus the sha, this way new tar packages would be created if the branch advances and also if a tag (or other supported refspec) changes via a rebase etc.

The particular thing I like to avoid is multiple version bumps in the package spec and tags in the package source repos during development. When many people are working on package project, expecting them all to take time to publish tags for every little change gets unrealistic.

Anyway, if this design is locked down, then I will just maintain this patch locally for us to use :)

Hello,

Yann already answered on some of the aspects, but I wanted to add one more thing.

Quoted 4 lines · Bob Beckett

If you wanted to make it more predictable, then maybe name the tar based on the version (branch name in this case) plus the sha, this way new tar packages would be created if the branch advances and also if a tag (or other supported refspec) changes via a rebase etc.

But then at every build you need to go and check the Git repository to see if a new version on this branch is available. That's clearly not something we want to do for all packages.

Quoted 2 lines

The particular thing I like to avoid is multiple version bumps in the package spec and tags in the package source repos during development.

When

Quoted 1 line

many people are working on package project, expecting them all to take

time

Quoted 1 line

to publish tags for every little change gets unrealistic.

During development, we would normally expect people to use <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local, which allows a package to use locally available source code instead of fetching it as a tarball or from Github.

Quoted 1 line

Anyway, if this design is locked down, then I will just maintain this

patch

Quoted 1 line

locally for us to use :)

The design is never locked down, Buildroot is open sourced, and influenced by all the people who use it and contribute to it. However, we are obviously careful to maintain a behavior and semantic that makes the most sense. And as Yann pointed out, reproducibility is a key thing.

That being said, Angelo is proposing something like a "taint" flag that tells the user "warning, your build is not reproducible". Perhaps we could extend the Git logic so that packages can say "always fetch the latest commit from this branch", and if they do that mark the build as tainted.

But I'm still unsure when should this re-fetch should happen. When you completely rebuild from scratch ? When you do "make <pkg>-rebuild" ?

If we want to alter the Buildroot design, we need to agree on a behavior that globally makes sense.

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering
https://bootlin.com

Maybe a variable _IN_DEVELOPMENT or something could be added to a package's makefile to indicate that the version is always considered out of date, and should be re-fetched.

That way development branches for package specs could have the package spec committed with that, while release branches do not and are expected to have a non-branch version specifier.

With this strategy you would be acknowledging that you are not getting reproducibility, but you are still getting reliable builds (the 2 reasons I use buildroot).
The test for taint would then be to check if any packages have an _IN_DEVELOPMENT variable set.

Given the current state, if you specify a branch of master, you get a warning
that it doesnt appear to be a special version, but that is all, and the package
doesnt build at all as it failed to get the source in the first place> I think if there is going to be an enforcement of not using branches, then the build should probably fail in a more explicit way e.g. check to see if the
version is a branch. If it is, then fail the build with a message telling the
user that branches are not to be used as versions

(perhaps unless something is
set?). When I hit this issue it took me a bit of hunting to figure out what was going on. The current state neither does what you would expect nor anything that is usable in this specific setup.

On Mon, Jun 4, 2018 at 5:23 PM, Thomas Petazzoni <[email protected] <mailto:[email protected]>> wrote:

Hello,

Yann already answered on some of the aspects, but I wanted to add one more thing.

On Mon, 4 Jun 2018 15:59:29 +0100, Bob Beckett wrote:

> If you wanted to make it more predictable, then maybe name the tar > based on the version (branch name in this case) plus the sha, this way > new tar packages would be created if the branch advances and also if a > tag (or other supported refspec) changes via a rebase etc.

But then at every build you need to go and check the Git repository to see if a new version on this branch is available. That's clearly not something we want to do for all packages.

> The particular thing I like to avoid is multiple version bumps in the > package spec and tags in the package source repos during development. When > many people are working on package project, expecting them all to take time > to publish tags for every little change gets unrealistic.

During development, we would normally expect people to use <pkg>_OVERRIDE_SRCDIR or <pkg>_SITE_METHOD = local, which allows a package to use locally available source code instead of fetching it as a tarball or from Github.

> Anyway, if this design is locked down, then I will just maintain this patch > locally for us to use :)

The design is never locked down, Buildroot is open sourced, and influenced by all the people who use it and contribute to it. However, we are obviously careful to maintain a behavior and semantic that makes the most sense. And as Yann pointed out, reproducibility is a key thing.

That being said, Angelo is proposing something like a "taint" flag that tells the user "warning, your build is not reproducible". Perhaps we could extend the Git logic so that packages can say "always fetch the latest commit from this branch", and if they do that mark the build as tainted.

But I'm still unsure when should this re-fetch should happen. When you completely rebuild from scratch ? When you do "make <pkg>-rebuild" ?

If we want to alter the Buildroot design, we need to agree on a behavior that globally makes sense.

Thomas
--
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering
https://bootlin.com

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 3 lines · Bob Beckett

Maybe a variable _IN_DEVELOPMENT or something could be added to a package's makefile to indicate that the version is always considered out of date, and should be re-fetched.

We're never going to consider something *always* out of date, I think. We might
only consider it out-of-date when you do an explicit foo-rebuild.

Quoted 9 lines

That way development branches for package specs could have the package spec committed with that, while release branches do not and are expected to have a non-branch version specifier.

With this strategy you would be acknowledging that you are not getting reproducibility, but you are still getting reliable builds (the 2 reasons I use buildroot).
The test for taint would then be to check if any packages have an _IN_DEVELOPMENT variable set.

I'm not so fond of the _IN_DEVELOPMENT idea. What I like more is to extend
_OVERRIDE_SRCDIR to support repositories in addition to local files. It would then check out directly into the build directory, without passing through DL_DIR or a tarball.

This would probably be a rather complicated change however.

Quoted 6 lines

Given the current state, if you specify a branch of master, you get a warning
that it doesnt appear to be a special version, but that is all, and the package
doesnt build at all as it failed to get the source in the first place> I think if there is going to be an enforcement of not using branches, then the build should probably fail in a more explicit way e.g. check to see if the
version is a branch. If it is, then fail the build with a message telling the
user that branches are not to be used as versions

I completely agree with this one, we should simply fail on branches.