New: Versions of packages retrieved from github.com are wrong
https://bugs.busybox.net/show_bug.cgi?id=6302
Summary: Versions of packages retrieved from github.com are
wrong
Product: buildroot
Version: unspecified
Platform: All
OS/Version: All
Status: NEW
Severity: normal
Priority: P5
Component: Outdated package
AssignedTo: unassigned at buildroot.uclibc.org
ReportedBy: uclibc-bugzilla at loowis.durge.org
CC: buildroot at uclibc.org
Estimated Hours: 0.0
I'm no buildroot or github expert (so apologies if I get some of the terminology wrong), but I think I've found a bit of a nasty problem with the way buildroot downloads specific versions of packages from github.com. At first I thought it was because the .mk files for the specific rpi-* packages I was looking at were set up wrong, but I've been investigating the problem for much of this afternoon and I think it actually affects the .mk files for many packages :-(
According to
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
packages in github with a specific tag can be downloaded with:FOO_VERSION = v1.0FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
(where here 'v1.0' is the tag in github)
And for those packages where you don't have a specific tag but instead want to download a specific revision, you can use:FOO_VERSION = 1234567FOO_SITE = http://github.com/<user>/<package>/tarball/<branch>
(where here '1234567' is the first 7 characters of the github commit hash)
In my tests, the former (downloading a specific tag) works okay, but the latter (downloading a specific revision) actually doesn't work as expected. I can't tell if this used to work and has become broken recently by a change to github.com, or if it's always been broken (but nobody ever noticed), or if I'm simply misunderstanding things (to save me looking stupid, hopefully it's not this last one!).
I'll stick with the two FOO examples above, but substituting 'fooauthor' for '<user>', 'foopackage' for '<package>' and 'master' for <branch>...
When downloading a specific tag, the first example above would download
http://github.com/fooauthor/foopackage/tarball/v1.0/foo-v1.0.tar.gz to dl/foo-v1.0.tar.gz and this is the part that works correctly. When downloading a specific revision, the second example above would download
http://github.com/fooauthor/foopackage/tarball/master/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this appears to work fine, but it actually downloads the latest revision of the master branch, and not the 1234567 revision as you might expect! So if 1234567 _is_ indeed the latest revision, then things all look okay, which may be why nobody noticed this bug before, but things go quickly wrong if there's commits after 1234567.
In my testing, it seems that _whatever_ filename you have after the .../master/ in the URL will get ignored, and you'll always get the latest commit from the master branch. However it seems that you can actually specify a revision instead of <branch> and it will then work as expected, i.e. AFAICT the 'download by revision' example above should be modified to:FOO_VERSION = 1234567FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
and then buildroot will download
http://github.com/fooauthor/foopackage/tarball/1234567/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this *will* then work as expected. Obviously you could set FOO_VERSION to 'master' and it would go back to the previous behaviour of downloading the latest commit on the master branch, but at least this behaviour would then be explicit rather than implicit.
Where the <branch> has been used (which I've replaced with $(FOO_VERSION)) I've found that you can either use a tag-name, a branch-name, or a (full or partial) commit-id, and then following another slash you can include any filename you want, which makes no difference to the file github.com serves to you.
To give a more concrete example, I found a random repository with lots of commits, tags and branches: https://github.com/libgit2/libgit2
As of 2012-06-06 these URLs all result in the identical file being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/master (wget saves to 'master')
http://github.com/libgit2/libgit2/tarball/master/ (wget saves to 'index.html')
http://github.com/libgit2/libgit2/tarball/master/libgit2-master.tar.gz (wget saves to 'libgit2-master.tar.gz')
http://github.com/libgit2/libgit2/tarball/master/libgit2-dummy-filename.tgz
[by tag-name]
http://github.com/libgit2/libgit2/tarball/v0.18.0
http://github.com/libgit2/libgit2/tarball/v0.18.0/libgit2-v0.18.0.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/a50086d
http://github.com/libgit2/libgit2/tarball/a50086d/libgit2-a50086d.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174/libgit2-a50086d174.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b/libgit2-a50086d174658914d4d6462afbc83b02825b1f5b.tar.gz
(I used 'sha1sum *' to verify all the downloaded files are identical. In each case the files inside the tarfile are inside a 'libgit2-libgit2-a50086d' subdirectory)
And these URLs result in identical files being downloaded:
[by commit-id]
http://github.com/libgit2/libgit2/tarball/36c2dfe
http://github.com/libgit2/libgit2/tarball/36c2dfe/libgit2-36c2dfe.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed69/libgit2-36c2dfed69.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed696f80a20ca1352f32ec8b136b800c30/libgit2-36c2dfed696f80a20ca1352f32ec8b136b800c30.tar.gz
And these URLs result in identical files being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/development
http://github.com/libgit2/libgit2/tarball/development/libgit2-development.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/2e1fa15
http://github.com/libgit2/libgit2/tarball/2e1fa15/libgit2-2e1fa15.tar.gz
etc. etc. you get the idea. The only (unlikely occurrence) I didn't test was a branch-name having the same name as a tag-name. Obviously the match-ups between branch-names and commit-ids shown above will go 'out of sync' once more commits have been made to those branches.
In summary, I believe
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
should be updated to say that FOO_SITE should always look like:FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
and then mention that FOO_VERSION can either specify a tag-name or a commit-id or a branch-name.
In the current git version of buildroot, by constructing a funky find command:
find package -exec grep -P '^[A-Z0-9_]+_SITE\s*=\s*https?://github\.com/.*/tarball/(?!.*\$\(.+\))' {} \; -print | grep \.mk | sort
...I believe these are the packages that need to have their _SITE setting updated in their .mk file:
jsmin, libcofi, linenoise, lua-msgpack-native, mtdev2tuio, ne10, omap-u-boot-utils, rpi-firmware, rpi-userland, socketcand, sunxi-boards, sunxi-tools, ti-utils, tslib
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
PATCH 1/14 jsmin: fix github tarball to use commit id
--- package/jsmin/jsmin.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/jsmin/jsmin.mk +1 -1
@@ -1,5 +1,5 @@ JSMIN_VERSION = a9b4755 -JSMIN_SITE = http://github.com/douglascrockford/JSMin/tarball/master +JSMIN_SITE = http://github.com/douglascrockford/JSMin/tarball/$(JSMIN_VERSION) define JSMIN_BUILD_CMDS $(TARGET_CONFIGURE_OPTS) $(MAKE) -C $(@D) jsmin
Signature · 1 line
1.7.9.5
PATCH 2/14 libcofi: fix github tarball to use commit id
--- package/libcofi/libcofi.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/libcofi/libcofi.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# LIBCOFI_VERSION = 7313fbe12b0593034d0a1b606bf33c7cf4ababce -LIBCOFI_SITE = http://github.com/simonjhall/copies-and-fills/tarball/master +LIBCOFI_SITE = http://github.com/simonjhall/copies-and-fills/tarball/$(LIBCOFI_VERSION) LIBCOFI_LICENSE = LGPLv2.1 LIBCOFI_LICENSE_FILES = README.md
Signature · 1 line
1.7.9.5
PATCH 3/14 linenoise: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/linenoise/linenoise.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/linenoise/linenoise.mk +1 -1
@@ -4,7 +4,7 @@ # ############################################################# LINENOISE_VERSION = g27a3b4d -LINENOISE_SITE = http://github.com/antirez/linenoise/tarball/master +LINENOISE_SITE = http://github.com/antirez/linenoise/tarball/$(LINENOISE_VERSION) LINENOISE_LICENSE = BSD-2c LINENOISE_INSTALL_STAGING = YES
Signature · 1 line
1.7.9.5
PATCH 4/14 lua-msgpack-native: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/lua-msgpack-native/lua-msgpack-native.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/lua-msgpack-native/lua-msgpack-native.mk +1 -1
@@ -4,7 +4,7 @@ # ############################################################# LUA_MSGPACK_NATIVE_VERSION = g41cce91 -LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/master +LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/${LUA_MSGPACK_NATIVE_VERSION) LUA_MSGPACK_NATIVE_DEPENDENCIES = lua LUA_MSGPACK_NATIVE_LICENSE = Apache-2.0 LUA_MSGPACK_NATIVE_LICENSE_FILES = LICENSE.txt
Signature · 1 line
1.7.9.5
PATCH 4/14 lua-msgpack-native: fix github tarball to use commit id
Spenser, All,
On 2013-06-06 11:47 -0500, Spenser Gilliland spake thusly:
Quoted 10 lines
Signed-off-by Spenser Gilliland <[email protected]>--- package/lua-msgpack-native/lua-msgpack-native.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/lua-msgpack-native/lua-msgpack-native.mk +0
-0
package/lua-msgpack-native/lua-msgpack-native.mk line 7
Quoted 5 lines
package/lua-msgpack-native/lua-msgpack-native.mk +1
-1@@ -4,7 +4,7 @@ # ############################################################# LUA_MSGPACK_NATIVE_VERSION = g41cce91 -LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/master +LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/${LUA_MSGPACK_NATIVE_VERSION)
$ make package/lua-msgpack-native/lua-msgpack-native.mk:8: *** unterminated variable reference. Stop. make: *** [all] Error 2
That's because you are using an opening curly-brace, but a closing parenthesis in: ${LUA_MSGPACK_NATIVE_VERSION)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH 4/14 lua-msgpack-native: fix github tarball to use commit id
Quoted 30 lines · Yann E. MORIN
Spenser, All,
On 2013-06-06 11:47 -0500, Spenser Gilliland spake thusly:
Quoted 10 lines
Signed-off-by Spenser Gilliland <[email protected]>--- package/lua-msgpack-native/lua-msgpack-native.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/lua-msgpack-native/lua-msgpack-native.mk +0
-0package/lua-msgpack-native/lua-msgpack-native.mk line 7
Quoted 5 lines
package/lua-msgpack-native/lua-msgpack-native.mk +1
-1@@ -4,7 +4,7 @@ # ############################################################# LUA_MSGPACK_NATIVE_VERSION = g41cce91 -LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/master +LUA_MSGPACK_NATIVE_SITE = http://github.com/kengonakajima/lua-msgpack-native/tarball/${LUA_MSGPACK_NATIVE_VERSION)$ make package/lua-msgpack-native/lua-msgpack-native.mk:8: *** unterminated variable reference. Stop. make: *** [all] Error 2That's because you are using an opening curly-brace, but a closing parenthesis in: ${LUA_MSGPACK_NATIVE_VERSION)
Regards,
Yann E. MORIN.
Quoted 7 lines
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
Good catch!
Spenser
--
Spenser Gilliland
Computer Engineer
Doctoral Candidate
PATCH 5/14 mtdev2tuio: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/mtdev2tuio/mtdev2tuio.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/mtdev2tuio/mtdev2tuio.mk +1 -1
@@ -4,7 +4,7 @@ # ############################################################# MTDEV2TUIO_VERSION = e1e7378 -MTDEV2TUIO_SITE = http://github.com/olivopaolo/mtdev2tuio/tarball/master +MTDEV2TUIO_SITE = http://github.com/olivopaolo/mtdev2tuio/tarball/$(MTDEV2TUIO_VERSION) MTDEV2TUIO_DEPENDENCIES = mtdev liblo MTDEV2TUIO_LICENSE = GPLv3+ MTDEV2TUIO_LICENSE_FILES = COPYING
Signature · 1 line
1.7.9.5
PATCH 6/14 ne10: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/ne10/ne10.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/ne10/ne10.mk +1 -1
@@ -7,7 +7,7 @@ # We use a Git commit ID because the last tagged version is more than # one year old. NE10_VERSION = 88c18f0 -NE10_SITE = http://github.com/projectNe10/Ne10/tarball/master +NE10_SITE = http://github.com/projectNe10/Ne10/tarball/$(NE10_VERSION) NE10_LICENSE = BSD-3c or Apache 2.0 NE10_LICENSE_FILES = doc/LICENSE
Signature · 1 line
1.7.9.5
PATCH 7/14 omap-u-boot-utils: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/omap-u-boot-utils/omap-u-boot-utils.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/omap-u-boot-utils/omap-u-boot-utils.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# OMAP_U_BOOT_UTILS_VERSION = 8aff852 -OMAP_U_BOOT_UTILS_SITE = http://github.com/nmenon/omap-u-boot-utils/tarball/master +OMAP_U_BOOT_UTILS_SITE = http://github.com/nmenon/omap-u-boot-utils/tarball/$(OMAP_U_BOOT_UTILS_VERSION) OMAP_U_BOOT_UTILS_LICENSE = GPLv2, GPLv2+ OMAP_U_BOOT_UTILS_LICENSE_FILES = COPYING
Signature · 1 line
1.7.9.5
PATCH 8/14 rpi-firmware: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/rpi-firmware/rpi-firmware.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/rpi-firmware/rpi-firmware.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# RPI_FIRMWARE_VERSION = 76d0ac38f16b6343c6155c80db1e4758b3a5838a -RPI_FIRMWARE_SITE = http://github.com/raspberrypi/firmware/tarball/master +RPI_FIRMWARE_SITE = http://github.com/raspberrypi/firmware/tarball/$(RPI_FIRMWARE_VERSION) RPI_FIRMWARE_LICENSE = BSD-3c RPI_FIRMWARE_LICENSE_FILES = boot/LICENCE.broadcom
Signature · 1 line
1.7.9.5
PATCH 9/14 rpi-userland: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/rpi-userland/rpi-userland.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/rpi-userland/rpi-userland.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# RPI_USERLAND_VERSION = 5e9a740a88a889dfc8a18bb1b00c17e5dd9d0108 -RPI_USERLAND_SITE = http://github.com/raspberrypi/userland/tarball/master +RPI_USERLAND_SITE = http://github.com/raspberrypi/userland/tarball/$(RPI_USERLAND_VERSION) RPI_USERLAND_LICENSE = BSD-3c RPI_USERLAND_LICENSE_FILES = LICENCE RPI_USERLAND_INSTALL_STAGING = YES
Signature · 1 line
1.7.9.5
PATCH 10/14 socketcand: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/socketcand/socketcand.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/socketcand/socketcand.mk +1 -1
@@ -4,7 +4,7 @@ # ############################################################# SOCKETCAND_VERSION = 7d06986 -SOCKETCAND_SITE = http://github.com/dschanoeh/socketcand/tarball/master +SOCKETCAND_SITE = http://github.com/dschanoeh/socketcand/tarball/$(SOCKETCAND_VERSION) SOCKETCAND_AUTORECONF = YES SOCKETCAND_DEPENDENCIES = libconfig
Signature · 1 line
1.7.9.5
PATCH 11/14 sunxi-boards: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/sunxi-boards/sunxi-boards.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/sunxi-boards/sunxi-boards.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# SUNXI_BOARDS_VERSION = 88d663db44f65b73ef65c4148a28c6fa3665d2b6 -SUNXI_BOARDS_SITE = https://github.com/linux-sunxi/sunxi-boards/tarball/master +SUNXI_BOARDS_SITE = https://github.com/linux-sunxi/sunxi-boards/tarball/$(SUNXI_BOARDS_VERSION) SUNXI_BOARDS_DEPENDENCIES = host-sunxi-tools SUNXI_BOARDS_INSTALL_IMAGES = YES SUNXI_BOARDS_INSTALL_TARGET = NO
Signature · 1 line
1.7.9.5
PATCH 12/14 sunxi-tools: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/sunxi-tools/sunxi-tools.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/sunxi-tools/sunxi-tools.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# SUNXI_TOOLS_VERSION = 3a94e721dd8d1e13d0b25da0a83463891e8e9ee0 -SUNXI_TOOLS_SITE = http://github.com/linux-sunxi/sunxi-tools/tarball/master +SUNXI_TOOLS_SITE = http://github.com/linux-sunxi/sunxi-tools/tarball/$(SUNXI_TOOLS_VERSION) SUNXI_TOOLS_LICENSE = GPLv2+ SUNXI_TOOLS_LICENSE_FILES = COPYING HOST_SUNXI_TOOLS_DEPENDENCIES = host-libusb
Signature · 1 line
1.7.9.5
PATCH 13/14 ti-utils: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/ti-utils/ti-utils.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/ti-utils/ti-utils.mk +1 -1
@@ -5,7 +5,7 @@ ############################################################# TI_UTILS_VERSION = 06dbdb2 -TI_UTILS_SITE = http://github.com/gxk/ti-utils/tarball/master +TI_UTILS_SITE = http://github.com/gxk/ti-utils/tarball/$(TI_UTILS_VERSION) TI_UTILS_DEPENDENCIES = libnl define TI_UTILS_BUILD_CMDS
Signature · 1 line
1.7.9.5
PATCH 14/14 tslib: fix github tarball to use commit id
In reply to Spenser Gilliland--- package/tslib/tslib.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/tslib/tslib.mk +1 -1
@@ -4,7 +4,7 @@ # ############################################################# TSLIB_VERSION = 158ee49b32f83cb7b02d5315f41c2e4cff38942d -TSLIB_SITE = http://github.com/kergoth/tslib/tarball/master +TSLIB_SITE = http://github.com/kergoth/tslib/tarball/$(TSLIB_VERSION) TSLIB_LICENSE = GPL, LGPL TSLIB_LICENSE_FILES = COPYING
Signature · 1 line
1.7.9.5
PATCH 1/14 jsmin: fix github tarball to use commit id
In reply to Spenser GillilandSpencer, all,
2013/6/6 Spenser Gilliland <[email protected]>:
Quoted 20 lines
Signed-off-by Spenser Gilliland <[email protected]>--- package/jsmin/jsmin.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/jsmin/jsmin.mk +1
-2@@ -1,5 +1,5 @@ JSMIN_VERSION = a9b4755 -JSMIN_SITE = http://github.com/douglascrockford/JSMin/tarball/master +JSMIN_SITE = http://github.com/douglascrockford/JSMin/tarball/$(JSMIN_VERSION) define JSMIN_BUILD_CMDS $(TARGET_CONFIGURE_OPTS) $(MAKE) -C $(@D) jsmin --1.7.9.5
Quoted 4 lines
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Also, it'd be nice to update the manual accordingly.
Regards,
Signature · 1 line
Samuel
PATCH 1/14 jsmin: fix github tarball to use commit id
In reply to Spenser GillilandQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Spenser" == Spenser Gilliland <[email protected]> writes:
Spenser> Signed-off-by: Spenser Gilliland <[email protected]>
Committed series (with the lua-msgpack-native issue pointed out by Yann fixed), thanks!
Signature · 1 line
Bye, Peter Korsgaard
Thanks for reporting this issue. I've created a patch series to remedy this issue.
Spenser
Quoted 149 lines · On Wed, Jun 5, 2013 at 8:40 PM, <[email protected]> wrote:
https://bugs.busybox.net/show_bug.cgi?id=6302
Summary: Versions of packages retrieved from github.com are
wrong
Product: buildroot
Version: unspecified
Platform: All
OS/Version: All
Status: NEW
Severity: normal
Priority: P5
Component: Outdated package
AssignedTo: unassigned at buildroot.uclibc.org
ReportedBy: uclibc-bugzilla at loowis.durge.org
CC: buildroot at uclibc.org
Estimated Hours: 0.0I'm no buildroot or github expert (so apologies if I get some of the terminology wrong), but I think I've found a bit of a nasty problem with the way buildroot downloads specific versions of packages from github.com. At first I thought it was because the .mk files for the specific rpi-* packages I was looking at were set up wrong, but I've been investigating the problem for much of this afternoon and I think it actually affects the .mk files for many packages :-(
According to
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
packages in github with a specific tag can be downloaded with:FOO_VERSION= v1.0FOO_SITE= http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
(where here 'v1.0' is the tag in github)And for those packages where you don't have a specific tag but instead want to download a specific revision, you can use:
FOO_VERSION= 1234567FOO_SITE= http://github.com/<user>/<package>/tarball/<branch>
(where here '1234567' is the first 7 characters of the github commit hash)In my tests, the former (downloading a specific tag) works okay, but the latter (downloading a specific revision) actually doesn't work as expected. I can't tell if this used to work and has become broken recently by a change to github.com, or if it's always been broken (but nobody ever noticed), or if I'm simply misunderstanding things (to save me looking stupid, hopefully it's not this last one!).
I'll stick with the two FOO examples above, but substituting 'fooauthor' for '<user>', 'foopackage' for '<package>' and 'master' for <branch>...
When downloading a specific tag, the first example above would download
http://github.com/fooauthor/foopackage/tarball/v1.0/foo-v1.0.tar.gz to dl/foo-v1.0.tar.gz and this is the part that works correctly. When downloading a specific revision, the second example above would download
http://github.com/fooauthor/foopackage/tarball/master/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this appears to work fine, but it actually downloads the latest revision of the master branch, and not the 1234567 revision as you might expect! So if 1234567 _is_ indeed the latest revision, then things all look okay, which may be why nobody noticed this bug before, but things go quickly wrong if there's commits after 1234567.In my testing, it seems that _whatever_ filename you have after the .../master/ in the URL will get ignored, and you'll always get the latest commit from the master branch. However it seems that you can actually specify a revision instead of <branch> and it will then work as expected, i.e. AFAICT the 'download by revision' example above should be modified to:
FOO_VERSION= 1234567FOO_SITE= http://github.com/<user>/<package>/tarball/$(FOO_VERSION)and then buildroot will download
http://github.com/fooauthor/foopackage/tarball/1234567/foo-1234567.tar.gz to dl/foo-1234567.tar.gz and this *will* then work as expected. Obviously you could setFOO_VERSIONto 'master' and it would go back to the previous behaviour of downloading the latest commit on the master branch, but at least this behaviour would then be explicit rather than implicit.Where the <branch> has been used (which I've replaced with $(FOO_VERSION)) I've found that you can either use a tag-name, a branch-name, or a (full or partial) commit-id, and then following another slash you can include any filename you want, which makes no difference to the file github.com serves to you.
To give a more concrete example, I found a random repository with lots of commits, tags and branches: https://github.com/libgit2/libgit2
As of 2012-06-06 these URLs all result in the identical file being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/master (wget saves to 'master')
http://github.com/libgit2/libgit2/tarball/master/ (wget saves to 'index.html')
http://github.com/libgit2/libgit2/tarball/master/libgit2-master.tar.gz (wget saves to 'libgit2-master.tar.gz')
http://github.com/libgit2/libgit2/tarball/master/libgit2-dummy-filename.tgz
[by tag-name]
http://github.com/libgit2/libgit2/tarball/v0.18.0
http://github.com/libgit2/libgit2/tarball/v0.18.0/libgit2-v0.18.0.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/a50086d
http://github.com/libgit2/libgit2/tarball/a50086d/libgit2-a50086d.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174/libgit2-a50086d174.tar.gz
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b
http://github.com/libgit2/libgit2/tarball/a50086d174658914d4d6462afbc83b02825b1f5b/libgit2-a50086d174658914d4d6462afbc83b02825b1f5b.tar.gz
(I used 'sha1sum *' to verify all the downloaded files are identical. In each case the files inside the tarfile are inside a 'libgit2-libgit2-a50086d' subdirectory)And these URLs result in identical files being downloaded:
[by commit-id]
http://github.com/libgit2/libgit2/tarball/36c2dfe
http://github.com/libgit2/libgit2/tarball/36c2dfe/libgit2-36c2dfe.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed69/libgit2-36c2dfed69.tar.gz
http://github.com/libgit2/libgit2/tarball/36c2dfed696f80a20ca1352f32ec8b136b800c30/libgit2-36c2dfed696f80a20ca1352f32ec8b136b800c30.tar.gzAnd these URLs result in identical files being downloaded:
[by branch-name]
http://github.com/libgit2/libgit2/tarball/development
http://github.com/libgit2/libgit2/tarball/development/libgit2-development.tar.gz
[by commit-id]
http://github.com/libgit2/libgit2/tarball/2e1fa15
http://github.com/libgit2/libgit2/tarball/2e1fa15/libgit2-2e1fa15.tar.gzetc. etc. you get the idea. The only (unlikely occurrence) I didn't test was a branch-name having the same name as a tag-name. Obviously the match-ups between branch-names and commit-ids shown above will go 'out of sync' once more commits have been made to those branches.
In summary, I believe
http://www.buildroot.org/downloads/manual/manual.html#github-download-url
should be updated to say thatFOO_SITEshould always look like:FOO_SITE= http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
and then mention thatFOO_VERSIONcan either specify a tag-name or a commit-id or a branch-name.In the current git version of buildroot, by constructing a funky find command:
find package -exec grep -P '^[A-Z0-9_]+_SITE\s*=\s*https?://github\.com/.*/tarball/(?!.*\$\(.+\))' {} \; -print | grep \.mk | sort...I believe these are the packages that need to have their
_SITEsetting updated in their .mk file:
jsmin, libcofi, linenoise, lua-msgpack-native, mtdev2tuio, ne10, omap-u-boot-utils, rpi-firmware, rpi-userland, socketcand, sunxi-boards, sunxi-tools, ti-utils, tslib--
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 3 lines
Spenser Gilliland Computer Engineer Doctoral Candidate
[Buildroot] [Bug 6302] Versions of packages retrieved from github.com are wrong
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=6302
Peter Korsgaard <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |FIXED--- Comment #1 from Peter Korsgaard <[email protected]> 2013-06-06 21:12:38 UTC --- Fixed in git by Spenser. Thanks both!
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
[Buildroot] [Bug 6302] Versions of packages retrieved from github.com are wrong
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=6302
Andrew Scheller <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |REOPENED
Resolution|FIXED |--- Comment #2 from Andrew Scheller <[email protected]> 2013-06-06 23:42:30 UTC --- Great!
But
http://git.buildroot.net/buildroot/tree/docs/manual/adding-packages-tips.txt
still needs to be updated (and
http://www.buildroot.org/downloads/manual/manual.html regenerated as necessary)
to prevent other people making the same mistake in future...
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
PATCH 1/1 manual: correct error in github tarball information
Updates to manual for bug 6302
--- docs/manual/adding-packages-tips.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
docs/manual/adding-packages-tips.txt +1 -1
@@ -58,7 +58,7 @@ enough): ------------------------ FOO_VERSION = 1234567 -FOO_SITE = http://github.com/<user>/<package>/tarball/<branch> +FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION) ------------------------ Note that the name of the tarball is the default +foo-1234567.tar.gz+
Signature · 1 line
1.7.9.5
PATCH 1/1 manual: correct error in github tarball information
Hi Spenser,
2013/6/7 Spenser Gilliland <[email protected]>:
Quoted 12 lines
Updates to manual for bug 6302
Signed-off-by Spenser Gilliland <[email protected]>--- docs/manual/adding-packages-tips.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)docs/manual/adding-packages-tips.txt +0
-0
docs/manual/adding-packages-tips.txt enough): · line 62
Quoted 6 lines
docs/manual/adding-packages-tips.txt +1
-1@@ -58,7 +58,7 @@ enough): ------------------------ FOO_VERSION = 1234567 -FOO_SITE = http://github.com/<user>/<package>/tarball/<branch> +FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION) ------------------------
I'd have slightly reword the whole section, since there is no difference between FOO_VERSION being a tag or a commit id, something like this:
---
------------------------FOO_VERSION = v1.0 # tag or commit IDFOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
------------------------
.Notes
- The FOO_VERSION can either be a tag or a commit ID.
- The tarball name generated by github matches the default one from Buildroot (e.g.: +foo-1234567.tar.gz+),
so it is not necessary to specify it in the +.mk+ file.
- When using a commit ID as version, usually the first 7 characters of the SHA1 are enough.
---
Regards,
Signature · 1 line
Samuel
PATCH 1/1 manual: correct error in github tarball information
Quoted 7 lines
docs/manual/adding-packages-tips.txt enough):
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Samuel" == Samuel Martin <[email protected]> writes:
------------------------
FOO_VERSION= 1234567docs/manual/adding-packages-tips.txt +1
-2-FOO_SITE = http://github.com/<user>/<package>/tarball/<branch> +FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION) ------------------------
Samuel> I'd have slightly reword the whole section, since there is no Samuel> difference between FOO_VERSION being a tag or a commit id, Samuel> something like this:
Samuel> ---
Samuel> ------------------------
Samuel> FOO_VERSION = v1.0 # tag or commit ID
Samuel> FOO_SITE = http://github.com/<user>/<package>/tarball/$(FOO_VERSION)
Samuel> ------------------------
Samuel> .Notes
Samuel> - The FOO_VERSION can either be a tag or a commit ID. Samuel> - The tarball name generated by github matches the default one from Samuel> Buildroot (e.g.: +foo-1234567.tar.gz+),
Samuel> so it is not necessary to specify it in the +.mk+ file. Samuel> - When using a commit ID as version, usually the first 7 characters of Samuel> the SHA1 are enough.
Agreed. I've fixed this in git now, thanks.
Signature · 1 line
Bye, Peter Korsgaard
PATCH 1/1 manual: correct error in github tarball information
In reply to Spenser GillilandQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Spenser" == Spenser Gilliland <[email protected]> writes:
Spenser> Updates to manual for bug 6302
Spenser> Signed-off-by: Spenser Gilliland <[email protected]>
Committed, thanks.
Signature · 1 line
Bye, Peter Korsgaard
[Buildroot] [Bug 6302] Versions of packages retrieved from github.com are wrong
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=6302
Peter Korsgaard <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|REOPENED |RESOLVED
Resolution| |FIXED--- Comment #3 from Peter Korsgaard <[email protected]> 2013-06-07 08:52:44 UTC --- (In reply to comment #2)
Quoted 6 lines
Great!
But
http://git.buildroot.net/buildroot/tree/docs/manual/adding-packages-tips.txt
still needs to be updated (and
http://www.buildroot.org/downloads/manual/manual.html regenerated as necessary) to prevent other people making the same mistake in future...
Manual updated in git and website updated, thanks.
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
[Buildroot] [Bug 6302] Versions of packages retrieved from github.com are wrong
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=6302
--- Comment #4 from Andrew Scheller <[email protected]> 2013-06-07 10:56:41 UTC --- Fantastic, thanks to all involved (and so quick too!).
Just for my future reference, this is the related mailing-list thread
http://thread.gmane.org/gmane.comp.lib.uclibc.buildroot/60950 and these are the git commit ids that were involved in fixing this bug:
f0f124f36a8616a1e0895b2eb9411a8459daa3c9 (jsmin) 9bb2757c461271f4458294d417f440f839513e33 (libcofi) b89e8141e3247e28387e14a2959d3ec8ea11f12c (linenoise) 55c90b10b3c73ff0708ceb73645c51658f4821aa (lua-msgpack-native) e9c9bd2d62a1d09f2f54b55f05301be0e06981df (mtdev2tuio) c2b205c038bc0dba3960917f2eb1bd30fbc24b03 (ne10) ae5be552b6435437db42b168b61aa15b8bed2fbc (omap-u-boot-utils) 6b16001d4f026fd08f14a4b2043710531b50a0ad (rpi-firmware) 8645437c0cc0fffa6962030ddd90698ab917891e (rpi-userland) fe64434df6c8a5f3516f92caf9c04d6dedda444b (socketcand) 8ad0ab6e16e67a1d1269aec7ddc1b70d10d86e82 (sunxi-boards) 63c39b52ce3cfacf12071e8d76a5a30c57fe54d6 (sunxi-tools) 7869f83bda815279f8245913d33062e525d4efe1 (ti-utils) 013365238f0b00f01dcc173edf2f3b6e37f7beb1 (tslib) 484b6dd8a3abccf0be34d1bed3761a8809ba23be (manual) 86bfe7cca9bca292f00f7a41221ae5879f4a5769 (manual)
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
[Buildroot] [Bug 6302] Versions of packages retrieved from github.com are wrong
In reply to bugzilla at busybox.nethttps://bugs.busybox.net/show_bug.cgi?id=6302
--- Comment #5 from Andrew Scheller <[email protected]> 2013-06-10 03:44:56 UTC --- Fixing this bug also involved commit ids:41190ef0f1ffb68675bfe77205e9207363546377 (linenoise) 0b8ed5b53d066f86f09e01d02a5e0bd715c17b70 (lua-msgpack-native)
because they were using invalid git hashes (which this bug had been masking).
Signature · 3 lines
Configure bugmail: https://bugs.busybox.net/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.