New: source-check broken for Github downloads

https://bugs.busybox.net/show_bug.cgi?id=9796

Bug ID: 9796
Summary: source-check broken for Github downloads
Product: buildroot
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: unassigned at buildroot.uclibc.org
Reporter: thomas.petazzoni at free-electrons.com
CC: buildroot at uclibc.org
Target Milestone: ---

source-check doesn't work for Github downloads it seems, see for example the Linux kernel download for the raspberrypi_defconfig:

$ make raspberrypi_defconfig
$ make source-check
[...]
Spider mode enabled. Check if remote file exists.
--2017-04-07 08:00:27-- 
https://github.com/raspberrypi/linux.git/linux-935c7ce84c982a26f567a03a58a1537424569938.tar.gz
Resolving github.com (github.com)... 192.30.253.112, 192.30.253.113
Connecting to github.com (github.com)|192.30.253.112|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

Spider mode enabled. Check if remote file exists.
--2017-04-07 08:00:27--
http://sources.buildroot.net/linux-935c7ce84c982a26f567a03a58a1537424569938.tar.gz
Resolving sources.buildroot.net (sources.buildroot.net)... 176.9.16.109 Connecting to sources.buildroot.net (sources.buildroot.net)|176.9.16.109|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

Some source-check fails because the kernel is not available. But in practice, if you do a build, it can download the kernel perfectly fine:

$ make linux-extract

Doing full clone
Cloning into 'linux-935c7ce84c982a26f567a03a58a1537424569938'...
remote: Counting objects: 5514757, done.
remote: Compressing objects: 100% (4789/4789), done.
remote: Total 5514757 (delta 3516), reused 11 (delta 11), pack-reused 5509948 Receiving objects: 100% (5514757/5514757), 1.55 GiB | 12.32 MiB/s, done. Resolving deltas: 100% (4579570/4579570), done.
Checking connectivity... done.
Checking out files: 100% (56709/56709), done.
warning: refname '935c7ce84c982a26f567a03a58a1537424569938' is ambiguous. Git normally never creates a ref that ends with 40 hex characters because it will be ignored when you just specify 40-hex. These refs may be created by mistake. For example,

  git checkout -b $br $(git rev-parse ...)

where "$br" is somehow empty and a 40-hex ref is created. Please examine these refs and maybe delete them. Turn this message off by running "git config advice.objectNameWarning false"

gzip -d -c /home/thomas/buildroot/dl/linux-935c7ce84c982a26f567a03a58a1537424569938.tar.gz | tar --strip-components=1 -C /home/thomas/buildroot/buildroot/output/build/linux-935c7ce84c982a26f567a03a58a1537424569938
-xf -

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

--- Comment #1 from Peter Korsgaard <[email protected]> ---
But this has nothing to do with github, right? It is simply the source-check infrastructure that gets confused and tries to use wget instead of git.

Debugging a bit further, it is because $(PKG) isn't known in the foo-source-check step, so it doesn't pick up LINUX_SITE_METHOD=git and defaults to git.

Thomas, you reworked the source-check infrastructure, so I'll let you figure out how/where to get PKG defined ;)

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|source-check broken for     |source-check broken for Git
                   |Github downloads            |downloads

--- Comment #2 from Thomas Petazzoni <[email protected]> --- Hum, I thought the defconfig was using an http download from Github, but indeed it's using a Git download instead. So indeed, it's an issue with the Git download helper.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

--- Comment #3 from Peter Korsgaard <[email protected]> ---
No, it is really a general problem with source-check (that we normally don't notice as an empty <pkg>_SITE_METHOD is handled with wget, which is correct for most packages). The git handler for source-check looks fine to me, but it never gets called.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|unassigned at buildroot.uclibc |yann.morin.1998 at free.fr
|.org |

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

Yann E. MORIN <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED

--- Comment #4 from Yann E. MORIN <[email protected]> --- The underlying reason s that foo-source-check has no stamp file associated to it, and eventually, $(PKG) is empty.

I'll fix soonish...

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

--- Comment #5 from Yann E. MORIN <[email protected]> --- Patch sent:

    https://patchwork.ozlabs.org/patch/761701/

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=9796

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #6 from Thomas Petazzoni <[email protected]> --- Fixed by
https://git.buildroot.org/buildroot/commit/?id=4c0aacbc317d298397cede593f74ece26d0a89f0.
Thanks a lot Yann!

You are receiving this mail because:
You are on the CC list for the bug.