New DL_DIR organisation; git cache feature

This feature is not used by anyone in the core developpers and makes a drastic simplification of the pkg-download infrastructure harder.

The future patch will move much of what's in the current pkg-download.mk file into the dl-wrapper which is a shell script.

Signed-off-by Maxime Hadjinlian <[email protected]>

From: "Yann E. MORIN" <[email protected]>

support/download/check-hash +0 -0

     case "${OPT}" in
     q)  exec >/dev/null;;
     \?) exit 1;;

support/download/dl-wrapper +2 -0

+            -o "${tmpf}" "${@}"
+    then
         rm -rf "${tmpd}"
         exit 1
     fi

Quoted 9 lines

From: "Yann E. MORIN" <[email protected]>

support/download/check-hash +0 -0

index c1ff53c02b..bf2dbb1a8b 100755
@@ -19,7 +19,7 @@ set -e
 #   3:  the hash file exists and there was no hash to check the file against
 #   4:  the hash file exists and at least one hash type is unknown
 

support/download/check-hash set -e · line 22

Quoted 2 lines

support/download/check-hash +1 -1

-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do

This doesn't make sense to me: the global DL_GETOPTS are never passed to
check-hash, and you're not using any of them. So I'd keep the explicit :q here.

Quoted 3 lines

support/download/check-hash +0 -0

     case "${OPT}" in
     q)  exec >/dev/null;;
     \?) exit 1;;

[snip]

support/download/dl-wrapper line 22

Quoted 9 lines

support/download/dl-wrapper +1 -0

index f944b71db5..a29411e0ae 100755
@@ -19,6 +19,8 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
+export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"

Since this variable is defined here, it makes sense to also document the
options here. In fact, to make it makes more sense to document them all here rather than in each individual backend. Of course, no need to go and remove it in each backend, but I would like to have then documented here. Especially since a later patch will also parse them here.

support/download/dl-wrapper main() { · line 89

Quoted 11 lines

support/download/dl-wrapper +3 -1

+
 main() {
     local OPT OPTARG
     local backend output hfile recurse quiet

@@ -83,7 +85,10 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" \
+            ${quiet} ${recurse} \

I don't like it very much when options are mixed like this: either fill lines
up completely, or do one option (or option combination, of course, like -o "...") per line. But since this will be changed again later it's not worth changing this patch.

Regards,
Arnout

Quoted 5 lines

support/download/dl-wrapper +2 -0

+            -o "${tmpf}" "${@}"
+    then
         rm -rf "${tmpd}"
         exit 1
     fi

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Arnout, All,

On 2017-07-05 01:09 +0200, Arnout Vandecappelle spake thusly:

Quoted 10 lines · Maxime Hadjinlian

Quoted 9 lines

From: "Yann E. MORIN" <[email protected]>

support/download/check-hash +0 -0

index c1ff53c02b..bf2dbb1a8b 100755
@@ -19,7 +19,7 @@ set -e
 #   3:  the hash file exists and there was no hash to check the file against
 #   4:  the hash file exists and at least one hash type is unknown
 

Quoted 5 lines

support/download/check-hash set -e · line 22

Quoted 2 lines

support/download/check-hash +1 -1

-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do

This doesn't make sense to me: the global DL_GETOPTS are never passed to
check-hash, and you're not using any of them. So I'd keep the explicit :q here.

It took me a bit of time to understand what you meant, because BR_BACKEND_DL_GETOPTS is actually exported by dl-wrapper, and it is dl-wrapper that calls check-hash. So let me rephrase it, and you'll tell me if I understood correctly:

This doesn't make sense to me: none of the variabels from BR_BACKEND_DL_GETOPTS are ever passed to check-hash, so [...]

Right? If so I agree.

Regards,
Yann E. MORIN.

Quoted 56 lines

Quoted 3 lines

support/download/check-hash +0 -0

     case "${OPT}" in
     q)  exec >/dev/null;;
     \?) exit 1;;

[snip]

support/download/dl-wrapper line 22

Quoted 9 lines

support/download/dl-wrapper +1 -0

index f944b71db5..a29411e0ae 100755
@@ -19,6 +19,8 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
+export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"

Since this variable is defined here, it makes sense to also document the
options here. In fact, to make it makes more sense to document them all here rather than in each individual backend. Of course, no need to go and remove it in each backend, but I would like to have then documented here. Especially since a later patch will also parse them here.

support/download/dl-wrapper main() { · line 89

Quoted 11 lines

support/download/dl-wrapper +3 -1

+
 main() {
     local OPT OPTARG
     local backend output hfile recurse quiet

@@ -83,7 +85,10 @@ main() {
     # If the backend fails, we can just remove the temporary directory to
     # remove all the cruft it may have left behind. Then we just exit in
     # error too.
-    if ! "${OLDPWD}/support/download/${backend}" ${quiet} ${recurse} "${tmpf}" "${@}"; then
+    if ! "${OLDPWD}/support/download/${backend}" \
+            ${quiet} ${recurse} \

I don't like it very much when options are mixed like this: either fill lines
up completely, or do one option (or option combination, of course, like -o "...") per line. But since this will be changed again later it's not worth changing this patch.

Regards,
Arnout

Quoted 5 lines

support/download/dl-wrapper +2 -0

+            -o "${tmpf}" "${@}"
+    then
         rm -rf "${tmpd}"
         exit 1
     fi

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Quoted 24 lines

Arnout, All,

On 2017-07-05 01:09 +0200, Arnout Vandecappelle spake thusly:

Quoted 10 lines · Maxime Hadjinlian

Quoted 9 lines

From: "Yann E. MORIN" <[email protected]>

support/download/check-hash +0 -0

index c1ff53c02b..bf2dbb1a8b 100755
@@ -19,7 +19,7 @@ set -e
 #   3:  the hash file exists and there was no hash to check the file against
 #   4:  the hash file exists and at least one hash type is unknown
 

Quoted 5 lines

support/download/check-hash set -e · line 22

Quoted 2 lines

support/download/check-hash +1 -1

-while getopts :q OPT; do
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do

This doesn't make sense to me: the global DL_GETOPTS are never passed to
check-hash, and you're not using any of them. So I'd keep the explicit :q here.

It took me a bit of time to understand what you meant, because BR_BACKEND_DL_GETOPTS is actually exported by dl-wrapper, and it is dl-wrapper that calls check-hash. So let me rephrase it, and you'll tell me if I understood correctly:

Quoted 2 lines

This doesn't make sense to me: none of the variabels from BR_BACKEND_DL_GETOPTS are ever passed to check-hash, so [...]

Yes, that's what I meant. I guess I don't formulate very clearly at 01:09 (much
less at 02:38...)

The goal here is to simplify the infrastructure by putting most of the code in the dl-wrapper as it's easier to implement and to read.

Most of the function where common already, this patch finalize it by making the pkg-download.mk pass all the parameters needed to the dl-wrapper which in turns will pass everything to every backend.

The backend will then cherry-pick what it needs from theses arguments and act accordingly.

It eases the transition to the addition of a sub directory per package in the DL_DIR, and later on, a git cache.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-download.mk     | 166 ++++++++------------------------------------

 support/download/bzr        |   2 +-
 support/download/cp         |   2 +-
 support/download/cvs        |   2 +-
 support/download/dl-wrapper |  65 ++++++++++-------
 support/download/wget       |   7 +-
 6 files changed, 77 insertions(+), 167 deletions(-)

package/pkg-download.mk +1 -4

+endif
 
-define DOWNLOAD_CVS
-	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(QUIET) \

package/pkg-download.mk +4 -0

+ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
+DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
+endif

package/pkg-download.mk +1 -0

+	$(EXTRA_ENV) $(DL_WRAPPER) \
 		-c $($(PKG)_DL_VERSION) \

support/download/cp +0 -0

     :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
     esac

support/download/cvs +1 -1

index 69d5c71f28..9caffb4b82 100755
@@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-Q;;
     o)  output="${OPTARG}";;
-    u)  uri="${OPTARG}";;
+    u)  uri="${OPTARG#*//}";;
     c)  rev="${OPTARG}";;
     N)  rawname="${OPTARG}";;
     n)  basename="${OPTARG}";;

support/download/dl-wrapper +0 -0

index a29411e0ae..50c14a2e16 100755

support/download/dl-wrapper +8 -11

 main() {
     local OPT OPTARG
     local backend output hfile recurse quiet
+    local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H:rq OPT; do
+    while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
-        b)  backend="${OPTARG}";;
+        c)  cset="${OPTARG}";;
         o)  output="${OPTARG}";;
+        n)  raw_base_name="${OPTARG}";;
+        N)  base_name="${OPTARG}";;
         H)  hfile="${OPTARG}";;
         r)  recurse="-r";;
+        f)  filename="${OPTARG}";;
+        u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
     done
+
     # Forget our options, and keep only those for the backend
     shift $((OPTIND-1))
 
-    if [ -z "${backend}" ]; then
-        error "no backend specified, use -b\n"
-    fi
     if [ -z "${output}" ]; then
         error "no output specified, use -o\n"
     fi

@@ -82,16 +85,29 @@ main() {
     # Doing the 'cd' here rather than in all backends is easier.
     cd "${tmpd}"
 
-    # If the backend fails, we can just remove the temporary directory to
-    # remove all the cruft it may have left behind. Then we just exit in
-    # error too.
-    if ! "${OLDPWD}/support/download/${backend}" \
-            ${quiet} ${recurse} \
-            -o "${tmpf}" "${@}"

support/download/dl-wrapper +8 -10

+    done
 
     # cd back to free the temp-dir, so we can remove it later
     cd "${OLDPWD}"

@@ -164,16 +180,13 @@ DESCRIPTION
 
     -h  This help text.
 
-    -b BACKEND
-        Wrap the specified BACKEND. Known backends are:
-            bzr     Bazaar
-            cp      Local files
-            cvs     Concurrent Versions System
-            git     Git
-            hg      Mercurial
-            scp     Secure copy
-            svn     Subversion
-            wget    HTTP download
+    -u URIs
+        The URI to get the file from, the URI must respect the format given in
+        the example.
+        You may give as many '-u URI' as you want, the script will stop at the
+        frist successful download.
+
+        Example: backend+URI; git+http://example.com or http+http://example.com
 
     -o FILE
         Store the downloaded archive in FILE.

support/download/wget +2 -0

index fece6663ca..3e6a6b446c 100755
@@ -8,6 +8,7 @@ set -e
 # Options:
 #   -q          Be quiet.
 #   -o FILE     Save into file FILE.
+#   -f FILENAME The filename of the tarball to get at URL
 #   -u URL      Download file at URL.
 #
 # Environment:

@@ -18,6 +19,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
     o)  output="${OPTARG}";;
+    f)  filename="${OPTARG}";;
     u)  url="${OPTARG}";;
     :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;

support/download/wget +2 -0

+
+_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"

This variable is currently not used by anyone.
The value is changed to match the path to DL_DIR.

The next patch will introduce the use of PKG_DL_DIR for packages that use specific EXTRACT_CMDS.

And then we will be able to change the value of PKG_DL_DIR to create a new directory structure in DL_DIR with a subdirectory for each package.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-generic.mk +1 -1

index 79bea906e2..c1cd41fda5 100644
@@ -434,7 +434,7 @@ endif
 
 $(2)_BASE_NAME	= $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
 $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR	=  $$(DL_DIR)/$$($(2)_BASE_NAME)
+$(2)_DL_DIR	=  $$(DL_DIR)
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
 ifndef $(2)_SUBDIR

This variable is currently not used by anyone.
The value is changed to match the path to DL_DIR.

The next patch will introduce the use of PKG_DL_DIR for packages that use specific EXTRACT_CMDS.

And then we will be able to change the value of PKG_DL_DIR to create a new directory structure in DL_DIR with a subdirectory for each package.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the transition into a new directory structure for DL_DIR.

This commit has been generated with the following scripts:

for i in package/*/*.mk; do

	if ! grep -q "\$(DL_DIR)" ${i}; then
		continue
	fi
	pkg_name="$(basename $(dirname ${i}))"
	raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
	pkg_dl_dir="${raw_pkg_name}_DL_DIR"
	sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}

done

Signed-off-by Maxime Hadjinlian <[email protected]>

Hello,

Quoted 13 lines · Maxime Hadjinlian

Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the transition into a new directory structure for DL_DIR.

This commit has been generated with the following scripts:

for i in package/*/*.mk; do

	if ! grep -q "\$(DL_DIR)" ${i}; then
		continue
	fi
	pkg_name="$(basename $(dirname ${i}))"
	raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
	pkg_dl_dir="${raw_pkg_name}_DL_DIR"
	sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}

Quoted 3 lines

done

Signed-off-by Maxime Hadjinlian <[email protected]>

I wanted to apply this one, but I believe there is a problem, see below:

Quoted 7 lines

package/gcc/gcc.mk +0 -0

index b52f9456b3..64fcfc70e7 100644
@@ -289,7 +289,7 @@ HOST_GCC_COMMON_MAKE_OPTS = \
 	gcc_cv_libc_provides_ssp=$(if $(BR2_TOOLCHAIN_HAS_SSP),yes,no)
 

package/gcc/gcc.mk HOST_GCC_COMMON_MAKE_OPTS = \ · line 292

Quoted 3 lines

package/gcc/gcc.mk +1 -1

 ifeq ($(BR2_CCACHE),y)
-HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE)
+HOST_GCC_COMMON_CCACHE_HASH_FILES += $(GCC_DL_DIR)/$(GCC_SOURCE)

I'm not sure this one works. Indeed, there is no package named just "gcc". There are two packages named gcc-initial and gcc-final, but no package named "gcc", so GCC_DL_DIR doesn't exist. Only GCC_INITIAL_DL_DIR and GCC_FINAL_DL_DIR exist.

Am I missing something here ?

Thanks!

Thomas

Signature · 7 lines

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Instead of DL_DIR, the package should now use $(PKG)_DL_DIR to ease the transition into a new directory structure for DL_DIR.

This commit has been generated with the following scripts:

for i in package/*/*.mk; do

	if ! grep -q "\$(DL_DIR)" ${i}; then
		continue
	fi
	pkg_name="$(basename $(dirname ${i}))"
	raw_pkg_name=$(echo ${pkg_name} | tr [a-z] [A-Z] | tr '-' '_')
	pkg_dl_dir="${raw_pkg_name}_DL_DIR"
	sed -i "s/\$(DL_DIR)/\$($pkg_dl_dir)/" ${i}

done

Signed-off-by Maxime Hadjinlian <[email protected]>

 48 files changed, 52 insertions(+), 52 deletions(-)

With all the previous changes, we are now ready to add a subdirectory to the DL_DIR.
The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2}

This is needed for multiple reasons:
- Avoid patches with name like SHA1.patch laying flat in DL_DIR,

which makes it hard to know to which packages they apply
- Avoid possible collisions if two releases have the same name (e.g: v01.tar)
- Allow the possibility to handle a git cache per package in the newly created subdirectory.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

package/pkg-generic.mk +1 -1

index 5d169b20d6..7364551471 100644
@@ -434,7 +434,7 @@ endif
 
 $(2)_BASE_NAME	= $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
 $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR	=  $$(DL_DIR)
+$(2)_DL_DIR 	=  $$(DL_DIR)/$$($(2)_RAWNAME)
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
 ifndef $(2)_SUBDIR

Since we introduced subdirectory to the DL_DIR, we need to support them

in the PRIMARY and BACKUP mirror as they evolve to the new tree structure.

We check first the new URI (with the subdir), and in case of failure, we go check without.
By checking both URIs, we ensure that old mirror are usable.

Signed-off-by Maxime Hadjinlian <[email protected]> ---

 package/pkg-download.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-download.mk +0 -0

index 9a455dc79b..47671e21a7 100644

package/pkg-download.mk +1 -0

 	-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
 endif
 

@@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
 	-u $($(PKG)_SITE_METHOD)+$(dir $(1))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))/$(notdir $($(PKG)_DL_DIR)) \
 	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
 endif
 endif

Quoted 1 line · Maxime Hadjinlian

Since we introduced subdirectory to the DL_DIR, we need to support them

subdirectories

Quoted 6 lines

in the PRIMARY and BACKUP mirror as they evolve to the new tree structure.

We check first the new URI (with the subdir), and in case of failure, we go check without.
By checking both URIs, we ensure that old mirror are usable.

mirrors

Quoted 10 lines

Signed-off-by Maxime Hadjinlian <[email protected]> ---

 package/pkg-download.mk | 2 ++
 1 file changed, 2 insertions(+)

package/pkg-download.mk +0 -0

index 9a455dc79b..47671e21a7 100644

package/pkg-download.mk export BR_NO_CHECK_HASH_FOR = · line 78

Quoted 4 lines

package/pkg-download.mk +1 -0

@@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR =
 
 ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
 DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR)) \  

I was thinking, it is a bit silly to first calculate that path and then
applying notdir to it, but I guess this is more robust to a change of PKG_DL_DIR.

There is just one thing to check: I think that this DOWNLOAD_URIS may also be
used outside the context of a package, in which case $(PKG) is not defined. It's not terrible because then we just go to PRIMARY_SITE without subdir, but it's a pity to do things twice. So maybe better

	$(if $(PKG),$(call getschemeplusuri,$(BR2_PRIMARY_SITE))/$(notdir $($(PKG)_DL_DIR))) \

Oh yeah, completely easy to read :-)

Regards,
Arnout

Quoted 12 lines

package/pkg-download.mk +1 -0

 	-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
 endif
 

@@ -83,6 +84,7 @@ DOWNLOAD_URIS += \
 	-u $($(PKG)_SITE_METHOD)+$(dir $(1))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
 DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))/$(notdir $($(PKG)_DL_DIR)) \
 	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
 endif
 endif

This per package variable can be used to specify that a package shares the same sources as another package.

The use case here is for example, you have linux-headers and linux, you don't want to download twice the kernel (because of the introduction of the subdirs in DL_DIR).

So you want to specify in linux-headers, this variable, so it will use the linux's source.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

package/pkg-generic.mk +3 -1

index 7364551471..a8e0686068 100644
@@ -434,7 +434,9 @@ endif
 
 $(2)_BASE_NAME	= $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
 $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR 	=  $$(DL_DIR)/$$($(2)_RAWNAME)
+$(2)_DL_DIR 	=  $$(if $$($(2)_SAME_SOURCE_AS), \
+	$$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
+	$$(DL_DIR)/$$($(2)_RAWNAME))
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
 ifndef $(2)_SUBDIR

Quoted 9 lines

This per package variable can be used to specify that a package shares the same sources as another package.

The use case here is for example, you have linux-headers and linux, you don't want to download twice the kernel (because of the introduction of the subdirs in DL_DIR).

So you want to specify in linux-headers, this variable, so it will use the linux's source.

That patch at least should be part of this series. We *really* don't want to
download linux twice.

However, Yann is reviving the package out-of-tree build series, which IIRC will
already do something similar but then reusing the source directory rather than the source tarball. Therefore, this patch may no longer be needed. Well, on condition that that series gets applied :-)

Regards,
Arnout

Quoted 22 lines

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

package/pkg-generic.mk +3 -1

index 7364551471..a8e0686068 100644
@@ -434,7 +434,9 @@ endif
 
 $(2)_BASE_NAME	= $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1))
 $(2)_RAW_BASE_NAME = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME))
-$(2)_DL_DIR 	=  $$(DL_DIR)/$$($(2)_RAWNAME)
+$(2)_DL_DIR 	=  $$(if $$($(2)_SAME_SOURCE_AS), \
+	$$(DL_DIR)/$$($$(call UPPERCASE,$$($(2)_SAME_SOURCE_AS))_RAWNAME), \
+	$$(DL_DIR)/$$($(2)_RAWNAME))
 $(2)_DIR	=  $$(BUILD_DIR)/$$($(2)_BASE_NAME)
 
 ifndef $(2)_SUBDIR

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 Config.in                               | 3 +++
 docs/manual/adding-packages-generic.txt | 6 +++---
 2 files changed, 6 insertions(+), 3 deletions(-)

Config.in +3 -0

index 72ceadfde6..0ad43679f3 100644
@@ -195,6 +195,9 @@ config BR2_DL_DIR
 	  If the Linux shell environment has defined the BR2_DL_DIR
 	  environment variable, then this overrides this configuration
 	  item.
+	  The directory is organized with a subdirectory for each package.
+	  Each package has its own $(LIBFOO_DL_DIR) variable that can be used
+	  to find the correct path.
 
 	  The default is $(TOPDIR)/dl
 

docs/manual/adding-packages-generic.txt +1 -1

index 12cf1605d6..6e3b503f6b 100644
@@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) :
   the file using this URL. Otherwise, Buildroot will assume the file
   to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do
   anything with those additional files, except download them: it will
-  be up to the package recipe to use them from +$(DL_DIR)+.
+  be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+.
 
 * +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the
   package source code. In many cases, Buildroot guesses the method

docs/manual/adding-packages-generic.txt +0 -0

 * +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target
   cross-compilation utilities
 

In order to introduce the cache mechanisms, we need to have a lock on the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel download (a shared DL_DIR for two buildroot instances).

To make sure the directory exists, the mkdir call has been removed from the dl-wrapper and put in the infrastructure.

Signed-off-by Maxime Hadjinlian <[email protected]> ---

 package/pkg-download.mk     | 4 +++-
 support/download/dl-wrapper | 1 -
 2 files changed, 3 insertions(+), 2 deletions(-)

package/pkg-download.mk +0 -0

index 47671e21a7..c67f1ba4e1 100644

package/pkg-download.mk +2 -1

 # DL_DIR may have been set already from the environment
 ifeq ($(origin DL_DIR),undefined)

@@ -91,7 +92,8 @@ endif
 
 define DOWNLOAD
 	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
-	$(EXTRA_ENV) $(DL_WRAPPER) \
+	$(Q)mkdir -p $($(PKG)_DL_DIR)/
+	$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \
 		-c $($(PKG)_DL_VERSION) \
 		-f $(notdir $(1)) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \

support/download/dl-wrapper +0 -1

index f14a60c173..50c14a2e16 100755
@@ -50,7 +50,6 @@ main() {
     if [ -z "${output}" ]; then
         error "no output specified, use -o\n"
     fi
-    mkdir -p "$(dirname "${output}")"
 
     # If the output file already exists and:
     # - there's no .hash file: do not download it again and exit promptly

Now we keep the git clone that we download and generates our tarball from there.

The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again.

This should speed the 'source' part of the build significantly.

The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 40 insertions(+), 29 deletions(-)

support/download/git +40 -29

index a49e448e60..834345b53a 100755
@@ -39,28 +39,34 @@ _git() {
     eval ${GIT} "${@}"
 }
 
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
-    printf "Doing shallow clone\n"
-    if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
-        git_done=1
-    else
-        printf "Shallow clone failed, falling back to doing a full clone\n"
+# We want to check if a cache of the git clone of this repo already exists.
+git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
+
+# If the cache directory already exists, don't try to clone.
+if [ ! -d "${git_cache}" ]; then
+    # Try a shallow clone, since it is faster than a full clone - but that
+    # only works if the versionis a ref (tag or branch). Before trying to do a
+    # shallow clone we check if ${cset} is in the list provided by git
+    # ls-remote. If not we fall back on a full clone.
+    #
+    # Messages for the type of clone used are provided to ease debugging in
+    # case of problems
+    git_done=0
+    if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
+        printf "Doing shallow clone\n"
+        if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
+            git_done=1
+        else
+            printf "Shallow clone failed, falling back to doing a full clone\n"
+        fi
+    fi
+    if [ ${git_done} -eq 0 ]; then
+        printf "Doing full clone\n"
+        _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
     fi
-fi
-if [ ${git_done} -eq 0 ]; then
-    printf "Doing full clone\n"
-    _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
 fi
 
-pushd "${basename}" >/dev/null
+pushd "${git_cache}" >/dev/null
 
 # Try to get the special refs exposed by some forges (pull-requests for
 # github, changes for gerrit...). There is no easy way to know whether

@@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
     _git submodule update --init --recursive
 fi
 
-# We do not want the .git dir; we keep other .git files, in case they
-# are the only files in their directory.
+# Generate the archive, sort with the C locale so that it is reproducible
+# We do not want the .git dir; we keep other .git
+# files, in case they are the only files in their directory.
 # The .git dir would generate non reproducible tarballs as it depends on
 # the state of the remote server. It also would generate large tarballs
 # (gigabytes for some linux trees) when a full clone took place.
-rm -rf .git
+find . -not -type d \
+	-and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
+LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
 
-popd >/dev/null
-
-# Generate the archive, sort with the C locale so that it is reproducible
-find "${basename}" -not -type d >"${basename}.list"
-LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
-         -T "${basename}.list.sorted" >"${output}.tar"
+tar cf - --transform="s/^\./${basename}/" \
+	--numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+         -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
 gzip -n <"${output}.tar" >"${output}"
+tar tf "${output}"
+
+rm -f "${BR2_DL_DIR}/${basename}.list"
+rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
+
+popd >/dev/null

Hello Maxime,

Quoted 2 lines · Maxime Hadjinlian

Now we keep the git clone that we download and generates our tarball from there.

For my Raspberry testcase with 'BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/raspberrypi/linux.git"' the cloned git repository is downloaded as 'dl/linux/git'.

This will lead to collisions e.g. for 'https://github.com/hardkernel/linux.git'.

I think a better approach would be to store the git clone under the complete source url (escaped), e.g. 'github-com-raspberrypi-linux-git' or 'github_com_raspberrypi_linux.git'?

Regards,
Peter

Quoted 108 lines

The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again.

This should speed the 'source' part of the build significantly.

The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays.

Signed-off-by Maxime Hadjinlian <[email protected]>

---
 support/download/git | 69 ++++++++++++++++++++++++++++++----------------------
 1 file changed, 40 insertions(+), 29 deletions(-)

support/download/git +40 -29

index a49e448e60..834345b53a 100755
@@ -39,28 +39,34 @@ _git() {
     eval ${GIT} "${@}"
 }
 
-# Try a shallow clone, since it is faster than a full clone - but that only
-# works if the version is a ref (tag or branch). Before trying to do a shallow
-# clone we check if ${cset} is in the list provided by git ls-remote. If not
-# we fall back on a full clone.
-#
-# Messages for the type of clone used are provided to ease debugging in case of
-# problems
-git_done=0
-if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
-    printf "Doing shallow clone\n"
-    if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then
-        git_done=1
-    else
-        printf "Shallow clone failed, falling back to doing a full clone\n"
+# We want to check if a cache of the git clone of this repo already exists.
+git_cache="${BR2_DL_DIR}/${basename%%-*}/git"
+
+# If the cache directory already exists, don't try to clone.
+if [ ! -d "${git_cache}" ]; then
+    # Try a shallow clone, since it is faster than a full clone - but that
+    # only works if the versionis a ref (tag or branch). Before trying to do a
+    # shallow clone we check if ${cset} is in the list provided by git
+    # ls-remote. If not we fall back on a full clone.
+    #
+    # Messages for the type of clone used are provided to ease debugging in
+    # case of problems
+    git_done=0
+    if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then
+        printf "Doing shallow clone\n"
+        if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then
+            git_done=1
+        else
+            printf "Shallow clone failed, falling back to doing a full clone\n"
+        fi
+    fi
+    if [ ${git_done} -eq 0 ]; then
+        printf "Doing full clone\n"
+        _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'"
     fi
-fi
-if [ ${git_done} -eq 0 ]; then
-    printf "Doing full clone\n"
-    _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'"
 fi
 
-pushd "${basename}" >/dev/null
+pushd "${git_cache}" >/dev/null
 
 # Try to get the special refs exposed by some forges (pull-requests for
 # github, changes for gerrit...). There is no easy way to know whether

@@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then
     _git submodule update --init --recursive
 fi
 
-# We do not want the .git dir; we keep other .git files, in case they
-# are the only files in their directory.
+# Generate the archive, sort with the C locale so that it is reproducible
+# We do not want the .git dir; we keep other .git
+# files, in case they are the only files in their directory.
 # The .git dir would generate non reproducible tarballs as it depends on
 # the state of the remote server. It also would generate large tarballs
 # (gigabytes for some linux trees) when a full clone took place.
-rm -rf .git
+find . -not -type d \
+	-and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list"
+LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted"
 
-popd >/dev/null
-
-# Generate the archive, sort with the C locale so that it is reproducible
-find "${basename}" -not -type d >"${basename}.list"
-LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted"
 # Create GNU-format tarballs, since that's the format of the tarballs on
 # sources.buildroot.org and used in the *.hash files
-tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
-         -T "${basename}.list.sorted" >"${output}.tar"
+tar cf - --transform="s/^\./${basename}/" \
+	--numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \
+         -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar"
 gzip -n <"${output}.tar" >"${output}"
+tar tf "${output}"
+
+rm -f "${BR2_DL_DIR}/${basename}.list"
+rm -f "${BR2_DL_DIR}/${basename}.list.sorted"
+
+popd >/dev/null

Hi all,

This series will progressively reorganize the DL_DIR folder, in order to hve a subdirectory for each packages.
This way, we store patches into subdirectory called by the package name instead of flattly into DL_DIR.
We also avoid possible conflicts between packages.

Finally, this series introduce the 'git cache' feature. What it does is that it keeps a copy of the Git clone in the subdirectory of the package.
This way, when you change the version of said package, instead of cloning from scratch, we only fetch the missing objects and create the tarball from that git clone.

It should speed up the build, especially for some use cases; drawback is that the disk usage for the DL_DIR will increase significantly. But disk space are pretty cheap, especially on build servers.

Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
pkg-generic: make PKG_DL_DIR equal to DL_DIR
packages: use new $($PKG)_DL_DIR) variable pkg-{download,generic}: use new $($(PKG)_DL_DIR) support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature

Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options

 Config.in                                          |   3 +
 docs/manual/adding-packages-generic.txt            |   6 +-
 package/amd-catalyst/amd-catalyst.mk               |   2 +-
 package/android-tools/android-tools.mk             |   2 +-
 package/angularjs/angularjs.mk                     |   2 +-
 package/bootstrap/bootstrap.mk                     |   2 +-
 package/cache-calibrator/cache-calibrator.mk       |   2 +-
 package/cracklib/cracklib.mk                       |   2 +-
 package/cryptopp/cryptopp.mk                       |   2 +-
 package/devmem2/devmem2.mk                         |   2 +-
 package/dhrystone/dhrystone.mk                     |   2 +-
 package/doom-wad/doom-wad.mk                       |   2 +-
 package/espeak/espeak.mk                           |   2 +-
 package/fan-ctrl/fan-ctrl.mk                       |   2 +-
 package/gcc/gcc.mk                                 |   2 +-
 package/irrlicht/irrlicht.mk                       |   2 +-
 package/jquery-mobile/jquery-mobile.mk             |   2 +-
 package/jquery-sparkline/jquery-sparkline.mk       |   2 +-
 package/jquery-ui-themes/jquery-ui-themes.mk       |   2 +-
 package/jquery-ui/jquery-ui.mk                     |   2 +-
 package/jquery-validation/jquery-validation.mk     |   2 +-
 package/jquery/jquery.mk                           |   2 +-
 package/kodi/kodi.mk                               |   6 +-
 package/libfreeimage/libfreeimage.mk               |   2 +-
 package/libjson/libjson.mk                         |   2 +-
 package/libsoil/libsoil.mk                         |   2 +-
 package/lsof/lsof.mk                               |   2 +-
 package/musl-compat-headers/musl-compat-headers.mk |   2 +-
 package/nmon/nmon.mk                               |   2 +-
 package/nvidia-driver/nvidia-driver.mk             |   2 +-
 package/openobex/openobex.mk                       |   2 +-
 package/opentyrian-data/opentyrian-data.mk         |   2 +-
 package/perl/perl.mk                               |   2 +-
 package/pkg-download.mk                            | 216 ++++-----------------
 package/pkg-generic.mk                             |  10 +-
 package/python-keyring/python-keyring.mk           |   2 +-
 .../python-simplegeneric/python-simplegeneric.mk   |   2 +-
 package/rapidxml/rapidxml.mk                       |   2 +-
 package/rpi-wifi-firmware/rpi-wifi-firmware.mk     |   2 +-
 package/sam-ba/sam-ba.mk                           |   2 +-
 package/spidev_test/spidev_test.mk                 |   2 +-
 package/tar/tar.mk                                 |   2 +-
 package/tesseract-ocr/tesseract-ocr.mk             |   2 +-
 package/ti-cgt-pru/ti-cgt-pru.mk                   |   4 +-
 package/ti-gfx/ti-gfx.mk                           |   4 +-
 package/ts4900-fpga/ts4900-fpga.mk                 |   2 +-
 package/unscd/unscd.mk                             |   2 +-
 package/urg/urg.mk                                 |   2 +-
 package/waf/waf.mk                                 |   2 +-
 package/whetstone/whetstone.mk                     |   2 +-
 package/wilc1000-firmware/wilc1000-firmware.mk     |   2 +-
 package/zynq-boot-bin/zynq-boot-bin.mk             |   2 +-
 support/download/bzr                               |  25 +--
 support/download/check-hash                        |   2 +-
 support/download/cp                                |  17 +-
 support/download/cvs                               |  34 ++--
 support/download/dl-wrapper                        |  62 +++---
 support/download/git                               |  96 +++++----
 support/download/hg                                |  25 +--
 support/download/scp                               |  19 +-
 support/download/svn                               |  25 +--
 support/download/wget                              |  24 ++-
 62 files changed, 286 insertions(+), 382 deletions(-)

--
2.13.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hello Maxime,

Quoted 8 lines · Maxime Hadjinlian

Hi all,

This series will progressively reorganize the DL_DIR folder, in order to hve a subdirectory for each packages.
This way, we store patches into subdirectory called by the package name instead of flattly into DL_DIR.
We also avoid possible conflicts between packages.

Quoted 7 lines

Finally, this series introduce the 'git cache' feature. What it does is that it keeps a copy of the Git clone in the subdirectory of the package.
This way, when you change the version of said package, instead of cloning from scratch, we only fetch the missing objects and create the tarball from that git clone.

Tested your patch series (hoping to speed up RPi kernel/userland/firmware update which are huge git downloads) but failed with the raspberrypi3_defconfig while compiling the linux kernel:

DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,

                 from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^
compilation terminated.
  DTC     arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
                 from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^

compilation terminated.

This is due to wrong dt-bindings link, the failing version (with your patchset):

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings

lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings

And from a working version (without your patchset):

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings

Regards,

Peter

Quoted 92 lines

It should speed up the build, especially for some use cases; drawback is that the disk usage for the DL_DIR will increase significantly. But disk space are pretty cheap, especially on build servers.

Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
pkg-generic: make PKG_DL_DIR equal to DL_DIR
packages: use new $($PKG)_DL_DIR) variable pkg-{download,generic}: use new $($(PKG)_DL_DIR) support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature

Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options

 Config.in                                          |   3 +
 docs/manual/adding-packages-generic.txt            |   6 +-
 package/amd-catalyst/amd-catalyst.mk               |   2 +-
 package/android-tools/android-tools.mk             |   2 +-
 package/angularjs/angularjs.mk                     |   2 +-
 package/bootstrap/bootstrap.mk                     |   2 +-
 package/cache-calibrator/cache-calibrator.mk       |   2 +-
 package/cracklib/cracklib.mk                       |   2 +-
 package/cryptopp/cryptopp.mk                       |   2 +-
 package/devmem2/devmem2.mk                         |   2 +-
 package/dhrystone/dhrystone.mk                     |   2 +-
 package/doom-wad/doom-wad.mk                       |   2 +-
 package/espeak/espeak.mk                           |   2 +-
 package/fan-ctrl/fan-ctrl.mk                       |   2 +-
 package/gcc/gcc.mk                                 |   2 +-
 package/irrlicht/irrlicht.mk                       |   2 +-
 package/jquery-mobile/jquery-mobile.mk             |   2 +-
 package/jquery-sparkline/jquery-sparkline.mk       |   2 +-
 package/jquery-ui-themes/jquery-ui-themes.mk       |   2 +-
 package/jquery-ui/jquery-ui.mk                     |   2 +-
 package/jquery-validation/jquery-validation.mk     |   2 +-
 package/jquery/jquery.mk                           |   2 +-
 package/kodi/kodi.mk                               |   6 +-
 package/libfreeimage/libfreeimage.mk               |   2 +-
 package/libjson/libjson.mk                         |   2 +-
 package/libsoil/libsoil.mk                         |   2 +-
 package/lsof/lsof.mk                               |   2 +-
 package/musl-compat-headers/musl-compat-headers.mk |   2 +-
 package/nmon/nmon.mk                               |   2 +-
 package/nvidia-driver/nvidia-driver.mk             |   2 +-
 package/openobex/openobex.mk                       |   2 +-
 package/opentyrian-data/opentyrian-data.mk         |   2 +-
 package/perl/perl.mk                               |   2 +-
 package/pkg-download.mk                            | 216 ++++-----------------
 package/pkg-generic.mk                             |  10 +-
 package/python-keyring/python-keyring.mk           |   2 +-
 .../python-simplegeneric/python-simplegeneric.mk   |   2 +-
 package/rapidxml/rapidxml.mk                       |   2 +-
 package/rpi-wifi-firmware/rpi-wifi-firmware.mk     |   2 +-
 package/sam-ba/sam-ba.mk                           |   2 +-
 package/spidev_test/spidev_test.mk                 |   2 +-
 package/tar/tar.mk                                 |   2 +-
 package/tesseract-ocr/tesseract-ocr.mk             |   2 +-
 package/ti-cgt-pru/ti-cgt-pru.mk                   |   4 +-
 package/ti-gfx/ti-gfx.mk                           |   4 +-
 package/ts4900-fpga/ts4900-fpga.mk                 |   2 +-
 package/unscd/unscd.mk                             |   2 +-
 package/urg/urg.mk                                 |   2 +-
 package/waf/waf.mk                                 |   2 +-
 package/whetstone/whetstone.mk                     |   2 +-
 package/wilc1000-firmware/wilc1000-firmware.mk     |   2 +-
 package/zynq-boot-bin/zynq-boot-bin.mk             |   2 +-
 support/download/bzr                               |  25 +--
 support/download/check-hash                        |   2 +-
 support/download/cp                                |  17 +-
 support/download/cvs                               |  34 ++--
 support/download/dl-wrapper                        |  62 +++---
 support/download/git                               |  96 +++++----
 support/download/hg                                |  25 +--
 support/download/scp                               |  19 +-
 support/download/svn                               |  25 +--
 support/download/wget                              |  24 ++-
 62 files changed, 286 insertions(+), 382 deletions(-)

--
2.13.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hi Peter, all

Quoted 42 lines · Peter Seiderer

Hello Maxime,

Quoted 8 lines · Maxime Hadjinlian

Hi all,

This series will progressively reorganize the DL_DIR folder, in order to hve a subdirectory for each packages.
This way, we store patches into subdirectory called by the package name instead of flattly into DL_DIR.
We also avoid possible conflicts between packages.

Quoted 7 lines

Finally, this series introduce the 'git cache' feature. What it does is that it keeps a copy of the Git clone in the subdirectory of the package.
This way, when you change the version of said package, instead of cloning from scratch, we only fetch the missing objects and create the tarball from that git clone.

Tested your patch series (hoping to speed up RPi kernel/userland/firmware update which are huge git downloads) but failed with the raspberrypi3_defconfig while compiling the linux kernel:

DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,

                 from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^
compilation terminated.
  DTC     arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
                 from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^

compilation terminated.

This is due to wrong dt-bindings link, the failing version (with your patchset):

Quoted 9 lines

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings

lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings

And from a working version (without your patchset):

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings

Thanks a lot for your test, made my realise I had a bug when I create the tarball from the git repo, I use a regex to transform './' into '$basename' but I forgot to put the '$' in my regex, hence the '../' becomming 'linux-sha1./'

It's fixed on my branch on github for now.

Quoted 98 lines

Regards,

Peter

Quoted 92 lines

It should speed up the build, especially for some use cases; drawback is that the disk usage for the DL_DIR will increase significantly. But disk space are pretty cheap, especially on build servers.

Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
pkg-generic: make PKG_DL_DIR equal to DL_DIR
packages: use new $($PKG)_DL_DIR) variable pkg-{download,generic}: use new $($(PKG)_DL_DIR) support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature

Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options

 Config.in                                          |   3 +
 docs/manual/adding-packages-generic.txt            |   6 +-
 package/amd-catalyst/amd-catalyst.mk               |   2 +-
 package/android-tools/android-tools.mk             |   2 +-
 package/angularjs/angularjs.mk                     |   2 +-
 package/bootstrap/bootstrap.mk                     |   2 +-
 package/cache-calibrator/cache-calibrator.mk       |   2 +-
 package/cracklib/cracklib.mk                       |   2 +-
 package/cryptopp/cryptopp.mk                       |   2 +-
 package/devmem2/devmem2.mk                         |   2 +-
 package/dhrystone/dhrystone.mk                     |   2 +-
 package/doom-wad/doom-wad.mk                       |   2 +-
 package/espeak/espeak.mk                           |   2 +-
 package/fan-ctrl/fan-ctrl.mk                       |   2 +-
 package/gcc/gcc.mk                                 |   2 +-
 package/irrlicht/irrlicht.mk                       |   2 +-
 package/jquery-mobile/jquery-mobile.mk             |   2 +-
 package/jquery-sparkline/jquery-sparkline.mk       |   2 +-
 package/jquery-ui-themes/jquery-ui-themes.mk       |   2 +-
 package/jquery-ui/jquery-ui.mk                     |   2 +-
 package/jquery-validation/jquery-validation.mk     |   2 +-
 package/jquery/jquery.mk                           |   2 +-
 package/kodi/kodi.mk                               |   6 +-
 package/libfreeimage/libfreeimage.mk               |   2 +-
 package/libjson/libjson.mk                         |   2 +-
 package/libsoil/libsoil.mk                         |   2 +-
 package/lsof/lsof.mk                               |   2 +-
 package/musl-compat-headers/musl-compat-headers.mk |   2 +-
 package/nmon/nmon.mk                               |   2 +-
 package/nvidia-driver/nvidia-driver.mk             |   2 +-
 package/openobex/openobex.mk                       |   2 +-
 package/opentyrian-data/opentyrian-data.mk         |   2 +-
 package/perl/perl.mk                               |   2 +-
 package/pkg-download.mk                            | 216 ++++-----------------
 package/pkg-generic.mk                             |  10 +-
 package/python-keyring/python-keyring.mk           |   2 +-
 .../python-simplegeneric/python-simplegeneric.mk   |   2 +-
 package/rapidxml/rapidxml.mk                       |   2 +-
 package/rpi-wifi-firmware/rpi-wifi-firmware.mk     |   2 +-
 package/sam-ba/sam-ba.mk                           |   2 +-
 package/spidev_test/spidev_test.mk                 |   2 +-
 package/tar/tar.mk                                 |   2 +-
 package/tesseract-ocr/tesseract-ocr.mk             |   2 +-
 package/ti-cgt-pru/ti-cgt-pru.mk                   |   4 +-
 package/ti-gfx/ti-gfx.mk                           |   4 +-
 package/ts4900-fpga/ts4900-fpga.mk                 |   2 +-
 package/unscd/unscd.mk                             |   2 +-
 package/urg/urg.mk                                 |   2 +-
 package/waf/waf.mk                                 |   2 +-
 package/whetstone/whetstone.mk                     |   2 +-
 package/wilc1000-firmware/wilc1000-firmware.mk     |   2 +-
 package/zynq-boot-bin/zynq-boot-bin.mk             |   2 +-
 support/download/bzr                               |  25 +--
 support/download/check-hash                        |   2 +-
 support/download/cp                                |  17 +-
 support/download/cvs                               |  34 ++--
 support/download/dl-wrapper                        |  62 +++---
 support/download/git                               |  96 +++++----
 support/download/hg                                |  25 +--
 support/download/scp                               |  19 +-
 support/download/svn                               |  25 +--
 support/download/wget                              |  24 ++-
 62 files changed, 286 insertions(+), 382 deletions(-)

--
2.13.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hello Maxime,

Quoted 59 lines · Maxime Hadjinlian

Hi Peter, all

Quoted 42 lines · Peter Seiderer

Hello Maxime,

Quoted 8 lines · Maxime Hadjinlian

Hi all,

This series will progressively reorganize the DL_DIR folder, in order to hve a subdirectory for each packages.
This way, we store patches into subdirectory called by the package name instead of flattly into DL_DIR.
We also avoid possible conflicts between packages.

Quoted 7 lines

Finally, this series introduce the 'git cache' feature. What it does is that it keeps a copy of the Git clone in the subdirectory of the package.
This way, when you change the version of said package, instead of cloning from scratch, we only fetch the missing objects and create the tarball from that git clone.

Tested your patch series (hoping to speed up RPi kernel/userland/firmware update which are huge git downloads) but failed with the raspberrypi3_defconfig while compiling the linux kernel:

DTC arch/arm/boot/dts/bcm2710-rpi-3-b.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,

                 from arch/arm/boot/dts/bcm2710-rpi-3-b.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^
compilation terminated.
  DTC     arch/arm/boot/dts/bcm2710-rpi-cm3.dtb
In file included from arch/arm/boot/dts/bcm2710.dtsi:1:0,
                 from arch/arm/boot/dts/bcm2710-rpi-cm3.dts:3:
arch/arm/boot/dts/bcm283x.dtsi:1:41: fatal error: dt-bindings/pinctrl/bcm2835.h: No such file or directory
 #include <dt-bindings/pinctrl/bcm2835.h>
                                         ^

compilation terminated.

This is due to wrong dt-bindings link, the failing version (with your patchset):

Quoted 9 lines

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings

lrwxrwxrwx 1 seiderer users 79 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> linux-9126e25b0934bd7bd843763310ea4b34c6e139d0./../../../../include/dt-bindings

And from a working version (without your patchset):

        $ ls -ld build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings
lrwxrwxrwx 1 seiderer users 34 Jul 11 23:15 build/linux-9126e25b0934bd7bd843763310ea4b34c6e139d0/arch/arm/boot/dts/include/dt-bindings -> ../../../../../include/dt-bindings

Thanks a lot for your test, made my realise I had a bug when I create the tarball from the git repo, I use a regex to transform './' into '$basename' but I forgot to put the '$' in my regex, hence the '../' becomming 'linux-sha1./'

Quoted 1 line

It's fixed on my branch on github for now.

Any updates on this one? Mind to resend an actual version (would like to see this feature integrated into buildroot, tired of waiting for hours in case RaspberryPi kernel/firmware/userland git versions are updated)?

Regards,
Peter

Quoted 98 lines

Quoted 98 lines

Regards,

Peter

Quoted 92 lines

It should speed up the build, especially for some use cases; drawback is that the disk usage for the DL_DIR will increase significantly. But disk space are pretty cheap, especially on build servers.

Maxime Hadjinlian (12):
pkg-{download,generic}: remove source-check
download: put most of the infra in dl-wrapper
pkg-generic: make PKG_DL_DIR equal to DL_DIR
packages: use new $($PKG)_DL_DIR) variable pkg-{download,generic}: use new $($(PKG)_DL_DIR) support/download: make sure the download folder is created
pkg-generic: add a subdirectory to the DL_DIR
pkg-download: support new subdir for mirrors
pkg-generic: introduce _SAME_SOURCE_AS help/manual: update help about the new $(LIBFOO_DL_DIR)
download: add flock call before dl-wrapper
download: git: introduce cache feature

Yann E. MORIN (1):
core/pkg-download: change all helpers to use common options

 Config.in                                          |   3 +
 docs/manual/adding-packages-generic.txt            |   6 +-
 package/amd-catalyst/amd-catalyst.mk               |   2 +-
 package/android-tools/android-tools.mk             |   2 +-
 package/angularjs/angularjs.mk                     |   2 +-
 package/bootstrap/bootstrap.mk                     |   2 +-
 package/cache-calibrator/cache-calibrator.mk       |   2 +-
 package/cracklib/cracklib.mk                       |   2 +-
 package/cryptopp/cryptopp.mk                       |   2 +-
 package/devmem2/devmem2.mk                         |   2 +-
 package/dhrystone/dhrystone.mk                     |   2 +-
 package/doom-wad/doom-wad.mk                       |   2 +-
 package/espeak/espeak.mk                           |   2 +-
 package/fan-ctrl/fan-ctrl.mk                       |   2 +-
 package/gcc/gcc.mk                                 |   2 +-
 package/irrlicht/irrlicht.mk                       |   2 +-
 package/jquery-mobile/jquery-mobile.mk             |   2 +-
 package/jquery-sparkline/jquery-sparkline.mk       |   2 +-
 package/jquery-ui-themes/jquery-ui-themes.mk       |   2 +-
 package/jquery-ui/jquery-ui.mk                     |   2 +-
 package/jquery-validation/jquery-validation.mk     |   2 +-
 package/jquery/jquery.mk                           |   2 +-
 package/kodi/kodi.mk                               |   6 +-
 package/libfreeimage/libfreeimage.mk               |   2 +-
 package/libjson/libjson.mk                         |   2 +-
 package/libsoil/libsoil.mk                         |   2 +-
 package/lsof/lsof.mk                               |   2 +-
 package/musl-compat-headers/musl-compat-headers.mk |   2 +-
 package/nmon/nmon.mk                               |   2 +-
 package/nvidia-driver/nvidia-driver.mk             |   2 +-
 package/openobex/openobex.mk                       |   2 +-
 package/opentyrian-data/opentyrian-data.mk         |   2 +-
 package/perl/perl.mk                               |   2 +-
 package/pkg-download.mk                            | 216 ++++-----------------
 package/pkg-generic.mk                             |  10 +-
 package/python-keyring/python-keyring.mk           |   2 +-
 .../python-simplegeneric/python-simplegeneric.mk   |   2 +-
 package/rapidxml/rapidxml.mk                       |   2 +-
 package/rpi-wifi-firmware/rpi-wifi-firmware.mk     |   2 +-
 package/sam-ba/sam-ba.mk                           |   2 +-
 package/spidev_test/spidev_test.mk                 |   2 +-
 package/tar/tar.mk                                 |   2 +-
 package/tesseract-ocr/tesseract-ocr.mk             |   2 +-
 package/ti-cgt-pru/ti-cgt-pru.mk                   |   4 +-
 package/ti-gfx/ti-gfx.mk                           |   4 +-
 package/ts4900-fpga/ts4900-fpga.mk                 |   2 +-
 package/unscd/unscd.mk                             |   2 +-
 package/urg/urg.mk                                 |   2 +-
 package/waf/waf.mk                                 |   2 +-
 package/whetstone/whetstone.mk                     |   2 +-
 package/wilc1000-firmware/wilc1000-firmware.mk     |   2 +-
 package/zynq-boot-bin/zynq-boot-bin.mk             |   2 +-
 support/download/bzr                               |  25 +--
 support/download/check-hash                        |   2 +-
 support/download/cp                                |  17 +-
 support/download/cvs                               |  34 ++--
 support/download/dl-wrapper                        |  62 +++---
 support/download/git                               |  96 +++++----
 support/download/hg                                |  25 +--
 support/download/scp                               |  19 +-
 support/download/svn                               |  25 +--
 support/download/wget                              |  24 ++-
 62 files changed, 286 insertions(+), 382 deletions(-)

--
2.13.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Hi Peter,

Quoted 4 lines · Peter Seiderer

Hello Maxime,

On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <[email protected]> wrote:

Quoted 4 lines

Quoted 1 line

It's fixed on my branch on github for now.

Any updates on this one? Mind to resend an actual version (would like to see this feature integrated into buildroot, tired of waiting for hours in case RaspberryPi kernel/firmware/userland git versions are updated)?

I think the most "up-to-date" is my branch on Github
(https://github.com/maximeh/buildroot/tree/git_cache).
I don't really think it's ready for a review, there's still many bugs in there, at least that's what I remember.
What would really help me is, if you could test this and tell me what breaks so I could look it up and try to fix this, this would be a huge help !
If you come up with patches, that's really nice too ;).

I won't make any promises on the time I can spend on this, we all know how this goes.. but that being said, I am still really motivated by this feature, I really need it too.

Quoted 7 lines · Maxime Hadjinlian

Hi Peter,

Quoted 4 lines · Peter Seiderer

Hello Maxime,

On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <[email protected]> wrote:

[snip]

Quoted 13 lines

Quoted 4 lines

Quoted 1 line

It's fixed on my branch on github for now.

Any updates on this one? Mind to resend an actual version (would like to see this feature integrated into buildroot, tired of waiting for hours in case RaspberryPi kernel/firmware/userland git versions are updated)?

I think the most "up-to-date" is my branch on Github
(https://github.com/maximeh/buildroot/tree/git_cache).
I don't really think it's ready for a review, there's still many bugs in there, at least that's what I remember.
What would really help me is, if you could test this and tell me what breaks so I could look it up and try to fix this, this would be a huge help !
If you come up with patches, that's really nice too ;).

Quoted 3 lines

I won't make any promises on the time I can spend on this, we all know how this goes.. but that being said, I am still really motivated by this feature, I really need it too.

Maxime, correct me if I'm wrong, but you'd be happy if Peter does a respin of
the series, right?

It would be nice if we could already apply the first few patches, just to keep
things going.

Regards,
Arnout

--

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Quoted 24 lines · Arnout Vandecappelle

Quoted 7 lines · Maxime Hadjinlian

Hi Peter,

Quoted 4 lines · Peter Seiderer

Hello Maxime,

On Sun, 30 Jul 2017 18:32:27 +0200, Maxime Hadjinlian <[email protected]> wrote:

[snip]

Quoted 13 lines

Quoted 4 lines

Quoted 1 line

It's fixed on my branch on github for now.

Any updates on this one? Mind to resend an actual version (would like to see this feature integrated into buildroot, tired of waiting for hours in case RaspberryPi kernel/firmware/userland git versions are updated)?

I think the most "up-to-date" is my branch on Github
(https://github.com/maximeh/buildroot/tree/git_cache).
I don't really think it's ready for a review, there's still many bugs in there, at least that's what I remember.
What would really help me is, if you could test this and tell me what breaks so I could look it up and try to fix this, this would be a huge help !
If you come up with patches, that's really nice too ;).

Quoted 6 lines

Quoted 3 lines

I won't make any promises on the time I can spend on this, we all know how this goes.. but that being said, I am still really motivated by this feature, I really need it too.

Maxime, correct me if I'm wrong, but you'd be happy if Peter does a respin of
the series, right?

Oh sure, for me, the important part is that it's merged, who gets to do is not. Since I clearly can't make the time to make it happen; if someone can respin this, then please do it :). At least the topic is out there :).

Quoted 3 lines

It would be nice if we could already apply the first few patches, just to keep
things going.

Agreed.

Quoted 11 lines

Regards,
Arnout

--

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF