core/sdk: generate the SDK tarball ourselves

PATCH New Yann E. MORIN 7 participants 26 messages Jun 9, 2018 – Jun 15, 2018 Kbuild

Share & tools

Yann E. MORIN

author

2018-06-09 21:06 UTC #0 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Currently, the wording in the manual instructs the user to generate tarball from "the contents of the +output/host+ directory".

This is pretty confusing, because taken literally, this would amount to runing a command like:

tar cf my-sdk.tar -C output/host/ .

This creates a tarbomb [0], which is very bad practice, because when extracted, it creates mutiple files in the current directory.

One one really want to do, is create a tarball of the host/ directory, with something like:

tar cf my-sdk.tar -C output host/

However, this is not much better, because the top-most dirdctory would have a very common name, host/, which is pretty easy to get conflict with.

So, we fix that mess by creating the archive ourselves, giving it and the top-most directory a recogniseable name, based on the target tuple and the Buildroot version.

Since this is an output file, we located it in the images/ directory.

Update the manual accordignly.

Speaking of the manual.. It was referring to "output/host/", but that is only valid for in-tree builds. For out-of-tree builds, this is just "host/". To avoid confusion, use the name of the vairiable $(HOST_DIR), which, fortunately, happens to be valid in both cases.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Wolfgang Grandegger <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Stefan Becker <[email protected]>

---
 Makefile                                  |  7 ++++++-
 docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
 2 files changed, 23 insertions(+), 10 deletions(-)

Makefile +6 -1

index 4b998bdb65..3f460f8d92 100644
@@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 
 .PHONY: sdk
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
+		.
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):

docs/manual/using-buildroot-toolchain.txt +17 -9

index 3246dc2411..d1490b4c38 100644
@@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
 +output/host/bin/+ to your PATH environment variable and then to
 use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
 
-It is possible to relocate the toolchain, this allows to distribute
-the toolchain to other developers to build applications for your
-target. To achieve this:
+Alternatively, Buildroot can also export the toolchain and all the
+development files of all selected packages, as an SDK, by running the
+command +make sdk+. This generates a tarball of the content of the host
+directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
+and located in the output directory +$(BINARIES_DIR)+.
 
-* run +make sdk+, which prepares the toolchain to be relocatable;
-* tarball the contents of the +output/host+ directory;
-* distribute the resulting tarball.
+This tarball can then be distributed to application developpers, when
+they want to develop their applications that are not (yet) packaged as
+a Buildroot package.
 
-Once the toolchain is installed to the new location, the user must run
-the +relocate-sdk.sh+ script to make sure all paths are updated with
-the new location.
+Upon extracting the SDK tarball, the user must run the script
++relocate-sdk.sh+, to make sure all paths are updated with the new
+location.
 
+.Note
+This SDK can not be re-used as an external toolchain, because it
+contains pre-built libraries that could be conflicting with the ones
+packaged in Buildroot (e.g. when an old SDK would be re-used with a
+newer Buildroot version), unless it was built from a configuration
+with no package enabled.

Signature · 1 line

2.14.1

Yann E. MORIN

author

2018-06-09 21:16 UTC #1 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

All,

On 2018-06-09 23:06 +0200, Yann E. MORIN spake thusly:

Quoted 9 lines

Currently, the wording in the manual instructs the user to generate tarball from "the contents of the +output/host+ directory".
This is pretty confusing, because taken literally, this would amount to runing a command like:
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when

I forgot to add the reference to the tarbomb:
https://en.wikipedia.org/wiki/Tar_(computing)#Tarbomb

Regards,
Yann E. MORIN.

Quoted 98 lines

extracted, it creates mutiple files in the current directory.

One one really want to do, is create a tarball of the host/ directory, with something like:

tar cf my-sdk.tar -C output host/

However, this is not much better, because the top-most dirdctory would have a very common name, host/, which is pretty easy to get conflict with.

So, we fix that mess by creating the archive ourselves, giving it and the top-most directory a recogniseable name, based on the target tuple and the Buildroot version.

Since this is an output file, we located it in the images/ directory.

Update the manual accordignly.

Signed-off-by "Yann E. MORIN" <[email protected]>

Cc Wolfgang Grandegger <[email protected]>

Cc Thomas Petazzoni <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Stefan Becker <[email protected]>

---
 Makefile                                  |  7 ++++++-
 docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
 2 files changed, 23 insertions(+), 10 deletions(-)

Makefile +6 -1

index 4b998bdb65..3f460f8d92 100644
@@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 
 .PHONY: sdk
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
+		.
 
 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):

docs/manual/using-buildroot-toolchain.txt +17 -9

index 3246dc2411..d1490b4c38 100644
@@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
 +output/host/bin/+ to your PATH environment variable and then to
 use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
 
-It is possible to relocate the toolchain, this allows to distribute
-the toolchain to other developers to build applications for your
-target. To achieve this:
+Alternatively, Buildroot can also export the toolchain and all the
+development files of all selected packages, as an SDK, by running the
+command +make sdk+. This generates a tarball of the content of the host
+directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
+and located in the output directory +$(BINARIES_DIR)+.
 
-* run +make sdk+, which prepares the toolchain to be relocatable;
-* tarball the contents of the +output/host+ directory;
-* distribute the resulting tarball.
+This tarball can then be distributed to application developpers, when
+they want to develop their applications that are not (yet) packaged as
+a Buildroot package.
 
-Once the toolchain is installed to the new location, the user must run
-the +relocate-sdk.sh+ script to make sure all paths are updated with
-the new location.
+Upon extracting the SDK tarball, the user must run the script
++relocate-sdk.sh+, to make sure all paths are updated with the new
+location.
 
+.Note
+This SDK can not be re-used as an external toolchain, because it
+contains pre-built libraries that could be conflicting with the ones
+packaged in Buildroot (e.g. when an old SDK would be re-used with a
+newer Buildroot version), unless it was built from a configuration
+with no package enabled.

Signature · 1 line

2.14.1

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Thomas Petazzoni

2018-06-10 06:03 UTC #2 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Yann E. MORIN

Hello,

Quoted 6 lines · Yann E. MORIN

Currently, the wording in the manual instructs the user to generate tarball from "the contents of the +output/host+ directory".
This is pretty confusing, because taken literally, this would amount to runing a command like:

Quoted 6 lines

tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when extracted, it creates mutiple files in the current directory.
One one really want to do, is create a tarball of the host/ directory,

"One one"

Quoted 5 lines

with something like:
tar cf my-sdk.tar -C output host/
However, this is not much better, because the top-most dirdctory would

Yann E. MORIN

author

2018-06-10 07:47 UTC #3 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Thomas, All,

On 2018-06-10 08:03 +0200, Thomas Petazzoni spake thusly:

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 2 lines

Quoted 1 line
So, we fix that mess by creating the archive ourselves, giving it and
remove the final "and"

Here, 'it' refers to 'the archive' listed just before. We want to give 'the archive' and 'the top-most directory'. So, this is really valid english grammar. ;-)

I'll rephrase to avoid confusion:

So, we fix that mess by giving the top-most directory a recogniseable name, based on the target tuple and the Buildroot version, which we also use as the name of the archive (suffixed with the usual +.tar.gz+.)

Quoted 6 lines

Quoted 4 lines
the top-most directory a recogniseable name, based on the target tuple and the Buildroot version.
Since this is an output file, we located it in the images/ directory.
located -> locate
or maybe "place", "store" ?

OK.

Quoted 23 lines

Quoted 4 lines
.PHONY: sdk
Makefile +1 -1
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 588
Quoted 10 lines
Makefile +5 -0
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
+		.
Generally, I am fine with the principle, I believe it indeed makes sense to provide a tarball that is ready to use.
I was a bit concerned about backward compatibility behavior for people already using "make sdk". But in fact your change is fine from this point of view: if people have scripts today that run "make sdk" and create a tarball from output/host, they will still work fine.

Quoted 8 lines

docs/manual/using-buildroot-toolchain.txt The toolchain generated by Buildroot is located by default in
Quoted 5 lines
docs/manual/using-buildroot-toolchain.txt +2 -3
-It is possible to relocate the toolchain, this allows to distribute
-the toolchain to other developers to build applications for your
-target. To achieve this:
+Alternatively, Buildroot can also export the toolchain and all the
+development files of all selected packages,
"all the development files" -> "the development files", otherwise the repetition of "all" is a bit annoying.

ACK.

Quoted 1 line

Looks good otherwise. Thanks!

OK, thanks for the review! :-)

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Arnout Vandecappelle

2018-06-10 21:21 UTC #4 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Yann E. MORIN

On 09-06-18 23:06, Yann E. MORIN wrote:
[snip]

Quoted 2 lines

So, we fix that mess by creating the archive ourselves, giving it and the top-most directory a recogniseable name, based on the target tuple

recognisable

Quoted 1 line

and the Buildroot version.

Great idea!

Quoted 3 lines

Since this is an output file, we located it in the images/ directory.
Update the manual accordignly.

accordingly

Quoted 4 lines

Speaking of the manual.. It was referring to "output/host/", but that is only valid for in-tree builds. For out-of-tree builds, this is just "host/". To avoid confusion, use the name of the vairiable $(HOST_DIR),

variable

Quoted 1 line

which, fortunately, happens to be valid in both cases.

Small nit about this part: this is the "user" section of the manual, and
HOST_DIR hasn't been introduced there. So I'm not so sure that HOST_DIR is such an improvement. If you want to be fully accurate, you could use "output/host or $(O)/host".

Alternatively, we could introduce HOST_DIR (and STAGING and TARGET probably) in
the O= section in common-usage.txt.

Quoted 18 lines

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Wolfgang Grandegger <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Stefan Becker <[email protected]>
---
 Makefile                                  |  7 ++++++-
 docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++---------
 2 files changed, 23 insertions(+), 10 deletions(-)
Makefile +0 -0
index 4b998bdb65..3f460f8d92 100644
@@ -574,13 +574,18 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 world: target-post-image
 
 .PHONY: sdk

Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 585

Quoted 10 lines

Makefile +3 -1

-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \

Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in 'make sdk' I don't mind that much.

Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 587

Quoted 2 lines

Makefile +2 -0

+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \

Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?

And let's do some bikeshedding about the name... I don't like the . between sdk
and the tuple. Also, I think it's better if sdk is at the beginning instead of buildroot. Or maybe even the tuple, that comes closer to how you invoke the compiler itself. Finally, Debian uses _ to distinguish the different parts in their .deb filenames, because both the package name and the version usually contain dashes and it may be hard to distinguish afterwards. So, I propose:

$(GNU_TARGET_NAME)_sdk-buildroot-$(BR2_VERSION_FULL).

Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 588

Quoted 1 line

Makefile +1 -0
+		.

It's a really stupid nit, but I don't like this single . isolated on a line,
because it's so easy to miss. So perhaps you could make it:

	--transform=... \
	-C $(HOST_DIR) .

(I also like the -C to be just before the files argument).

Quoted 14 lines

Makefile +0 -0

 # Populating the staging with the base directories is handled by the skeleton package
 $(STAGING_DIR):

docs/manual/using-buildroot-toolchain.txt +0 -3

index 3246dc2411..d1490b4c38 100644
@@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in
 +output/host/bin/+ to your PATH environment variable and then to
 use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc.
 
-It is possible to relocate the toolchain, this allows to distribute
-the toolchain to other developers to build applications for your
-target. To achieve this:

docs/manual/using-buildroot-toolchain.txt The toolchain generated by Buildroot is located by default in · line 21

Quoted 10 lines

docs/manual/using-buildroot-toolchain.txt +6 -3

+Alternatively, Buildroot can also export the toolchain and all the
+development files of all selected packages, as an SDK, by running the
+command +make sdk+. This generates a tarball of the content of the host
+directory +$(HOST_DIR)+, named +buildroot-sdk.<TARGET-TUPLE>-<BR-VERSION>.tar.gz+
+and located in the output directory +$(BINARIES_DIR)+.
 
-* run +make sdk+, which prepares the toolchain to be relocatable;
-* tarball the contents of the +output/host+ directory;
-* distribute the resulting tarball.
+This tarball can then be distributed to application developpers, when

developers

docs/manual/using-buildroot-toolchain.txt The toolchain generated by Buildroot is located by default in · line 26

Quoted 8 lines

docs/manual/using-buildroot-toolchain.txt +4 -3

+they want to develop their applications that are not (yet) packaged as
+a Buildroot package.
 
-Once the toolchain is installed to the new location, the user must run
-the +relocate-sdk.sh+ script to make sure all paths are updated with
-the new location.
+Upon extracting the SDK tarball, the user must run the script
++relocate-sdk.sh+, to make sure all paths are updated with the new

Maybe make it explicit that this script is included at the top directory of the
SDK.

docs/manual/using-buildroot-toolchain.txt The toolchain generated by Buildroot is located by default in · line 34

Quoted 8 lines

docs/manual/using-buildroot-toolchain.txt +7 -0

+location.
 
+.Note
+This SDK can not be re-used as an external toolchain, because it
+contains pre-built libraries that could be conflicting with the ones
+packaged in Buildroot (e.g. when an old SDK would be re-used with a
+newer Buildroot version), unless it was built from a configuration
+with no package enabled.

This part really should be a separate patch, it has nothing to do with the SDK
tarball. Also, I disagree with the statement: it is perfectly usable as an external toolchain. The problem is: any library packages included in the SDK are not going to be used by the new Buildroot build. There is no chance of conflict, though, since the external toolchain logic only copies over the required libraries.

Regards,
Arnout

Quoted 1 line

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Trent Piepho

2018-06-11 17:20 UTC #5 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Quoted 17 lines · Arnout Vandecappelle

Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 585
Quoted 12 lines
.PHONY: sdk
Makefile +3 -1
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in 'make sdk' I don't mind that much.

I'm using the SDK with Docker. It can add tar.gz files into the container, but now tar.xz. Having to recompress the archive would be a significant increase in build time.

Quoted 5 lines

Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 587
Quoted 2 lines
Makefile +2 -0
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?

So having extracted an SDK, in an automated CI script, how does one use it? I'll add it to the Docker container:

ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}

Now it needs to be relocated. Something like:

RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh

Is the problem with using this tarball apparent? The dockerfile needs to be hand edited on every single build to use the correct path to the SDK.

I've already added a "sdk-tar" to my external.mk to do this, but used the tarbomb approach. While annoying for those who extract it in their home dir, it's much easier to use in an automated processes since there is no unknown path name component.

It is good if archives that are going to be used in an automated process do not contain unpredictable path names!

Now one might say that I can just make a tarball with a different process. But that would mean creating two SDK tarballs, since there is no way to prep the host dir for making a SDK tar without also making the tarball. That's a significant amount of wasted time.

Thomas Petazzoni

2018-06-11 19:01 UTC #6 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hello,

Quoted 8 lines · Trent Piepho

Is the problem with using this tarball apparent? The dockerfile needs to be hand edited on every single build to use the correct path to the SDK.
I've already added a "sdk-tar" to my external.mk to do this, but used the tarbomb approach. While annoying for those who extract it in their home dir, it's much easier to use in an automated processes since there is no unknown path name component.

You want to use the --strip-components=1 option of tar, which Buildroot is already using when extracting the tarballs of upstream packages into the output/build/<package>-<version>/ directories.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Trent Piepho

2018-06-11 22:10 UTC #7 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Quoted 8 lines · Thomas Petazzoni

Hello,
Quoted 4 lines · Trent Piepho
Is the problem with using this tarball apparent? The dockerfile needs to be hand edited on every single build to use the correct path to the SDK.

Quoted 8 lines

Quoted 4 lines
I've already added a "sdk-tar" to my external.mk to do this, but used the tarbomb approach. While annoying for those who extract it in their home dir, it's much easier to use in an automated processes since there is no unknown path name component.
You want to use the --strip-components=1 option of tar, which Buildroot is already using when extracting the tarballs of upstream packages into the output/build/<package>-<version>/ directories.

Which is great when using GNU tar that has that option. But Docker does not.

I can make a Docker friendly tarball myself, as I do now. But after this patch there is no way to do that without also making an unused non-Docker friendly tarball. Which is pretty costly on VMs with poor io performance.

Some ideas to avoid this:
1. Make tarball generation a new target instead of "sdk".
2. Make tarball generation the "sdk" target but add another target that does what "sdk" used to do.
3. Allow the tarball path prefix to be specified in some way, so that it is easier to inject into an automated process that uses the SDK. E.g., it can be defined as a constant in the defconfig, or passed from a higher level to both buildroot when the SDK is made and to whatever uses the SDK.

Thomas Petazzoni

2018-06-12 09:57 UTC #8 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hello,

Quoted 4 lines · Trent Piepho

Some ideas to avoid this:
1. Make tarball generation a new target instead of "sdk".
2. Make tarball generation the "sdk" target but add another target that does what "sdk" used to do.

For both (1) and (2), the proposal of Yann is to have "make sdk" do what it does today + some new thing.

So there is nothing that prevents to continue using "make sdk" as you're using it today, and ignore the new tarball that is generated. I.e, the change Yann is proposing is backward compatible: "make sdk" is still doing what it used to do.

Quoted 5 lines

3. Allow the tarball path prefix to be specified in some way, so that it is easier to inject into an automated process that uses the SDK. E.g., it can be defined as a constant in the defconfig, or passed from a higher level to both buildroot when the SDK is made and to whatever uses the SDK.

That is another option indeed. Perhaps that makes sense, yes.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Trent Piepho

2018-06-12 17:11 UTC #9 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Quoted 9 lines · Thomas Petazzoni

Hello,
Quoted 4 lines · Trent Piepho
Some ideas to avoid this:
1. Make tarball generation a new target instead of "sdk".
2. Make tarball generation the "sdk" target but add another target that does what "sdk" used to do.

Quoted 7 lines

For both (1) and (2), the proposal of Yann is to have "make sdk" do what it does today + some new thing.
So there is nothing that prevents to continue using "make sdk" as you're using it today, and ignore the new tarball that is generated. I.e, the change Yann is proposing is backward compatible: "make sdk" is still doing what it used to do

But it will be much slower now, as creating and compressing gigabyte tarballs takes a while. If there were a prep-sdk target, then I could use that.

Quoted 7 lines

Quoted 5 lines
3. Allow the tarball path prefix to be specified in some way, so that it is easier to inject into an automated process that uses the SDK. E.g., it can be defined as a constant in the defconfig, or passed from a higher level to both buildroot when the SDK is made and to whatever uses the SDK.
That is another option indeed. Perhaps that makes sense, yes.

I'd find it better, as I have jobs that make different types of SDK. I could give them different prefixes, but not have the prefix change from commit to commit.

Arnout Vandecappelle

2018-06-12 13:30 UTC #10 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Trent Piepho

Quoted 19 lines · Trent Piepho

Quoted 17 lines · Arnout Vandecappelle
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 585
Quoted 12 lines
.PHONY: sdk
Makefile +3 -1
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in 'make sdk' I don't mind that much.

Quoted 3 lines

I'm using the SDK with Docker. It can add tar.gz files into the container, but now tar.xz. Having to recompress the archive would be a significant increase in build time.

Duh. Calling tar inside the container explicitly, rather than letting docker to
the extract directly, isn't too much of a problem either, no?

Quoted 8 lines

Quoted 7 lines
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 587
Quoted 4 lines
Makefile +2 -0
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?

Quoted 6 lines

So having extracted an SDK, in an automated CI script, how does one use it? I'll add it to the Docker container:
ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}

My docker-fu is not too great, but you can do something similar to expose the
tarball directly, and then do

RUN tar -xf ${sdk_file} --strip-components=1

no?

Quoted 3 lines

Now it needs to be relocated. Something like:
RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh

Here you could to

RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh

Quoted 10 lines

Is the problem with using this tarball apparent? The dockerfile needs to be hand edited on every single build to use the correct path to the SDK.
I've already added a "sdk-tar" to my external.mk to do this, but used the tarbomb approach. While annoying for those who extract it in their home dir, it's much easier to use in an automated processes since there is no unknown path name component.

Quoted 7 lines

It is good if archives that are going to be used in an automated process do not contain unpredictable path names!
Now one might say that I can just make a tarball with a different process. But that would mean creating two SDK tarballs, since there is no way to prep the host dir for making a SDK tar without also making the tarball. That's a significant amount of wasted time.

That's what Thomas proposed in his last mail, and I agree it's not great.

That said, if you're building an SDK, it's probably because you're not changing
it too often. As long as you're developing the Buildroot OS itself, there's generally no point generating the SDK. If you really are closely co-developing your application and the Buildroot OS, you're probably better of including the application as a package rather than building it externally.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Trent Piepho

2018-06-12 17:47 UTC #11 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Quoted 21 lines · Arnout Vandecappelle

Quoted 19 lines · Trent Piepho
Quoted 17 lines · Arnout Vandecappelle
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 585
Quoted 12 lines
.PHONY: sdk
Makefile +3 -1
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in 'make sdk' I don't mind that much.

Quoted 6 lines

Quoted 3 lines
I'm using the SDK with Docker. It can add tar.gz files into the container, but now tar.xz. Having to recompress the archive would be a significant increase in build time.
Duh. Calling tar inside the container explicitly, rather than letting docker to
the extract directly, isn't too much of a problem either, no?

I believe COPY + RUN ends up being more layers than just ADD. Though current docker supports xz and bz2 so it would work to use a better compression.

But, in the tests I've done on our vsphere system, using gz was still fastest. I'm more interested in making the CI process faster than in saving disk space.

Quoted 14 lines

Quoted 7 lines
Quoted 6 lines
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 587
Quoted 3 lines
Makefile +2 -0
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?
Quoted 6 lines
So having extracted an SDK, in an automated CI script, how does one use it? I'll add it to the Docker container:
ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}

Quoted 6 lines

My docker-fu is not too great, but you can do something similar to expose the
tarball directly, and then do
RUN tar -xf ${sdk_file} --strip-components=1
no?

It would be more like:

RUN dnf install -y tar
ARG sdk_file
ENV SDK=/mnt/sdk
COPY ${sdk_file} ${SDK}
RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 --owner user:user RUN rm ${SDK}/${sdk_file}

decompress option needs to be specified to tar, based on the compressor used.

This uses more layers.

Quoted 7 lines

Quoted 3 lines
Now it needs to be relocated. Something like:
RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh
Here you could to
RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh

Assumes only one directory will match the pattern. Would also require the user of the SDK to do the same thing, assume the only directory in ${SDK} is the one they want and use a shell wildcard expansion to find it. Which I admit could work, though I think it's a kludge. I needed to do the same thing to find the sysroot location and compiler prefix inside the SDK, and I think that's a kludge too, and wish buildroot put a pointer in a known location in the sdk to where things in unknown locations (sysroot) are. But I haven't gotten around to making a patch to do that yet.

Quoted 5 lines

That said, if you're building an SDK, it's probably because you're not changing
it too often. As long as you're developing the Buildroot OS itself, there's generally no point generating the SDK. If you really are closely co-developing your application and the Buildroot OS, you're probably better of including the application as a package rather than building it externally.

The latter, both the SDK and the applications using the SDK change very
frequently. Which is why I'm so concerned about speed.

While the applications are buildroot packages as well, there are downsides to doing development that way. So they exist as stand alone packages that can be compiled outside of buildroot. No different than any other package in buildroot in that regard.

But how to build a package stand alone in a CI process, in an environment than can build it correctly, and also closely tracks the image buildroot will make for the final product? Use the buildroot SDK! So that's what I'm using it for.

Stefan Becker

2018-06-12 18:07 UTC #12 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Trent, All,

Quoted 1 line · Trent Piepho

I didn't want to write anything to this thread at first, but then I saw that what Trent wrote matches 100% what I was thinking when looking at the change.

I'm very disappointed with the attitude displayed by the buildroot developers in this thread. Trent has valid points and IMHO has made sensible suggestions how to address those. But all I see from the project developers is the typical knee-jerk reaction to SW build or SW integration issues, i.e. "buildroot is always correct, your system is doing it wrong".

Can we please all take a step backward and agree that there is a whole universe outside buildroot with different (integration) requirements? For the change under review I would suggest:

(A) there are existing users of the "sdk" target and this change would modify its behaviour in an undesired/surprising fashion.

(B) instead this change should introduce e.g. the target "sdk-tarball", which depends on "sdk" and generates the new tarball format. It is then up to the users of "sdk" to decide if they can use the new build artifact or not.

(C) having a build artefact that changes the name in every build is bad for automatic build setups. So (B) must offer a way for automatic build scripts to set the generated tarball name. If the user can use the new tarball format in his integration, then he can switch from "sdk" to "sdk-tarball" with the existing name (and possibly using --strip-components in downstream build scripts).

Just my ?0.02...

Stefan

Thomas Petazzoni

2018-06-12 19:01 UTC #13 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hello,

Quoted 6 lines · Stefan Becker

I didn't want to write anything to this thread at first, but then I saw that what Trent wrote matches 100% what I was thinking when looking at the change.
I'm very disappointed with the attitude displayed by the buildroot developers in this thread.

I believe you are also very abrupt with this statement. For example, I myself asked as a reply to Yann's proposal if there wasn't a backward compatibility issue.

Quoted 8 lines

Trent has valid points and IMHO has made
sensible suggestions how to address those. But all I see from the project developers is the typical knee-jerk reaction to SW build or SW integration issues, i.e. "buildroot is always correct, your system is doing it wrong".
Can we please all take a step backward and agree that there is a whole universe outside buildroot with different (integration) requirements?

For the record, I am using Buildroot for toolchains.bootlin.com, which is running Buildroot in a CI system to produce toolchains, and it uses "make sdk". So I am also affected by behavior changes in "make sdk".

Quoted 9 lines

For the change under review I would suggest:
(A) there are existing users of the "sdk" target and this change would modify its behaviour in an undesired/surprising fashion.
(B) instead this change should introduce e.g. the target "sdk-tarball", which depends on "sdk" and generates the new tarball format. It is then up to the users of "sdk" to decide if they can use the new build artifact or not.

As said above, when I first looked at Yann's proposal, I thought there could be a backward compatibility problem. But in fact, what the new "make sdk" does is a strict super-set of what the old "make sdk" was doing. So if you're not happy with the new "make sdk" behavior, this shouldn't really matter: it still does the exact same thing as before, it just *additionally* provides a convenience tarball.

However, one could argue that despite the behavior being preserved, it still changes a few things:

- More time required due to generating the tarball.

- More disk space being consumed, especially with the tarball name changing pretty frequently due to containing the Git commit id.

Quoted 2 lines

(C) having a build artefact that changes the name in every build is bad for automatic build setups.

True.

Quoted 5 lines

So (B) must offer a way for automatic
build scripts to set the generated tarball name. If the user can use the new tarball format in his integration, then he can switch from "sdk" to "sdk-tarball" with the existing name (and possibly using --strip-components in downstream build scripts).

But where did we say that we would not accept something that allows to customize the output tarball name ? Where did we say that using a separate target was not acceptable ?

I think you're really rushing way too fast to a conclusion that we're not listening. Yes, we are asking questions about the use cases, yes we are trying to understand such use cases, and see how they fit in the big picture.

I believe one of the reason we all like and use Buildroot is because of its simplicity. And simplicity is a very difficult thing to preserve:
the tendency of all projects is to become more and more complicated, to address more and more special use cases. So it is quite normal that we question the use cases, try to understand them, and see which part of those use cases should be solved by Buildroot itself, and which use cases should be solved by external tooling. There is always going to be a tension between people wanted to solve more use cases inside Buildroot and therefore increasing its complexity, and the wish to keep Buildroot simple.

Let's try to keep the discussion open-minded, please :-)

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Stefan Becker

2018-06-12 19:25 UTC #14 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Thomas, All,

Quoted 3 lines · Thomas Petazzoni

I believe one of the reason we all like and use Buildroot is because of its simplicity. And simplicity is a very difficult thing to preserve:

IMHO this change is the very opposite of "preserving simplicity". It adds unwanted things to the buildroot build. The unwanted things might even be unusable for existing users of "sdk".

Please don't get me wrong. I'm not against the SDK tarball itself.

But I'm against building something for naught. If the tarball turns out to be unusable I get slower buildroot builds. Which in turns will force me to extract what "original sdk without tarball" was doing out of buildroot makefile and hard-code it into my build script. And then I can remove "sdk" from the buildroot make command. I fail to see how that is simpler.

If you don't offer a way to specify the generated tarball name, fine. Then I won't use sdk-tarball at all, even if the tarball itself would be usable in our downstream integration. Or I could come up with a way to "guess" the build artefact name in my build script and symlink it to the wanted build artefact name (again, I fail to see how this is simpler).

I don't understand why "sdk-tarball: sdk" is not an acceptable solution. It would be much simpler.

Regards, Stefan

PS: just to be complete, this is what my current build script does after "make all sdk" has been completed:

    (cd output/host && tar -cvJf ../host.tar.xz *)

The collected build artefact is "host.tar.xz". AFAIR that name can't be changed dynamically without breaking the automatic downstream build flow in Jenkins.

Andreas Naumann

2018-06-13 08:32 UTC #15 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi,

on a general note, I do find that buildroot does an outstanding job in providing a lot of simple to use functionality while keeping it within a well defined scope!
To help achieve that for SDK generation, I'll describe my use cases/preferences.

Am 12.06.2018 um 21:25 schrieb Stefan Becker:

Quoted 2 lines

Hi Thomas, All,

<snip>

Quoted 7 lines

But I'm against building something for naught. If the tarball turns out to be unusable I get slower buildroot builds. Which in turns will force me to extract what "original sdk without tarball" was doing out of buildroot makefile and hard-code it into my build script. And then I can remove "sdk" from the buildroot make command. I fail to see how that is simpler.

I also prefer speed over storage size (gz vs. xz) and definitely want to spend both only for building what I need.

Quoted 7 lines

If you don't offer a way to specify the generated tarball name, fine. Then I won't use sdk-tarball at all, even if the tarball itself would be usable in our downstream integration. Or I could come up with a way to "guess" the build artefact name in my build script and symlink it to the wanted build artefact name (again, I fail to see how this is simpler).

Even though the sdk target has been around for a while, I still use custom steps to a) build the SDK on a absolute path to avoid relocation needs and b) tar it with a project-specific name.

So if b) is provided by buildroot I'd be very interested to finally give up my addon mess (modifiying HOST_DIR before the build) and switch to buildroots way.
However, for maintenance reasons I want to support existing project setups, which would only be "simple" if there was an easy way to
- run the relocation previous to packing
- adjust the location to be packed accordingly
- and define the name of the tarball with a set of "beyond buildroot" Project-Variables

I guess one way to achieve that would be baking this sort of customization into the defconfig. However, that makes usage of savedefconfig more or less useless to me because it leads to constantly changing defconfigs. (Which is undesirable as my messing with HOST_DIR has showed me).
Allowing environment variables to control the process would work too, but I dont know if thats considered "good practise" or "simple" either.

best regards,
Andreas

Arnout Vandecappelle

2018-06-13 10:12 UTC #16 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Andreas,

Quoted 5 lines · Andreas Naumann

Hi,
on a general note, I do find that buildroot does an outstanding job in providing a lot of simple to use functionality while keeping it within a well defined scope! To help achieve that for SDK generation, I'll describe my use cases/preferences.

Thanks for your input!

Quoted 12 lines

Am 12.06.2018 um 21:25 schrieb Stefan Becker:
Quoted 2 lines
Hi Thomas, All,
<snip>
Quoted 7 lines
But I'm against building something for naught. If the tarball turns out to be unusable I get slower buildroot builds. Which in turns will force me to extract what "original sdk without tarball" was doing out of buildroot makefile and hard-code it into my build script. And then I can remove "sdk" from the buildroot make command. I fail to see how that is simpler.

Quoted 2 lines

I also prefer speed over storage size (gz vs. xz) and definitely want to spend both only for building what I need.

Yep, it's pretty clear that my .xz idea doesn't get much traction :-)

Quoted 7 lines

Quoted 6 lines
If you don't offer a way to specify the generated tarball name, fine. Then I won't use sdk-tarball at all, even if the tarball itself would be usable in our downstream integration. Or I could come up with a way to "guess" the build artefact name in my build script and symlink it to the wanted build artefact name (again, I fail to see how this is simpler).

Quoted 6 lines

Even though the sdk target has been around for a while, I still use custom steps to a) build the SDK on a absolute path to avoid relocation needs and b) tar it with a project-specific name.
So if b) is provided by buildroot I'd be very interested to finally give up my addon mess (modifiying HOST_DIR before the build and switch to buildroots way.

Why is specifying a BR2_HOST_DIR on the make command line messy? It is in fact
Buildroot's way :-)

Quoted 6 lines

However, for maintenance reasons I want to support existing project setups, which would only be "simple" if there was an easy way to
- run the relocation previous to packing
- adjust the location to be packed accordingly
- and define the name of the tarball with a set of "beyond buildroot" Project-Variables

This starts to sound complicated... I would say, if you're content with a
non-relocatable SDK, setting BR2_HOST_DIR is the way to go.

Perhaps we could consider a post-sdk script. That would get called with the
relevant variables so you could set up all the symlinks you want to find the cross-compile tuple etc. Although, really, calling 'make printvars VARS=GNU_TARGET_NAME' from your CI script doesn't sound like the end of the world to me.

Quoted 6 lines

I guess one way to achieve that would be baking this sort of customization into the defconfig. However, that makes usage of savedefconfig more or less useless to me because it leads to constantly changing defconfigs. (Which is undesirable as my messing with HOST_DIR has showed me).
Allowing environment variables to control the process would work too, but I dont know if thats considered "good practise" or "simple" either.

I don't really like putting stuff in environment variables. I'd rather have it
in local.mk variables then (which automatically means that they can also be set on the command line). But even so, it smells like unneeded complexity to me.

Summarizing where we are in this thread now, I think:

1. The tarball generation should be separated from the sdk generation itself, with e.g.

sdk-tar: sdk

	@$(TAR) ...

2. The tarball should really be created as a .gz, not as an .xz or anything else.

3. In addition to the unique name, there should be a symlink/hardlink with a constant name.

Note that this doesn't solve the potential issue that the images/ directory is
constantly growing. So perhaps we should just give up on the unique name, and instead only generate a constant name, like we currently do for all our build artefacts.

With all this in mind, I've marked the patch as Changes Requested.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Andreas Naumann

2018-06-13 15:46 UTC #17 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Arnout,

Am 13.06.2018 um 12:12 schrieb Arnout Vandecappelle:

Quoted 11 lines

Hi Andreas,
Quoted 7 lines
Quoted 6 lines
If you don't offer a way to specify the generated tarball name, fine. Then I won't use sdk-tarball at all, even if the tarball itself would be usable in our downstream integration. Or I could come up with a way to "guess" the build artefact name in my build script and symlink it to the wanted build artefact name (again, I fail to see how this is simpler).

Quoted 8 lines

Quoted 6 lines
Even though the sdk target has been around for a while, I still use custom steps to a) build the SDK on a absolute path to avoid relocation needs and b) tar it with a project-specific name.
So if b) is provided by buildroot I'd be very interested to finally give up my addon mess (modifiying HOST_DIR before the build and switch to buildroots way.
Why is specifying a BR2_HOST_DIR on the make command line messy? It is in fact
Buildroot's way :-)

Specifying on command line would have been nice but remember this being ignored in favour of the .config setting. So I always modified .config before the build, which I do regard as somewhat messy.

Quoted 10 lines

Quoted 7 lines
However, for maintenance reasons I want to support existing project setups, which would only be "simple" if there was an easy way to
- run the relocation previous to packing
- adjust the location to be packed accordingly
- and define the name of the tarball with a set of "beyond buildroot" Project-Variables
This starts to sound complicated... I would say, if you're content with a
non-relocatable SDK, setting BR2_HOST_DIR is the way to go.

I just tried again to set BR_HOST_DIR via environment but it doesnt propagate. It would work with make -e ... but I fear hard to discover side effects doing that. So I'd rather go with the relocation in the future.

Quoted 6 lines

Perhaps we could consider a post-sdk script. That would get called with the
relevant variables so you could set up all the symlinks you want to find the cross-compile tuple etc. Although, really, calling 'make printvars VARS=GNU_TARGET_NAME' from your CI script doesn't sound like the end of the world to me.

Actually, I'm using the printvars solution right now, no problem.

Quoted 9 lines

Quoted 8 lines
I guess one way to achieve that would be baking this sort of customization into the defconfig. However, that makes usage of savedefconfig more or less useless to me because it leads to constantly changing defconfigs. (Which is undesirable as my messing with HOST_DIR has showed me).
Allowing environment variables to control the process would work too, but I dont know if thats considered "good practise" or "simple" either.

Quoted 3 lines

I don't really like putting stuff in environment variables. I'd rather have it
in local.mk variables then (which automatically means that they can also be set on the command line). But even so, it smells like unneeded complexity to me.

Just as I fail in overwriting BR2_HOST_DIR from commandline I fail in setting BR2_PACKAGE_OVERRIDE_FILE to something else than defined in .config. I guess it would be really helpful if you could give me an example on how to do that.

Quoted 11 lines

Summarizing where we are in this thread now, I think:
1. The tarball generation should be separated from the sdk generation itself, with e.g.
sdk-tar: sdk
	@$(TAR) ...
2. The tarball should really be created as a .gz, not as an .xz or anything else.

Quoted 7 lines

3. In addition to the unique name, there should be a symlink/hardlink with a constant name.
Note that this doesn't solve the potential issue that the images/ directory is
constantly growing. So perhaps we should just give up on the unique name, and instead only generate a constant name, like we currently do for all our build artefacts.

Since the SDK tarball is something that is "thrown around" between developers on it's own, it might make sense to somehow being able to distinguish it?
For my case, I include a project version in the name, but no buildnumber or SHA. That way it just changes between project releases. Also, I have a separate sdk/ folder for it, because I dont like mixing target with host stuff. (and it can be cleaned independently).

So while it's easy enough to rename the tarball to an appropriate name after creating it with make sdk-tar, the contained top-level folder should somehow match this name right?

regards,
Andreas

Quoted 7 lines

With all this in mind, I've marked the patch as Changes Requested.
Regards,
Arnout

Arnout Vandecappelle

2018-06-13 09:47 UTC #18 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Thomas Petazzoni

Quoted 8 lines · Thomas Petazzoni

Hello,
Quoted 4 lines · Stefan Becker
I didn't want to write anything to this thread at first, but then I saw that what Trent wrote matches 100% what I was thinking when looking at the change.

Quoted 6 lines

Quoted 2 lines
I'm very disappointed with the attitude displayed by the buildroot developers in this thread.
I believe you are also very abrupt with this statement. For example, I myself asked as a reply to Yann's proposal if there wasn't a backward compatibility issue.

Thomas, you can't deny that, whatever your (and my) intention was in this
thread, it clearly has the effect on (some) people that they think "the buildroot developers" are not paying attention to their needs.

I don't know why that is or what we can do about it (without investing still
more time in composing our mails), but the least we can do is acknowledge this situation.

Regards,
Arnout

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Thomas Petazzoni

2018-06-13 09:57 UTC #19 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Arnout,

Quoted 11 lines · Arnout Vandecappelle

Quoted 10 lines
Quoted 6 lines
I didn't want to write anything to this thread at first, but then I saw that what Trent wrote matches 100% what I was thinking when looking at the change.
I'm very disappointed with the attitude displayed by the buildroot developers in this thread.
I believe you are also very abrupt with this statement. For example, I myself asked as a reply to Yann's proposal if there wasn't a backward compatibility issue.

Quoted 7 lines

Thomas, you can't deny that, whatever your (and my) intention was in this
thread, it clearly has the effect on (some) people that they think "the buildroot developers" are not paying attention to their needs.
I don't know why that is or what we can do about it (without investing still
more time in composing our mails), but the least we can do is acknowledge this situation.

I will have to read this specific thread again to see what could communicate this feeling.

However, generally speaking, I do agree that we tend to push back on people proposing to add more complexity to Buildroot, for the reasons I've highlighted before. And I definitely understand that such regular push back, or sometimes slowness to take a decision on specific topics can be frustrating. I do completely acknowledge that.

However, this push back and/or slowness on topics that suggest changing some core aspect of Buildroot or its complexity level is actually more a feature than a bug. It is what allows to preserve Buildroot core principles on the long run. The contributors pushing for a little bit more complexity here only see a little bit more complexity in one specific place. People who have been around for a much longer period of time (I'm "celebrating" this year my 10th year contributing to Buildroot, and Peter has been contributing for longer than that, etc.) have a different perspective, as they see the overall complexity increase, not just the specific complexity increase caused by one patch.

But, yes I acknowledge that we are reluctant to change, and sometimes need a lot of convincing arguments to change our minds.

Best regards,

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Stefan Becker

2018-06-13 10:03 UTC #20 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Thomas,

Quoted 4 lines · Thomas Petazzoni

However, generally speaking, I do agree that we tend to push back on people proposing to add more complexity to Buildroot, for the reasons I've highlighted before.

Wouldn't it then be much simpler to reject this change, i.e. *not* adding tarball generation to buildroot SW build at all?

Regards, Stefan

Thomas Petazzoni

2018-06-13 11:58 UTC #21 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hello,

Quoted 9 lines · Stefan Becker

Quoted 4 lines · Thomas Petazzoni
However, generally speaking, I do agree that we tend to push back on people proposing to add more complexity to Buildroot, for the reasons I've highlighted before.
Wouldn't it then be much simpler to reject this change, i.e. *not* adding tarball generation to buildroot SW build at all?

That's one option. Another option is to:

(1) Use another make target, such as sdk-tarball, as you suggested

(2) Use a stable name for the output, like output/images/buildroot-sdk.tar.gz. Indeed for no other Buildroot output we include the Git commit ID or the target tuple in the name of the file being generated.

Thomas

Signature · 3 lines

Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com

Peter Korsgaard

2018-06-15 18:27 UTC #22 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Thomas Petazzoni

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:

Hi,

Quoted 16 lines

I think you're really rushing way too fast to a conclusion that we're not listening. Yes, we are asking questions about the use cases, yes we are trying to understand such use cases, and see how they fit in the big picture.
I believe one of the reason we all like and use Buildroot is because of its simplicity. And simplicity is a very difficult thing to preserve:
the tendency of all projects is to become more and more complicated, to address more and more special use cases. So it is quite normal that we question the use cases, try to understand them, and see which part of those use cases should be solved by Buildroot itself, and which use cases should be solved by external tooling. There is always going to be a tension between people wanted to solve more use cases inside Buildroot and therefore increasing its complexity, and the wish to keep Buildroot simple.

Quoted 1 line

Let's try to keep the discussion open-minded, please :-)

Completely agreed. Thank you for describing it so clearly!

Signature · 1 line

Bye, Peter Korsgaard

Arnout Vandecappelle

2018-06-13 07:46 UTC #23 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Trent Piepho

Quoted 22 lines · Trent Piepho

Quoted 21 lines · Arnout Vandecappelle
Quoted 19 lines · Trent Piepho
Quoted 17 lines · Arnout Vandecappelle
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 585
Quoted 12 lines
.PHONY: sdk
Makefile +3 -1
-sdk: world
+sdk: world $(BR2_TAR_HOST_DEPENDENCY)
 	@$(call MESSAGE,"Rendering the SDK relocatable")
 	$(TOPDIR)/support/scripts/fix-rpath host
 	$(TOPDIR)/support/scripts/fix-rpath staging
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
+	$(Q)mkdir -p $(BINARIES_DIR)
+	$(TAR) czf $(BINARIES_DIR)/buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL).tar.gz \
Wouldn't it make more sense to make a .xz, or perhaps a .bz2? Although, that
probably gives a significant hit to the build time. But since it's explicit in 'make sdk' I don't mind that much.

Quoted 8 lines

Quoted 6 lines
Quoted 3 lines
I'm using the SDK with Docker. It can add tar.gz files into the container, but now tar.xz. Having to recompress the archive would be a significant increase in build time.
Duh. Calling tar inside the container explicitly, rather than letting docker to
the extract directly, isn't too much of a problem either, no?
I believe COPY + RUN ends up being more layers than just ADD.

You can collapse the RUN commands, see below.

Quoted 7 lines

Though
current docker supports xz and bz2 so it would work to use a better compression.
But, in the tests I've done on our vsphere system, using gz was still fastest. I'm more interested in making the CI process faster than in saving disk space.

Good point, gz is much faster than xz, both for compressing and decompressing.
So it indeed makes sense to just keep it as a .gz. IIRC lz4 would be even better, but that requires installing lz4...

Quoted 16 lines

Quoted 16 lines
Quoted 6 lines
Quoted 5 lines
Makefile prepare: $(BUILD_DIR)/buildroot-config/auto.conf · line 587
Quoted 2 lines
Makefile +2 -0
+		-C $(HOST_DIR) \
+		--transform='s#^\.#buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)#' \
Perhaps move that "buildroot-sdk.$(GNU_TARGET_NAME)-$(BR2_VERSION_FULL)" into a
variable?
Quoted 6 lines
So having extracted an SDK, in an automated CI script, how does one use it? I'll add it to the Docker container:
ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}
My docker-fu is not too great, but you can do something similar to expose the
tarball directly, and then do

Quoted 7 lines

Quoted 3 lines
RUN tar -xf ${sdk_file} --strip-components=1
no?
It would be more like:
RUN dnf install -y tar

You could just have tar already in your base image, of course...

Quoted 3 lines

ARG sdk_file
ENV SDK=/mnt/sdk
COPY ${sdk_file} ${SDK}

Isn't there something that "links" the file rather than copying it into the image?

Quoted 2 lines

RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 --owner user:user RUN rm ${SDK}/${sdk_file}

I think this will create 3 layers: one for the COPY and one for each RUN. You
can reduce it to two with:

RUN tar -xf ${SDK}/${sdk_file} -C ${SDK} --strip-components=1 &&

    rm ${SDK}/${sdk_file}

Quoted 8 lines

decompress option needs to be specified to tar, based on the compressor used.
This uses more layers.
Quoted 2 lines
Quoted 2 lines
Now it needs to be relocated. Something like:

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh
Here you could to
RUN cd ${SDK}/buildroot-sdk.*/ && ./relocate-sdk.sh
Assumes only one directory will match the pattern.

If you re-create the docker image every time (like is done above), only one
will match.

Quoted 4 lines

Would also require
the user of the SDK to do the same thing, assume the only directory in ${SDK} is the one they want and use a shell wildcard expansion to find it.

Indeed, not so nice. So --strip-components is better :-)

Quoted 5 lines

Which I admit could work, though I think it's a kludge. I needed
to do the same thing to find the sysroot location and compiler prefix inside the SDK, and I think that's a kludge too, and wish buildroot put a pointer in a known location in the sdk to where things in unknown locations (sysroot) are.

Good point. In the output/ directory, we have ${STAGING_DIR} which does
something like that (points to the sysroot, still doesn't explicitly say what the CROSS_COMPILE prefix is but you could parse it out of the symlink). So we could add the staging symlink to the tarball as well, and while we're at it also add a host symlink. Although actually, no, I think that just looks ugly. There must be a better way.

Quoted 17 lines

But I haven't gotten around to making a patch
to do that yet.
Quoted 5 lines
That said, if you're building an SDK, it's probably because you're not changing
it too often. As long as you're developing the Buildroot OS itself, there's generally no point generating the SDK. If you really are closely co-developing your application and the Buildroot OS, you're probably better of including the application as a package rather than building it externally.
The latter, both the SDK and the applications using the SDK change very
frequently. Which is why I'm so concerned about speed.
While the applications are buildroot packages as well, there are downsides to doing development that way. So they exist as stand alone packages that can be compiled outside of buildroot. No different than any other package in buildroot in that regard.

Quoted 4 lines

But how to build a package stand alone in a CI process, in an environment than can build it correctly, and also closely tracks the image buildroot will make for the final product? Use the buildroot SDK! So that's what I'm using it for.

Very good point indeed. I'm glad I asked :-)

Is the idea that you have one buildroot build, it creates a docker image, and
that docker image is then used for several application builds? Sounds like a pretty neat setup.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Stefan Becker

2018-06-13 07:59 UTC #24 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

Hi Arnout, All,

Quoted 4 lines · Arnout Vandecappelle

Is the idea that you have one buildroot build, it creates a docker image, and
that docker image is then used for several application builds? Sounds like a pretty neat setup.

Docker isn't really a requirement for such an approach, but it can make life much simpler in a fractured IT environment.

We use the sdk build artefact from a common buildroot automated build in our CI system without using docker. Having fast CI builds (< 5mins) for our own components/device images and that no developer ever has to run slow buildroot builds are a big plus :-) As there is only one buildroot build makes tracking buildroot upstream much easier.

Regards, Stefan

Peter Korsgaard

2018-06-15 18:12 UTC #25 lore raw

PATCH 1/1 core/sdk: generate the SDK tarball ourselves

In reply to Trent Piepho

Quoted 1 line

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Trent" == Trent Piepho <[email protected]> writes:

Hi,

Quoted 7 lines

So having extracted an SDK, in an automated CI script, how does one use it? I'll add it to the Docker container:
ARG sdk_file
ENV SDK=/mnt/sdk
ADD --chown=user:user ${sdk_file} ${SDK}

Quoted 7 lines

Now it needs to be relocated. Something like:
RUN cd ${SDK}/buildroot-sdk.arm-buildroot-linux-gnueabihf-2018.02-00035-ge588bdd3e8 && ./relocate-sdk.sh
Is the problem with using this tarball apparent? The dockerfile needs to be hand edited on every single build to use the correct path to the SDK.

Why not use a wildcard? E.G. cd ${SDK}/buildroot-sdk* && ./relocate-sdk.sh?

Nothing else should match the wildcard inside the container.

Signature · 1 line

Bye, Peter Korsgaard