core/sdk: generate the SDK tarball ourselves
PATCH v2 1/1 core/sdk: generate the SDK tarball ourselves
Currently, the wording in the manual instructs the user to generate a tarball from "the contents of the +output/host+ directory".
This is pretty confusing, because taken literally, this would amount to running a command like:
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when extracted, it creates multiple files in the current directory.
What one really wants to do, is create a tarball of the host/ directory, with something like:
tar cf my-sdk.tar -C output host/
However, this is not much better, because the top-most directory would have a very common name, host/, which is pretty easy to get conflict with when it gets extracted.
So, we fix that mess by giving the top-most directory a recognisable name, based on the target tuple and the Buildroot version, which we also use as the name of the archive (suffixed with the usual +.tar.gz+.) We offer the user the possibility to override that default by specifying the +BR2_SDK_PREFIX+ variable on the command line.
Since this is an output file, we place it in the images/ directory.
As some users expressed a very strong feeling that they do not want to generate a tarball at all, and that doing so would badly hurt their workflows [1], we actually prepare the SDK as was previously done, but under the new, intermediate rule 'prepare-sdk'. The existing 'sdk' rule obviously depend on that before generating the tarball.
We choose to make the existing rule to generate the tarball, and introduce a new rule to just prepare the SDK, rather than keep the existing rule as-is and introduce a new one to generate the tarball, because it makes sense to have the simplest rule do the correct thing, leaving advanced, power users use the longest command. If someone already had a wrapper that called 'sdk' and expected just the host directory to be prepared, then this is not broken; it just takes a bit longer (gzip is pretty fast).
Update the manual accordingly.
[0] https://en.wikipedia.org/wiki/Tar_(computing)#Tarbomb
[1] http://lists.busybox.net/pipermail/buildroot/2018-June/thread.html#223377
and some messages in the ensuing thread...
---
Changes v1 -> v2:
- drop the part of the manual stating this can't be re-used as a pre-built toolchain (Arnout)
- make the top-level directory and tarball name configurable (Trent, Thomas)
- change the default name (Arnout)
- allow just preparing the SDK (Stefan, Trent)
- typoes (Thomas)
--- Makefile | 13 +++++++++++-- docs/manual/using-buildroot-toolchain.txt | 26 +++++++++++++++++--------- 2 files changed, 28 insertions(+), 11 deletions(-)
Makefile +11 -2
@@ -573,8 +573,8 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf .PHONY: world world: target-post-image -.PHONY: sdk -sdk: world +.PHONY: prepare-sdk +prepare-sdk: world @$(call MESSAGE,"Rendering the SDK relocatable") $(TOPDIR)/support/scripts/fix-rpath host $(TOPDIR)/support/scripts/fix-rpath staging
@@ -582,6 +582,15 @@ sdk: world mkdir -p $(HOST_DIR)/share/buildroot echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location +BR2_SDK_PREFIX ?= $(GNU_TARGET_NAME)_sdk-buildroot-$(BR2_VERSION_FULL) +.PHONY: sdk +sdk: $(BR2_TAR_HOST_DEPENDENCY) + $(if $(BR2_SDK_PREFIX),,$(error BR2_SDK_PREFIX must not be empty)) + $(Q)mkdir -p $(BINARIES_DIR) + $(TAR) czf "$(BINARIES_DIR)/$(BR2_SDK_PREFIX).tar.gz" \ + --transform='s#^\.#$(BR2_SDK_PREFIX)/#' \ + -C $(HOST_DIR) "." + # Populating the staging with the base directories is handled by the skeleton package $(STAGING_DIR): @mkdir -p $(STAGING_DIR)
docs/manual/using-buildroot-toolchain.txt +17 -9
@@ -12,15 +12,23 @@ The toolchain generated by Buildroot is located by default in +output/host/bin/+ to your PATH environment variable and then to use +ARCH-linux-gcc+, +ARCH-linux-objdump+, +ARCH-linux-ld+, etc. -It is possible to relocate the toolchain, this allows to distribute -the toolchain to other developers to build applications for your -target. To achieve this: +Alternatively, Buildroot can also export the toolchain and the development +files of all selected packages, as an SDK, by running the command ++make sdk+. This generates a tarball of the content of the host directory ++output/host/+, named +<TARGET-TUPLE>_sdk-buildroot-<BR-VERSION>.tar.gz+ +(which can be overriden by setting the environment variable ++BR2_SDK_PREFIX+) and located in the output directory +output/images/+. -* run +make sdk+, which prepares the toolchain to be relocatable; -* tarball the contents of the +output/host+ directory; -* distribute the resulting tarball. +This tarball can then be distributed to application developers, when +they want to develop their applications that are not (yet) packaged as +a Buildroot package. -Once the toolchain is installed to the new location, the user must run -the +relocate-sdk.sh+ script to make sure all paths are updated with -the new location. +Upon extracting the SDK tarball, the user must run the script ++relocate-sdk.sh+ (located at the top directory of the SDK), to make +sure all paths are updated with the new location. +Alternatively, if you just want to prepare the SDK without generating +the tarball (e.g. because you will just be moving the +host+ directory, +or will be generating the tarball on your own), Buildroot also allows +you to just prepare the SDK with +make prepare-sdk+ without actually +generating a tarball.
Signature · 1 line
2.14.1
PATCH v2 1/1 core/sdk: generate the SDK tarball ourselves
Hi Yann, All,
Quoted 4 lines · Yann E. MORIN
This is pretty confusing, because taken literally, this would amount to running a command like:
Quoted 8 lines
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when extracted, it creates multiple files in the current directory.
...
So, we fix that mess by giving the top-most directory a recognisable name, based on the target tuple and the Buildroot version, which we also use as the name of the archive (suffixed with the usual +.tar.gz+.)
Sorry, but I can neither follow nor agree with this logic. The above tar command line is IMHO *exactly* how the SDK tarball should be generated, because it leaves the choice up to the user of the tarball *how* to use/integrate it downstream.
Anything else, especially directory names that change with every build, only creates unnecessary hoops the user of the SDK has to jump through every time.
Quoted 2 lines
We offer the user the possibility to override that default by specifying the +BR2_SDK_PREFIX+ variable on the command line.
Does "export BR2_SDK_PREFIX=" make sure that no prefix is used, i.e. you get the same output as the tar command above? If the answer is YES then this should be mentioned in the manual.
Does "export BR2_SDK_PREFIX=" disable the variable part of the tarball name? Judging from the diffs the answer is likely NO, i.e. you can't disable the version part of the tarball name.
Quoted 5 lines
We choose to make the existing rule to generate the tarball, and introduce a new rule to just prepare the SDK, rather than keep the existing rule as-is and introduce a new one to generate the tarball, because it makes sense to have the simplest rule do the correct thing, leaving advanced, power users use the longest command.
I'm not really happy with this, but at least I now have the choice and just need to update one line in the build script to disable all this useless stuff.
Quoted 4 lines
If someone
already had a wrapper that called 'sdk' and expected just the host directory to be prepared, then this is not broken; it just takes a bit longer (gzip is pretty fast).
I would be careful with such statements. Based on my experience with our automated build system generating the tarballs from output/target & output/host takes a non-trivial amount of the total build time.
Regards, Stefan
PATCH v2 1/1 core/sdk: generate the SDK tarball ourselves
Stefan, All,
On 2018-06-30 19:03 +0300, Stefan Becker spake thusly:
Quoted 14 lines · Yann E. MORIN
Quoted 4 lines
This is pretty confusing, because taken literally, this would amount to running a command like:
Quoted 8 lines
tar cf my-sdk.tar -C output/host/ .
This creates a tarbomb [0], which is very bad practice, because when extracted, it creates multiple files in the current directory.
...
So, we fix that mess by giving the top-most directory a recognisable name, based on the target tuple and the Buildroot version, which we also use as the name of the archive (suffixed with the usual +.tar.gz+.)
Quoted 8 lines
Sorry, but I can neither follow nor agree with this logic. The above tar command line is IMHO *exactly* how the SDK tarball should be generated, because it leaves the choice up to the user of the tarball *how* to use/integrate it downstream.
Anything else, especially directory names that change with every build, only creates unnecessary hoops the user of the SDK has to jump through every time.
Well, on my side, I *do* want an artefact that is versioned, so that I am sure users do not extract a new one over a previous one, so that they can work on different versions at the same time.
Versionned artefacts are a requirement for some people.
And I do want that version to represent something that I can actually identify. By default, I *do* want it to identify the version of Buildroot that was used to generate that SDK. But I understand that someople may want another identifier, hence the variable to customise it.
Quoted 6 lines
Quoted 2 lines
We offer the user the possibility to override that default by specifying the +BR2_SDK_PREFIX+ variable on the command line.
Does "export BR2_SDK_PREFIX=" make sure that no prefix is used, i.e. you get the same output as the tar command above? If the answer is YES then this should be mentioned in the manual.
Using an empty value is indeed not supported. This should indeed be documented in the manual.
The rationale behind that limitation, is two fold. First, if you don;t want a prefix at all, then you can just generate the tarball easily, directly from output/host/. Second, it is not trivial to ensure that the prefix is really interpreted as directory component, because we append '/' in the trasnform regexp; with an empty value, we would not have to append it.
Quoted 3 lines
Does "export BR2_SDK_PREFIX=" disable the variable part of the tarball name? Judging from the diffs the answer is likely NO, i.e. you can't disable the version part of the tarball name.
Well, again, if you don't like the tarball that is generated, you can just use 'prepare-sdk' and get back to your previous situation.
Quoted 9 lines
Quoted 5 lines
We choose to make the existing rule to generate the tarball, and introduce a new rule to just prepare the SDK, rather than keep the existing rule as-is and introduce a new one to generate the tarball, because it makes sense to have the simplest rule do the correct thing, leaving advanced, power users use the longest command.
I'm not really happy with this, but at least I now have the choice and just need to update one line in the build script to disable all this useless stuff.
Well, I prefer the rule of least surprise, so that users by default do not have surprises when they use (i.e. extract) the generated tarball. Getting a tarbomb is very bad for most users. And I prefer to adapt my scripts to accomodate with that (tar's --strip-components=1, as already mentionned), rather than trash my home dir, or any other precious location.
Quoted 8 lines
Quoted 4 lines
If someone
already had a wrapper that called 'sdk' and expected just the host directory to be prepared, then this is not broken; it just takes a bit longer (gzip is pretty fast).I would be careful with such statements. Based on my experience with our automated build system generating the tarballs from output/target & output/host takes a non-trivial amount of the total build time.
It does, yes, but compare the time it takes to generate a multi-gigabyte archive, with the actual time it takes to do the build that lead to that multi-gigabyte archive.
But now, I took *all* the feedback for the first iteration, to provide a situation that works for everyone. If the discussion on this new patch makes it that the maintainer do not want it in the end, then fine.
Quoted 1 line
Regards, Stefan
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 1/1 core/sdk: generate the SDK tarball ourselves
Hello,
Quoted 13 lines · Yann E. MORIN
Quoted 8 lines
Sorry, but I can neither follow nor agree with this logic. The above tar command line is IMHO *exactly* how the SDK tarball should be generated, because it leaves the choice up to the user of the tarball *how* to use/integrate it downstream.
Anything else, especially directory names that change with every build, only creates unnecessary hoops the user of the SDK has to jump through every time.
Well, on my side, I *do* want an artefact that is versioned, so that I am sure users do not extract a new one over a previous one, so that they can work on different versions at the same time.
Quoted 7 lines
Versionned artefacts are a requirement for some people.
And I do want that version to represent something that I can actually identify. By default, I *do* want it to identify the version of Buildroot that was used to generate that SDK. But I understand that someople may want another identifier, hence the variable to customise it.
Nowhere in Buildroot we have versioned artefacts. rootfs.tar is always rootfs.tar, rootfs.ext2 is always rootfs.ext2, etc.
I don't think it makes sense to change this policy just for the tarball.
Yocto has versioned artefacts, with a non-versioned symlink. And it's super annoying IMO, as you collect gazillions of leftovers from previous builds, making the image folder quickly unreadable.
Quoted 9 lines
Quoted 3 lines
I'm not really happy with this, but at least I now have the choice and just need to update one line in the build script to disable all this useless stuff.
Well, I prefer the rule of least surprise, so that users by default do not have surprises when they use (i.e. extract) the generated tarball. Getting a tarbomb is very bad for most users. And I prefer to adapt my scripts to accomodate with that (tar's --strip-components=1, as already mentionned), rather than trash my home dir, or any other precious location.
On that topic, I side with you. Tarballs with everything inside a sub-directory is the norm, and we should do this as well.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin (formerly Free Electrons) Embedded Linux and Kernel engineering https://bootlin.com
PATCH v2 1/1 core/sdk: generate the SDK tarball ourselves
In reply to Yann E. MORINYann, All,
Quoted 4 lines · Yann E. MORIN
Well, on my side, I *do* want an artefact that is versioned, so that I am sure users do not extract a new one over a previous one, so that they can work on different versions at the same time.
Sorry, but the SW build system is the wrong place to generate the versioned artefact.
Quoted 1 line
Versionned artefacts are a requirement for some people.
I agree. But again you hard-code your versioning scheme instead of letting the user decide what versioning scheme is appropriate for his/her use case when she/he exports the build artefact from the automated build system to his/her "versioned storage system".
Quoted 3 lines
But now, I took *all* the feedback for the first iteration, to provide a situation that works for everyone. If the discussion on this new patch makes it that the maintainer do not want it in the end, then fine.
This new patch set (with small improvements) is acceptable and I will not vote against it.
But IMHO the implemented functionality is pretty useless, i.e. all SDK users will switch to 'prepare-sdk' instead.
Regards, Stefan