mtree: new package
PATCH v4 1/1 mtree: new package
From: Esben Haabendal <[email protected]>
Rebased against latest master, fixing merge conflict with DEVELOPERS file.
--- DEVELOPERS | 1 + package/Config.in | 1 + package/mtree/Config.in | 15 +++++++++++++++ package/mtree/mtree.hash | 5 +++++ package/mtree/mtree.mk | 17 +++++++++++++++++ 5 files changed, 39 insertions(+) create mode 100644 package/mtree/Config.in create mode 100644 package/mtree/mtree.hash create mode 100644 package/mtree/mtree.mk
DEVELOPERS +1 -0
@@ -700,6 +700,7 @@ F: package/szip/ N: Esben Haabendal <[email protected]> F: boot/gummiboot/ +F: package/mtree/ N: Etienne Carriere <[email protected]> F: boot/optee-os/
package/Config.in +1 -0
@@ -197,6 +197,7 @@ menu "Filesystem and flash utilities" source "package/mmc-utils/Config.in" source "package/mtd/Config.in" source "package/mtools/Config.in" + source "package/mtree/Config.in" source "package/nfs-utils/Config.in" source "package/nilfs-utils/Config.in" source "package/ntfs-3g/Config.in"
package/mtree/Config.in new file +15 -0
@@ -0,0 +1,15 @@ +config BR2_PACKAGE_MTREE + bool "mtree" + # mtree uses <fts.h> which is not included by default in uClibc + depends on BR2_TOOLCHAIN_USES_GLIBC + # fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions + # older than 2.23, and codesourcery-arm is currently using glibc 2.18 + depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM + select BR2_PACKAGE_OPENSSL + help + mtree - utility for creating and verifying file hierarchies. + + Port of the mtree utility, originally only available in BSD + distributions, to Linux and other operating systems. + + https://github.com/archiecobbs/mtree-port
package/mtree/mtree.hash new file +5 -0
@@ -0,0 +1,5 @@ +# Locally computed +sha256 fb5e99b28e270ab9a4622bd7142e9a7e9703fa08db3086398c52d33bb0505327 mtree-1.0.4.tar.gz + +# License file (locally computed as well) +sha256 024360ed813a773260cee77ae84196da44a0842f45cd9bddad869de13d6c4bd2 COPYING
package/mtree/mtree.mk new file +17 -0
@@ -0,0 +1,17 @@ +################################################################################ +# +# mtree +# +################################################################################ + +MTREE_VERSION = 1.0.4 +MTREE_SOURCE = mtree-$(MTREE_VERSION).tar.gz +MTREE_SITE = https://s3.amazonaws.com/archie-public/mtree-port +MTREE_LICENSE = BSD-3-Clause +MTREE_LICENSE_FILES = COPYING + +MTREE_DEPENDENCIES = openssl + +MTREE_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) install-exec + +$(eval $(autotools-package))
Signature · 1 line
2.21.0
PATCH v4 1/1 mtree: new package
Hello Esben,
package/mtree/Config.in line 7
Quoted 13 lines · On Tue, 26 Mar 2019 15:24:11 +0100 Esben Haabendal <[email protected]>…
package/mtree/Config.in new file +7
-0@@ -0,0 +1,15 @@ +config BR2_PACKAGE_MTREE + bool "mtree" + # mtree uses <fts.h> which is not included by default in uClibc + depends on BR2_TOOLCHAIN_USES_GLIBC + # fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions + # older than 2.23, and codesourcery-arm is currently using glibc 2.18 + depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM
My memory is a bit fuzzy on this, but we have some workarounds for this issue in other packages:
# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
# large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992 # for more information. LIBCGROUP_CONF_ENV = \ CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \ CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"
# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h
# large file support. # See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information RESTORECOND_MAKE_OPTS += \ $(TARGET_CONFIGURE_OPTS) \ CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \ CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \ ARCH="$(BR2_ARCH)"
and in a few other places.
Is it the same problem ? If it has been resolved in glibc 2.23, should we remove those work-arounds ?
I just would like to make sure we don't work around this problem in different ways in different packages.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v4 1/1 mtree: new package
Hi Thomas
Quoted 37 lines · Thomas Petazzoni
package/mtree/Config.in line 7
Quoted 13 lines · On Tue, 26 Mar 2019 15:24:11 +0100 Esben Haabendal <[email protected]>…
package/mtree/Config.in new file +7
-0@@ -0,0 +1,15 @@ +config BR2_PACKAGE_MTREE + bool "mtree" + # mtree uses <fts.h> which is not included by default in uClibc + depends on BR2_TOOLCHAIN_USES_GLIBC + # fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions + # older than 2.23, and codesourcery-arm is currently using glibc 2.18 + depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARMMy memory is a bit fuzzy on this, but we have some workarounds for this issue in other packages:
# Undefining
_FILE_OFFSET_BITShere because of a "bug" with glibcfts.h# large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992 # for more information. LIBCGROUP_CONF_ENV = \ CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \ CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"# Undefining
_FILE_OFFSET_BITShere because of a "bug" with glibcfts.h# large file support. # See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information RESTORECOND_MAKE_OPTS += \ $(TARGET_CONFIGURE_OPTS) \ CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \ CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \ ARCH="$(BR2_ARCH)"
Quoted 3 lines
and in a few other places.
Is it the same problem ?
Yes, sounds very much like the same problem.
Quoted 2 lines
If it has been resolved in glibc 2.23, should
we remove those work-arounds ?
It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.
Quoted 2 lines
I just would like to make sure we don't work around this problem in different ways in different packages.
Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.
For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.
If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...
/Esben
PATCH v4 1/1 mtree: new package
Hello Esben,
Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines
If it has been resolved in glibc 2.23, should
we remove those work-arounds ?It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.
Quoted 2 lines
I just would like to make sure we don't work around this problem in different ways in different packages.
Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.
Quoted 8 lines
For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.
If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...
OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).
So, from there, you have two options:
(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this fts.h problem.
(2) You have more time. In this case, add a hidden option in toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled (including mtree, of course).
Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have broken fts.h.
Let me know which option you chose :-)
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v4 1/1 mtree: new package
Quoted 45 lines · Thomas Petazzoni
Hello Esben,
Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines
If it has been resolved in glibc 2.23, should
we remove those work-arounds ?It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.
Quoted 2 lines
I just would like to make sure we don't work around this problem in different ways in different packages.
Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.
Quoted 8 lines
For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.
If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...
OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).
So, from there, you have two options:
(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this
fts.hproblem.(2) You have more time. In this case, add a hidden option in toolchain/Config.in called
BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it whenBR2_TOOLCHAIN_HAS_BROKEN_FTSis enabled (including mtree, of course).
Quoted 8 lines
Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have brokenfts.h.Let me know which option you chose :-)
I think I will give option 2 a try. Sounds like a nice solution :-)
/Esben
PATCH v4 1/1 mtree: new package
Quoted 54 lines · Esben Haabendal
Quoted 52 lines · Thomas Petazzoni
Hello Esben,
Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines
If it has been resolved in glibc 2.23, should
we remove those work-arounds ?It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.
Quoted 2 lines
I just would like to make sure we don't work around this problem in different ways in different packages.
Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.
Quoted 8 lines
For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.
If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...
OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).
So, from there, you have two options:
(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this
fts.hproblem.(2) You have more time. In this case, add a hidden option in toolchain/Config.in called
BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it whenBR2_TOOLCHAIN_HAS_BROKEN_FTSis enabled (including mtree, of course).Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have brokenfts.h.
Quoted 3 lines
Quoted 1 line
Let me know which option you chose :-)
I think I will give option 2 a try. Sounds like a nice solution :-)
To be honest, I'm more inclined to remove the -U completely and add an
autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.
Regards,
Arnout
PATCH v4 1/1 mtree: new package
Quoted 5 lines · On Thu, 28 Mar 2019 19:13:16 +0100 Arnout Vandecappelle <[email protected]> wrote:
Quoted 1 line
I think I will give option 2 a try. Sounds like a nice solution :-)
To be honest, I'm more inclined to remove the -U completely and add an
autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.
I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v4 1/1 mtree: new package
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
On Thu, 28 Mar 2019 19:13:16 +0100
Arnout Vandecappelle <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
I think I will give option 2 a try. Sounds like a nice solution :-)
To be honest, I'm more inclined to remove the -U completely and add an autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.
I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.
Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):
http://lists.busybox.net/pipermail/buildroot/2019-February/244056.html
I don't have a problem dropping support for ancient toolchains when they add too much complexity, but the lack of large file support is mtree is probably not really reason enough.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v4 1/1 mtree: new package
Quoted 5 lines · On Thu, 28 Mar 2019 21:59:59 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 2 lines
I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.
Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):
In fact, my last sentence lacked an ending question mark. I wanted to ask if we were dropping support entirely for glibc 2.19, or just saying "glibc 2.19 is not long important enough to worry too much about it, especially when it's just for the few packages that use the <fts.h> interface".
But if your point is to have me say that I don't have a clear and well-defined opinion, then yes it's the case. I do see a number of companies/customers continue to use old toolchains and therefore dropping support too quickly tends to be annoying. On the other side, we clearly see the maintenance burden that keeping support for old toolchains creates.
Quoted 3 lines
I don't have a problem dropping support for ancient toolchains when they add too much complexity, but the lack of large file support is mtree is probably not really reason enough.
So, what do you suggest ? What I proposed with the hidden option ?
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v4 1/1 mtree: new package
Quoted 6 lines · Thomas Petazzoni
Quoted 3 lines · On Thu, 28 Mar 2019 21:59:59 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 2 lines
I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.
Quoted 8 lines
Quoted 2 lines
Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):
In fact, my last sentence lacked an ending question mark. I wanted to ask if we were dropping support entirely for glibc 2.19, or just saying "glibc 2.19 is not long important enough to worry too much about it, especially when it's just for the few packages that use the <
fts.h> interface".
Just to be clear: we don't remove support for these old toolchains. We remove
the "guarantee" (which was anyway never much of a guarantee) that all packages that you can select in menuconfig will build with it.
Quoted 4 lines
But if your point is to have me say that I don't have a clear and well-defined opinion, then yes it's the case. I do see a number of companies/customers continue to use old toolchains and therefore dropping support too quickly tends to be annoying.
But these will typically *not* be one of our external toolchains, but rather a
custom external toolchain. So the solution of a hidden option is not much of a solution.
Of course, it can be added to the custom external toolchain options as well.
But then, that's one more option that *everybody* has to set for their custom external toolchains, while it only really applies to the few people that use such an old toolchain *and* that want to select packages that were not even available to them at the time that they started using this toolchain. In my opinion, the "few" here is 0, maybe 1.
A really fundamental solution could be that at build time we detect in these
packages whether the toolchain has a broken fts.h, and if yes we add -Ulargefile. But that's also complicated.
That is actually my main point: I don't think adding such an option is worth
the bother, because it can only really solve the issue by adding a lot of complexity.
Quoted 5 lines
On the other side,
we clearly see the maintenance burden that keeping support for old toolchains creates.Quoted 1 line
I don't have a problem dropping support for ancient toolchains when they
The important thing to note is: we drop support for such ancient toolchains
trying to build the mtree package.
Regards,
Arnout
Quoted 7 lines
Quoted 2 lines
add too much complexity, but the lack of large file support is mtree is probably not really reason enough.
So, what do you suggest ? What I proposed with the hidden option ?
Thomas
PATCH v4 1/1 mtree: new package
Quoted 4 lines · On Fri, 29 Mar 2019 10:31:22 +0100 Arnout Vandecappelle <[email protected]> wrote:
Quoted 1 line
Quoted 1 line
I don't have a problem dropping support for ancient toolchains when they
The important thing to note is: we drop support for such ancient toolchains
trying to build the mtree package.
Fair enough. So let's drop all these -U_FILE_OFFSET_BITS from all packages that use this workaround, and add the corresponding autobuilder exceptions in utils/genrandconfig.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com