Skip to contentSkip to conversation

mtree: new package

PATCH Superseded Esben Haabendal 4 participants 11 messages Mar 26, 2019 – Mar 29, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 mtree: new package

From: Esben Haabendal <[email protected]>

Rebased against latest master, fixing merge conflict with DEVELOPERS file.

---
 DEVELOPERS               |  1 +
 package/Config.in        |  1 +
 package/mtree/Config.in  | 15 +++++++++++++++
 package/mtree/mtree.hash |  5 +++++
 package/mtree/mtree.mk   | 17 +++++++++++++++++
 5 files changed, 39 insertions(+)
 create mode 100644 package/mtree/Config.in
 create mode 100644 package/mtree/mtree.hash
 create mode 100644 package/mtree/mtree.mk
DEVELOPERS +1 -0
index d21e494b1426..9496b30d3429 100644
@@ -700,6 +700,7 @@ F:	package/szip/
 
 N:	Esben Haabendal <[email protected]>
 F:	boot/gummiboot/
+F:	package/mtree/
 
 N:	Etienne Carriere <[email protected]>
 F:	boot/optee-os/
package/Config.in +1 -0
index e372e0c43009..747eb39435f8 100644
@@ -197,6 +197,7 @@ menu "Filesystem and flash utilities"
 	source "package/mmc-utils/Config.in"
 	source "package/mtd/Config.in"
 	source "package/mtools/Config.in"
+	source "package/mtree/Config.in"
 	source "package/nfs-utils/Config.in"
 	source "package/nilfs-utils/Config.in"
 	source "package/ntfs-3g/Config.in"
package/mtree/Config.in new file +15 -0
new file mode 100644
index 000000000000..175d05d3bb0d
@@ -0,0 +1,15 @@
+config BR2_PACKAGE_MTREE
+	bool "mtree"
+	# mtree uses <fts.h> which is not included by default in uClibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC
+	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
+	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
+	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM
+	select BR2_PACKAGE_OPENSSL
+	help
+	  mtree - utility for creating and verifying file hierarchies.
+
+	  Port of the mtree utility, originally only available in BSD
+	  distributions, to Linux and other operating systems.
+
+	  https://github.com/archiecobbs/mtree-port
package/mtree/mtree.hash new file +5 -0
new file mode 100644
index 000000000000..8dc54a201133
@@ -0,0 +1,5 @@
+# Locally computed
+sha256 fb5e99b28e270ab9a4622bd7142e9a7e9703fa08db3086398c52d33bb0505327  mtree-1.0.4.tar.gz
+
+# License file (locally computed as well)
+sha256 024360ed813a773260cee77ae84196da44a0842f45cd9bddad869de13d6c4bd2  COPYING
package/mtree/mtree.mk new file +17 -0
new file mode 100644
index 000000000000..6227508106c7
@@ -0,0 +1,17 @@
+################################################################################
+#
+# mtree
+#
+################################################################################
+
+MTREE_VERSION = 1.0.4
+MTREE_SOURCE = mtree-$(MTREE_VERSION).tar.gz
+MTREE_SITE = https://s3.amazonaws.com/archie-public/mtree-port
+MTREE_LICENSE = BSD-3-Clause
+MTREE_LICENSE_FILES = COPYING
+
+MTREE_DEPENDENCIES = openssl
+
+MTREE_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) install-exec
+
+$(eval $(autotools-package))
Signature · 1 line
2.21.0
#1 lore raw

PATCH v4 1/1 mtree: new package

Hello Esben,

package/mtree/Config.in line 7

Quoted 13 lines · On Tue, 26 Mar 2019 15:24:11 +0100 Esben Haabendal <[email protected]>…
package/mtree/Config.in new file +7 -0
new file mode 100644
index 000000000000..175d05d3bb0d
@@ -0,0 +1,15 @@
+config BR2_PACKAGE_MTREE
+	bool "mtree"
+	# mtree uses <fts.h> which is not included by default in uClibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC
+	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
+	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
+	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM

My memory is a bit fuzzy on this, but we have some workarounds for this issue in other packages:

# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h

# large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992
# for more information.
LIBCGROUP_CONF_ENV = \
        CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"

# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h

# large file support.
# See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information
RESTORECOND_MAKE_OPTS += \
        $(TARGET_CONFIGURE_OPTS) \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \
        CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \
        ARCH="$(BR2_ARCH)"

and in a few other places.

Is it the same problem ? If it has been resolved in glibc 2.23, should we remove those work-arounds ?

I just would like to make sure we don't work around this problem in different ways in different packages.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

PATCH v4 1/1 mtree: new package

Hi Thomas

Quoted 37 lines · Thomas Petazzoni

package/mtree/Config.in line 7

Quoted 13 lines · On Tue, 26 Mar 2019 15:24:11 +0100 Esben Haabendal <[email protected]>…
package/mtree/Config.in new file +7 -0
new file mode 100644
index 000000000000..175d05d3bb0d
@@ -0,0 +1,15 @@
+config BR2_PACKAGE_MTREE
+	bool "mtree"
+	# mtree uses <fts.h> which is not included by default in uClibc
+	depends on BR2_TOOLCHAIN_USES_GLIBC
+	# fts.h does not support LFS (_FILE_OFFSET_BITS==64) in glibc versions
+	# older than 2.23, and codesourcery-arm is currently using glibc 2.18
+	depends on !BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM

My memory is a bit fuzzy on this, but we have some workarounds for this issue in other packages:

# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h

# large file support. See https://bugzilla.redhat.com/show_bug.cgi?id=574992
# for more information.
LIBCGROUP_CONF_ENV = \
        CXXFLAGS="$(TARGET_CXXFLAGS) -U_FILE_OFFSET_BITS" \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS"

# Undefining _FILE_OFFSET_BITS here because of a "bug" with glibc fts.h

# large file support.
# See https://bugzilla.redhat.com/show_bug.cgi?id=574992 for more information
RESTORECOND_MAKE_OPTS += \
        $(TARGET_CONFIGURE_OPTS) \
        CFLAGS="$(TARGET_CFLAGS) -U_FILE_OFFSET_BITS" \
        CPPFLAGS="$(TARGET_CPPFLAGS) -U_FILE_OFFSET_BITS" \
        ARCH="$(BR2_ARCH)"
Quoted 3 lines

and in a few other places.

Is it the same problem ?

Yes, sounds very much like the same problem.

Quoted 2 lines

If it has been resolved in glibc 2.23, should
we remove those work-arounds ?

It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.

Quoted 2 lines

I just would like to make sure we don't work around this problem in different ways in different packages.

Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.

For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.

If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...

/Esben

#3 lore raw

PATCH v4 1/1 mtree: new package

Hello Esben,

Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines

If it has been resolved in glibc 2.23, should
we remove those work-arounds ?

It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.

Quoted 2 lines

I just would like to make sure we don't work around this problem in different ways in different packages.

Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.

Quoted 8 lines

For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.

If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...

OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).

So, from there, you have two options:

(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this fts.h problem.

(2) You have more time. In this case, add a hidden option in toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled (including mtree, of course).

Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have broken fts.h.

Let me know which option you chose :-)

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH v4 1/1 mtree: new package

Quoted 45 lines · Thomas Petazzoni

Hello Esben,

Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines

If it has been resolved in glibc 2.23, should
we remove those work-arounds ?

It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.

Quoted 2 lines

I just would like to make sure we don't work around this problem in different ways in different packages.

Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.

Quoted 8 lines

For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.

If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...

OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).

So, from there, you have two options:

(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this fts.h problem.

(2) You have more time. In this case, add a hidden option in toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled (including mtree, of course).

Quoted 8 lines

Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have broken fts.h.

Let me know which option you chose :-)

I think I will give option 2 a try. Sounds like a nice solution :-)

/Esben

#5 lore raw

PATCH v4 1/1 mtree: new package

Quoted 54 lines · Esben Haabendal
Quoted 52 lines · Thomas Petazzoni

Hello Esben,

Quoted 12 lines · On Wed, 27 Mar 2019 23:28:58 +0100 Esben Haabendal <[email protected]>…
Quoted 2 lines

If it has been resolved in glibc 2.23, should
we remove those work-arounds ?

It depends. If we do that, people using external toolchains with glibc older than 2.23 would see breakage.

Quoted 2 lines

I just would like to make sure we don't work around this problem in different ways in different packages.

Sounds like a good plan. Unfortunately, I don't see that any of the proposed workaround generally good.

Quoted 8 lines

For packages where large file support is not deemed an important feature, The -U_FILE_OFFSET_BITS workaround is good. But for packages where large file support could be relevant, it seems like a bad idea to disable it just to stay compatible with old glibc versions that only a few people use together with new buildroot versions.

If you like, I will switch to the -U_FILE_OFFSET_BITS for mtree package. I personally don't need large file support...

OK, I did a bit of testing, taking as example the libcgroup package. I confirmed that without -U_FILE_OFFSET_BITS, libcgroup fails to build with the Sourcery ARM toolchain (old glibc), but builds perfectly fine with the Linaro toolchain (recent glibc).

So, from there, you have two options:

(1) You don't have much time and you're not willing to not much additional work. In this case, use the -U_FILE_OFFSET_BITS trick in mtree. This way, we have the same quirk/workaround everywhere for this fts.h problem.

(2) You have more time. In this case, add a hidden option in toolchain/Config.in called BR2_TOOLCHAIN_HAS_BROKEN_FTS. This hidden option should be selected by the broken ARM Codesourcery toolchain. Then all packages that do the -U_FILE_OFFSET_BITS trick can be changed to only do it when BR2_TOOLCHAIN_HAS_BROKEN_FTS is enabled (including mtree, of course).

Obviously, this leaves old custom external toolchains potentially broken, as for these, the work around will not be applied. If this turns out to really be a problem in practice, we can always add an option BR2_TOOLCHAIN_EXTERNAL_CUSTOM_HAS_BROKEN_FTS, and let the user enable it + a check that if this option is not enabled, the external toolchain really does not have broken fts.h.

Quoted 3 lines
Quoted 1 line

Let me know which option you chose :-)

I think I will give option 2 a try. Sounds like a nice solution :-)

To be honest, I'm more inclined to remove the -U completely and add an
autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.

Regards,
Arnout

#6 lore raw

PATCH v4 1/1 mtree: new package

Quoted 5 lines · On Thu, 28 Mar 2019 19:13:16 +0100 Arnout Vandecappelle <[email protected]> wrote:
Quoted 1 line

I think I will give option 2 a try. Sounds like a nice solution :-)

To be honest, I'm more inclined to remove the -U completely and add an
autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.

I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH v4 1/1 mtree: new package

Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Thomas" == Thomas Petazzoni <[email protected]> writes:

On Thu, 28 Mar 2019 19:13:16 +0100
Arnout Vandecappelle <[email protected]> wrote:

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

I think I will give option 2 a try. Sounds like a nice solution :-)

To be honest, I'm more inclined to remove the -U completely and add an autobuilder exception... I don't believe we have to go too far to work around bugs in old toolchains.

I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.

Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):

http://lists.busybox.net/pipermail/buildroot/2019-February/244056.html

I don't have a problem dropping support for ancient toolchains when they add too much complexity, but the lack of large file support is mtree is probably not really reason enough.

Signature · 1 line
Bye, Peter Korsgaard
#8 lore raw

PATCH v4 1/1 mtree: new package

Quoted 5 lines · On Thu, 28 Mar 2019 21:59:59 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 2 lines

I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.

Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):

In fact, my last sentence lacked an ending question mark. I wanted to ask if we were dropping support entirely for glibc 2.19, or just saying "glibc 2.19 is not long important enough to worry too much about it, especially when it's just for the few packages that use the <fts.h> interface".

But if your point is to have me say that I don't have a clear and well-defined opinion, then yes it's the case. I do see a number of companies/customers continue to use old toolchains and therefore dropping support too quickly tends to be annoying. On the other side, we clearly see the maintenance burden that keeping support for old toolchains creates.

Quoted 3 lines

I don't have a problem dropping support for ancient toolchains when they add too much complexity, but the lack of large file support is mtree is probably not really reason enough.

So, what do you suggest ? What I proposed with the hidden option ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#9 lore raw

PATCH v4 1/1 mtree: new package

Quoted 6 lines · Thomas Petazzoni
Quoted 3 lines · On Thu, 28 Mar 2019 21:59:59 +0100 Peter Korsgaard <[email protected]> wrote:
Quoted 2 lines

I'm fine with this as well. It means we would no longer support glibc 2.19 anymore.

Quoted 8 lines
Quoted 2 lines

Didn't you just last month argue against removing support for glibc < 2.19 support (the runc security fix):

In fact, my last sentence lacked an ending question mark. I wanted to ask if we were dropping support entirely for glibc 2.19, or just saying "glibc 2.19 is not long important enough to worry too much about it, especially when it's just for the few packages that use the <fts.h> interface".

Just to be clear: we don't remove support for these old toolchains. We remove
the "guarantee" (which was anyway never much of a guarantee) that all packages that you can select in menuconfig will build with it.

Quoted 4 lines

But if your point is to have me say that I don't have a clear and well-defined opinion, then yes it's the case. I do see a number of companies/customers continue to use old toolchains and therefore dropping support too quickly tends to be annoying.

But these will typically *not* be one of our external toolchains, but rather a
custom external toolchain. So the solution of a hidden option is not much of a solution.

Of course, it can be added to the custom external toolchain options as well.
But then, that's one more option that *everybody* has to set for their custom external toolchains, while it only really applies to the few people that use such an old toolchain *and* that want to select packages that were not even available to them at the time that they started using this toolchain. In my opinion, the "few" here is 0, maybe 1.

A really fundamental solution could be that at build time we detect in these
packages whether the toolchain has a broken fts.h, and if yes we add -Ulargefile. But that's also complicated.

That is actually my main point: I don't think adding such an option is worth
the bother, because it can only really solve the issue by adding a lot of complexity.

Quoted 5 lines

On the other side,
we clearly see the maintenance burden that keeping support for old toolchains creates.

Quoted 1 line

I don't have a problem dropping support for ancient toolchains when they

The important thing to note is: we drop support for such ancient toolchains
trying to build the mtree package.

Regards,
Arnout

Quoted 7 lines
Quoted 2 lines

add too much complexity, but the lack of large file support is mtree is probably not really reason enough.

So, what do you suggest ? What I proposed with the hidden option ?

Thomas

#10 lore raw

PATCH v4 1/1 mtree: new package

Quoted 4 lines · On Fri, 29 Mar 2019 10:31:22 +0100 Arnout Vandecappelle <[email protected]> wrote:
Quoted 1 line
Quoted 1 line

I don't have a problem dropping support for ancient toolchains when they

The important thing to note is: we drop support for such ancient toolchains
trying to build the mtree package.

Fair enough. So let's drop all these -U_FILE_OFFSET_BITS from all packages that use this workaround, and add the corresponding autobuilder exceptions in utils/genrandconfig.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com