Skip to contentSkip to conversation

More HOST_DIR/usr fixes

PATCH New Arnout Vandecappelle (Essensium/Mind) 3 participants 22 messages Jul 9, 2017 – Jul 10, 2017 Kbuild

lore
Share & tools
lore

Arnout Vandecappelle (Essensium/Mind)

author
#0 lore raw

cover More HOST_DIR/usr fixes

Thomas started rebuilding the toolchains after the HOST_DIR/usr removal and noticed a corner case that didn't work. While investigating that, I uncovered a slew of other issues.

I was also in the process of adding a test for BR2_TOOLCHAIN_EXTERNAL_PREINSTALLED=y but that turns out to require a lot of refactoring of the testing infra. So it's for another day.

Regards,
Arnout

author
#1 lore raw

PATCH 1/7 Makefile: properly create $(HOST_DIR)/usr compatibility symlink

Up to now we created the $(HOST_DIR)/usr compatibility symlink as part of the creation of $(HOST_DIR) itself. However, when the user specifies a custom BR2_HOST_DIR, it is possible that the directory already exists so this rule will never trigger.

Therefore, add an explicit rule for creating $(HOST_DIR)/usr and add this rule to the dependencies of the dirs target. HOST_DIR itself goes back to the standard rule for directories. The order-only dependency of STAGING_DIR isn't needed any more either: HOST_DIR is implicitly created if needed by mkdir -p, and we don't need to trigger the HOST_DIR rule any more if the directory already exists.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
Makefile +6 -7
index a22e87f29e..188ce9adc7 100644
@@ -237,7 +237,7 @@ LEGAL_REPORT = $(LEGAL_INFO_DIR)/README
 # dependencies anywhere else
 #
 ################################################################################
-$(BUILD_DIR) $(TARGET_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
+$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST):
 	@mkdir -p $@
 
 BR2_CONFIG = $(CONFIG_DIR)/.config

@@ -541,7 +541,7 @@ endif
 
 .PHONY: dirs
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
-	$(HOST_DIR) $(BINARIES_DIR)
+	$(HOST_DIR) $(HOST_DIR)/usr $(BINARIES_DIR)
 
 $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 	$(MAKE1) $(EXTRAMAKEARGS) HOSTCC="$(HOSTCC_NOCCACHE)" HOSTCXX="$(HOSTCXX_NOCCACHE)" silentoldconfig

@@ -552,13 +552,12 @@ prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 .PHONY: world
 world: target-post-image
 
-# When creating HOST_DIR, also symlink usr -> .
-$(HOST_DIR):
-	@mkdir -p $@
-	@ln -snf . $@/usr
+# Compatibility symlink in case a post-build script still uses $(HOST_DIR)/usr
+$(HOST_DIR)/usr: $(HOST_DIR)
+	@ln -snf . $@
 
 # Populating the staging with the base directories is handled by the skeleton package
-$(STAGING_DIR): | $(HOST_DIR)
+$(STAGING_DIR):
 	@mkdir -p $(STAGING_DIR)
 	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
Signature · 1 line
2.13.2
Applied
#2 lore raw

PATCH 1/7 Makefile: properly create $(HOST_DIR)/usr compatibility symlink

Hello,

Quoted 12 lines · Arnout Vandecappelle (Essensium/Mind)

Up to now we created the $(HOST_DIR)/usr compatibility symlink as part of the creation of $(HOST_DIR) itself. However, when the user specifies a custom BR2_HOST_DIR, it is possible that the directory already exists so this rule will never trigger.

Therefore, add an explicit rule for creating $(HOST_DIR)/usr and add this rule to the dependencies of the dirs target. HOST_DIR itself goes back to the standard rule for directories. The order-only dependency of STAGING_DIR isn't needed any more either: HOST_DIR is implicitly created if needed by mkdir -p, and we don't need to trigger the HOST_DIR rule any more if the directory already exists.

Quoted 4 lines
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 Makefile | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#3 lore raw

PATCH 2/7 toolchain-wrapper: remove remaining references to HOST_DIR/usr

In reply to Arnout Vandecappelle (Essensium/Mind)

Commit 14151d77af20ec50eeba6e30465debf87b35faaa that eliminated $(HOST_DIR)/usr seriously missed the toolchain-wrapper - only a single reference was updated, the other three were missed. Commit 015d68c84c9c6ad6f6d41f181d19d813f309088b removed one more. This commit finally removes the two remaining ones.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 toolchain/toolchain-wrapper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
toolchain/toolchain-wrapper.c +2 -2
index a73e6d6f94..c048992656 100644
@@ -208,14 +208,14 @@ int main(int argc, char **argv)
 #elif defined(BR_CROSS_PATH_ABS)
 	ret = snprintf(path, sizeof(path), BR_CROSS_PATH_ABS "/%s" BR_CROSS_PATH_SUFFIX, basename);
 #else
-	ret = snprintf(path, sizeof(path), "%s/usr/bin/%s" BR_CROSS_PATH_SUFFIX, absbasedir, basename);
+	ret = snprintf(path, sizeof(path), "%s/bin/%s" BR_CROSS_PATH_SUFFIX, absbasedir, basename);
 #endif
 	if (ret >= sizeof(path)) {
 		perror(__FILE__ ": overflow");
 		return 3;
 	}
 #ifdef BR_CCACHE
-	ret = snprintf(ccache_path, sizeof(ccache_path), "%s/usr/bin/ccache", absbasedir);
+	ret = snprintf(ccache_path, sizeof(ccache_path), "%s/bin/ccache", absbasedir);
 	if (ret >= sizeof(ccache_path)) {
 		perror(__FILE__ ": overflow");
 		return 3;
Signature · 1 line
2.13.2
Applied
#4 lore raw

PATCH 2/7 toolchain-wrapper: remove remaining references to HOST_DIR/usr

Hello,

Quoted 10 lines · Arnout Vandecappelle (Essensium/Mind)

Commit 14151d77af20ec50eeba6e30465debf87b35faaa that eliminated $(HOST_DIR)/usr seriously missed the toolchain-wrapper - only a single reference was updated, the other three were missed. Commit 015d68c84c9c6ad6f6d41f181d19d813f309088b removed one more. This commit finally removes the two remaining ones.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 toolchain/toolchain-wrapper.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#5 lore raw

PATCH 3/7 support/testing: strip /usr/ part from HOST_DIR

In reply to Arnout Vandecappelle (Essensium/Mind)
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/infra/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
support/testing/infra/__init__.py +2 -2
index a55ad7fdf5..27e2a2708d 100644
@@ -55,7 +55,7 @@ def get_elf_arch_tag(builddir, prefix, fpath, tag):
     v5TEJ
     >>>
     """
-    cmd = ["host/usr/bin/{}-readelf".format(prefix),
+    cmd = ["host/bin/{}-readelf".format(prefix),
            "-A", os.path.join("target", fpath)]
     out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
     regexp = re.compile("^  {}: (.*)$".format(tag))

@@ -80,7 +80,7 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
     /lib/ld-linux-armhf.so.3
     >>>
     """
-    cmd = ["host/usr/bin/{}-readelf".format(prefix),
+    cmd = ["host/bin/{}-readelf".format(prefix),
            "-l", os.path.join("target", fpath)]
     out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
     regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$")
Signature · 1 line
2.13.2
Applied
#6 lore raw

PATCH 3/7 support/testing: strip /usr/ part from HOST_DIR

Hello,

Quoted 4 lines · Arnout Vandecappelle (Essensium/Mind)
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/infra/__init__.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#7 lore raw

PATCH 4/7 support/testing: move BRTest initialisation to __init__

In reply to Arnout Vandecappelle (Essensium/Mind)

BRTest's setUp() method contains a few assignments that initialize its member variables. Since we will want to use these in test case overrides, move them to the __init__ function.

Also allow the config member to be overridden, rather than always taking the class member.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/infra/basetest.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
support/testing/infra/basetest.py +7 -4
index d75458a02c..2a5c9ec939 100644
@@ -36,15 +36,18 @@ class BRTest(unittest.TestCase):
     keepbuilds = False
     jlevel = None
 
+    def __init__(self, names):
+        super(BRTest, self).__init__(names)
+        self.testname = self.__class__.__name__
+        self.builddir = os.path.join(self.__class__.outputdir, self.testname)
+        self.emulator = None
+
     def show_msg(self, msg):
         print "{} {:40s} {}".format(datetime.datetime.now().strftime("%H:%M:%S"),
                                     self.testname, msg)
     def setUp(self):
-        self.testname = self.__class__.__name__
-        self.builddir = os.path.join(self.__class__.outputdir, self.testname)
-        self.emulator = None
         self.show_msg("Starting")
-        config = self.__class__.config
+        config = self.config
         if self.jlevel:
             config += "BR2_JLEVEL={}\n".format(self.jlevel)
         self.b = Builder(config, self.builddir, self.logtofile)
Signature · 1 line
2.13.2
Applied
#8 lore raw

PATCH 4/7 support/testing: move BRTest initialisation to __init__

Hello,

Quoted 11 lines · Arnout Vandecappelle (Essensium/Mind)

BRTest's setUp() method contains a few assignments that initialize its member variables. Since we will want to use these in test case overrides, move them to the __init__ function.

Also allow the config member to be overridden, rather than always taking the class member.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/infra/basetest.py | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
author
#9 lore raw

PATCH 5/7 support/testing: add test of BR2_CCACHE with an external toolchain

In reply to Arnout Vandecappelle (Essensium/Mind)

We piggy-back on an existing test.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/tests/toolchain/test_external.py | 8 ++++++++
 1 file changed, 8 insertions(+)
support/testing/tests/toolchain/test_external.py +8 -0
index afb4bb0b50..c315ef8055 100644
@@ -207,9 +207,13 @@ BR2_TOOLCHAIN_EXTERNAL_CXX=y
         self.emulator.login()
 
 class TestExternalToolchainBuildrootuClibc(TestExternalToolchain):
+    # On this test we piggy-back a test of BR2_CCACHE in combination
+    # with an external toolchain.
     config = BASIC_CONFIG + \
 """
 BR2_arm=y
+BR2_CCACHE=y
+BR2_CCACHE_DIR="{builddir}/ccache-dir"
 BR2_TOOLCHAIN_EXTERNAL=y
 BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
 BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y

@@ -222,6 +226,10 @@ BR2_TOOLCHAIN_EXTERNAL_CXX=y
 """
     toolchain_prefix = "arm-linux"
 
+    def __init__(self, names):
+        super(TestExternalToolchainBuildrootuClibc, self).__init__(names)
+        self.config = self.config.format(builddir = self.builddir)
+
     def test_run(self):
         TestExternalToolchain.common_check(self)
         img = os.path.join(self.builddir, "images", "rootfs.cpio")
Signature · 1 line
2.13.2
#10 lore raw

PATCH 5/7 support/testing: add test of BR2_CCACHE with an external toolchain

Hello,

Quoted 12 lines · Arnout Vandecappelle (Essensium/Mind)

We piggy-back on an existing test.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/tests/toolchain/test_external.py | 8 ++++++++
 1 file changed, 8 insertions(+)
support/testing/tests/toolchain/test_external.py +0 -0
index afb4bb0b50..c315ef8055 100644

support/testing/tests/toolchain/test_external.py BR2_TOOLCHAIN_EXTERNAL_CXX=y · line 211

Quoted 5 lines
support/testing/tests/toolchain/test_external.py +2 -0
@@ -207,9 +207,13 @@ BR2_TOOLCHAIN_EXTERNAL_CXX=y
         self.emulator.login()
 
 class TestExternalToolchainBuildrootuClibc(TestExternalToolchain):
+    # On this test we piggy-back a test of BR2_CCACHE in combination
+    # with an external toolchain.

I'm not sure I'm a big fan of "hijacking" an existing test to validate ccache. Shouldn't we have a separate test for that, clearly identified as being ccache related?

Could be a test that inherits from this one, and simply adds more options. But at least in case of failure it will be clearly identified as a CCache related test in the Gitlab CI results.

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#11 lore raw

PATCH 5/7 support/testing: add test of BR2_CCACHE with an external toolchain

Quoted 26 lines · Thomas Petazzoni

Hello,

Quoted 12 lines · Arnout Vandecappelle (Essensium/Mind)

We piggy-back on an existing test.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 support/testing/tests/toolchain/test_external.py | 8 ++++++++
 1 file changed, 8 insertions(+)
support/testing/tests/toolchain/test_external.py +0 -0
index afb4bb0b50..c315ef8055 100644

support/testing/tests/toolchain/test_external.py BR2_TOOLCHAIN_EXTERNAL_CXX=y · line 211

Quoted 5 lines
support/testing/tests/toolchain/test_external.py +2 -0
@@ -207,9 +207,13 @@ BR2_TOOLCHAIN_EXTERNAL_CXX=y
         self.emulator.login()
 
 class TestExternalToolchainBuildrootuClibc(TestExternalToolchain):
+    # On this test we piggy-back a test of BR2_CCACHE in combination
+    # with an external toolchain.

I'm not sure I'm a big fan of "hijacking" an existing test to validate ccache. Shouldn't we have a separate test for that, clearly identified as being ccache related?

Quoted 3 lines

Could be a test that inherits from this one, and simply adds more options. But at least in case of failure it will be clearly identified as a CCache related test in the Gitlab CI results.

I didn't want to duplicate stuff, that's why. Inheriting is a great idea.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#12 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

In reply to Arnout Vandecappelle (Essensium/Mind)

We create a compatibility symlink for $(HOST_DIR)/usr. However, if that exists already as a directory, the link can't be created. Make will not even try since it exists already and has no dependencies. If it exists as a directory, any post-build script that is still using $(HOST_DIR)/usr will fail to find what it needs. Therefore, add a check that it is not a directory.

This check has to be made as part of some PHONY target. We can reuse the dirs target, on which all packages depend.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Yann E. MORIN <[email protected]>

---
Personally I don't think it's important enough to add such a check. If someone ends up in this situation, they'll be motivated to fix their post-build scripts :-)

---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)
Makefile +5 -0
index 188ce9adc7..a94555d130 100644
@@ -542,6 +542,11 @@ endif
 .PHONY: dirs
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(HOST_DIR)/usr $(BINARIES_DIR)
+	@if [ -d "$(HOST_DIR)/usr" -a ! -L "$(HOST_DIR)/usr" ]; then \
+		printf '%s is not allowed to be a directory\n' "$(HOST_DIR)/usr" 1>&2; \
+		printf 'Please remove this directory and rebuild\n' 1>&2; \
+		exit 1; \
+	fi
 
 $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG)
 	$(MAKE1) $(EXTRAMAKEARGS) HOSTCC="$(HOSTCC_NOCCACHE)" HOSTCXX="$(HOSTCXX_NOCCACHE)" silentoldconfig
Signature · 1 line
2.13.2
#13 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Hello,

Quoted 10 lines · Arnout Vandecappelle (Essensium/Mind)

We create a compatibility symlink for $(HOST_DIR)/usr. However, if that exists already as a directory, the link can't be created. Make will not even try since it exists already and has no dependencies. If it exists as a directory, any post-build script that is still using $(HOST_DIR)/usr will fail to find what it needs. Therefore, add a check that it is not a directory.

This check has to be made as part of some PHONY target. We can reuse the dirs target, on which all packages depend.

Quoted 6 lines
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Yann E. MORIN <[email protected]>

---
Personally I don't think it's important enough to add such a check. If someone ends up in this situation, they'll be motivated to fix their post-build scripts :-)

I also don't like very much adding such checks. In which situation can our users fall into this problem?

They have an existing Buildroot tree, they do "git pull" and do a partial rebuild ?

Makefile endif · line 547

Quoted 11 lines
Makefile +3 -0
index 188ce9adc7..a94555d130 100644
@@ -542,6 +542,11 @@ endif
 .PHONY: dirs
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(HOST_DIR)/usr $(BINARIES_DIR)
+	@if [ -d "$(HOST_DIR)/usr" -a ! -L "$(HOST_DIR)/usr" ]; then \
+		printf '%s is not allowed to be a directory\n' "$(HOST_DIR)/usr" 1>&2; \
+		printf 'Please remove this directory and rebuild\n' 1>&2; \

What about saying "Please run make clean all" ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
#14 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Thomas, All,

On 2017-07-10 18:02 +0200, Thomas Petazzoni spake thusly:

Quoted 19 lines
Quoted 10 lines · Arnout Vandecappelle (Essensium/Mind)

We create a compatibility symlink for $(HOST_DIR)/usr. However, if that exists already as a directory, the link can't be created. Make will not even try since it exists already and has no dependencies. If it exists as a directory, any post-build script that is still using $(HOST_DIR)/usr will fail to find what it needs. Therefore, add a check that it is not a directory.

This check has to be made as part of some PHONY target. We can reuse the dirs target, on which all packages depend.

Quoted 6 lines
Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
Cc Yann E. MORIN <[email protected]>

---
Personally I don't think it's important enough to add such a check. If someone ends up in this situation, they'll be motivated to fix their post-build scripts :-)

Quoted 2 lines

I also don't like very much adding such checks. In which situation can our users fall into this problem?

I've explained it in details when you initially reported the issue of regenerating the toolchains:

    http://lists.busybox.net/pipermail/buildroot/2017-July/197456.html
Quoted 2 lines

They have an existing Buildroot tree, they do "git pull" and do a partial rebuild ?

No, that occurs when a user points HOST_DIR outside, like you do for the toolchains, and that host-dir already exists, and it already contains a usr/ directory.

My opinion is that we should not allow using HOST_DIR if it is not empty at all.

Regards,
Yann E. MORIN.

Quoted 19 lines

Makefile endif · line 547

Quoted 11 lines
Makefile +3 -0
index 188ce9adc7..a94555d130 100644
@@ -542,6 +542,11 @@ endif
 .PHONY: dirs
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(HOST_DIR)/usr $(BINARIES_DIR)
+	@if [ -d "$(HOST_DIR)/usr" -a ! -L "$(HOST_DIR)/usr" ]; then \
+		printf '%s is not allowed to be a directory\n' "$(HOST_DIR)/usr" 1>&2; \
+		printf 'Please remove this directory and rebuild\n' 1>&2; \

What about saying "Please run make clean all" ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#15 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Quoted 3 lines · Yann E. MORIN

Thomas, All,

On 2017-07-10 18:02 +0200, Thomas Petazzoni spake thusly:

[snip]

Quoted 6 lines
Quoted 2 lines

I also don't like very much adding such checks. In which situation can our users fall into this problem?

I've explained it in details when you initially reported the issue of regenerating the toolchains:

Quoted 8 lines
    http://lists.busybox.net/pipermail/buildroot/2017-July/197456.html
Quoted 2 lines

They have an existing Buildroot tree, they do "git pull" and do a partial rebuild ?

No, that occurs when a user points HOST_DIR outside, like you do for the toolchains, and that host-dir already exists, and it already contains a usr/ directory.

Actually, I realize now: when people do a git pull, we expect them to do "make
clean all" afterwards. And "make clean" will delete HOST_DIR - so no HOST_DIR/usr will exist anymore.

So the original report from Thomas only happened because Thomas didn't do a
"make clean" like he should have...

Or am I missing something?

Quoted 2 lines

My opinion is that we should not allow using HOST_DIR if it is not empty at all.

Not sure what you mean with "using HOST_DIR". Do you mean "using a non-default
HOST_DIR"? Do you mean that we should force people to do "make clean all" *all the time* when using a non-default HOST_DIR?

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#16 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Quoted 22 lines · Arnout Vandecappelle
Quoted 3 lines · Yann E. MORIN

Thomas, All,

On 2017-07-10 18:02 +0200, Thomas Petazzoni spake thusly:

[snip]

Quoted 6 lines
Quoted 2 lines

I also don't like very much adding such checks. In which situation can our users fall into this problem?

I've explained it in details when you initially reported the issue of regenerating the toolchains:

Quoted 8 lines
    http://lists.busybox.net/pipermail/buildroot/2017-July/197456.html
Quoted 2 lines

They have an existing Buildroot tree, they do "git pull" and do a partial rebuild ?

No, that occurs when a user points HOST_DIR outside, like you do for the toolchains, and that host-dir already exists, and it already contains a usr/ directory.

Quoted 8 lines

Actually, I realize now: when people do a git pull, we expect them to do "make
clean all" afterwards. And "make clean" will delete HOST_DIR - so no HOST_DIR/usr will exist anymore.

So the original report from Thomas only happened because Thomas didn't do a
"make clean" like he should have...

Or am I missing something?

By the way, since I think this patch is useless, I'm not going to respin it.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#17 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

In reply to Arnout Vandecappelle

Hello,

Quoted 6 lines · Arnout Vandecappelle

Actually, I realize now: when people do a git pull, we expect them to do "make
clean all" afterwards. And "make clean" will delete HOST_DIR - so no HOST_DIR/usr will exist anymore.

So the original report from Thomas only happened because Thomas didn't do a
"make clean" like he should have...

Not correct, each time my output directory and host directories were empty. The only thing that was different is that the host directory is not set to the default of $(BASE_DIR)/host, but to /opt/something, and this /opt/something is created (empty) before I start the build.

So my build was perfectly clean, it is just that I created the host directory prior to starting the build, which IMO isn't crazy, especially when such host directory was customized in the Buildroot configuration.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
author
#18 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Quoted 8 lines · Thomas Petazzoni

Hello,

Quoted 4 lines · Arnout Vandecappelle

Actually, I realize now: when people do a git pull, we expect them to do "make
clean all" afterwards. And "make clean" will delete HOST_DIR - so no HOST_DIR/usr will exist anymore.

Quoted 7 lines
Quoted 2 lines

So the original report from Thomas only happened because Thomas didn't do a
"make clean" like he should have...

Not correct, each time my output directory and host directories were empty. The only thing that was different is that the host directory is not set to the default of $(BASE_DIR)/host, but to /opt/something, and this /opt/something is created (empty) before I start the build.

The part about a non-standard HOST_DIR is irrelevant, it's doing

make clean
mkdir -p output/host
make

that triggered the issue. Which obviously makes no sense at all in the normal situation, but does make some kind of twisted sense in the non-standard HOST_DIR situation.

Quoted 4 lines

So my build was perfectly clean, it is just that I created the host directory prior to starting the build, which IMO isn't crazy, especially when such host directory was customized in the Buildroot configuration.

Well, it doesn't make *that* much sense to me, because pre-populating that
directory with stuff is kind of fragile, given that it will all be removed again by "make clean".

Anyway, I still don't think that this patch is very useful :-)

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
#19 lore raw

PATCH 6/7 Makefile: add check that $(HOST_DIR)/usr is not a directory

Arnout, all,

On 2017-07-10 23:36 +0200, Arnout Vandecappelle spake thusly:

Quoted 27 lines · Thomas Petazzoni
Quoted 8 lines

Hello,

Quoted 4 lines · Arnout Vandecappelle

Actually, I realize now: when people do a git pull, we expect them to do "make
clean all" afterwards. And "make clean" will delete HOST_DIR - so no HOST_DIR/usr will exist anymore.

Quoted 7 lines
Quoted 2 lines

So the original report from Thomas only happened because Thomas didn't do a
"make clean" like he should have...

Not correct, each time my output directory and host directories were empty. The only thing that was different is that the host directory is not set to the default of $(BASE_DIR)/host, but to /opt/something, and this /opt/something is created (empty) before I start the build.

The part about a non-standard HOST_DIR is irrelevant, it's doing

make clean
mkdir -p output/host
make

that triggered the issue. Which obviously makes no sense at all in the normal situation, but does make some kind of twisted sense in the non-standard HOST_DIR situation.

Quoted 8 lines
Quoted 4 lines

So my build was perfectly clean, it is just that I created the host directory prior to starting the build, which IMO isn't crazy, especially when such host directory was customized in the Buildroot configuration.

Well, it doesn't make *that* much sense to me, because pre-populating that
directory with stuff is kind of fragile, given that it will all be removed again by "make clean".

It is not about running "make clean" but having stuff pre-isntalled in there, from a clean build.

I.e. people would do:
mkdir /my/host-dir
tar xf mystuff.tar -C /my/host-dir
make -C buildroot my_toolchain_defconfig
make -C buildroot toolchain

and then their stuff is poulting the host-dir, which we should try to avoid.

Granted, there is nothing that would prevent them from extracting it after the build...

Quoted 1 line

Anyway, I still don't think that this patch is very useful :-)

Until we have weird build errors that are reported, because some weirdo will have already extracted a few toolchains in there. Or tries to re-use the same host dir to isntall many toolchains in there... ;-)

(Note: I take it that Thomas uses different host dirs for each of the toolchains, ot that the host dir is empty at the start of each toolchain build.)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
author
#20 lore raw

PATCH 7/7 toolchain-external: default BR2_TOOLCHAIN_EXTERNAL_PATH to empty

In reply to Arnout Vandecappelle (Essensium/Mind)

It makes no sense to default to an arbitrary path. In addition, it in fact works correctly when it is empty. In that case, the toolchain will be searched in PATH.

Update the help text to explain the above, and also that the compiler is supposed to be in the bin subdirectory.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 toolchain/toolchain-external/Config.in | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
toolchain/toolchain-external/Config.in +6 -2
index 4cd9ee4752..c4063b177b 100644
@@ -67,10 +67,14 @@ endchoice
 
 config BR2_TOOLCHAIN_EXTERNAL_PATH
 	string "Toolchain path"
-	default "/path/to/toolchain/usr"
+	default ""
 	depends on BR2_TOOLCHAIN_EXTERNAL_PREINSTALLED
 	help
-	  Path to where the external toolchain is installed.
+	  Path to where the external toolchain is installed. The
+	  compiler itself is expected to be in the "bin" subdirectory
+	  of this path.
+
+	  If empty, the compiler will be searched in $PATH.
 
 config BR2_TOOLCHAIN_EXTERNAL_GLIBC
 	bool
Signature · 1 line
2.13.2
Applied
#21 lore raw

PATCH 7/7 toolchain-external: default BR2_TOOLCHAIN_EXTERNAL_PATH to empty

Hello,

Quoted 11 lines · Arnout Vandecappelle (Essensium/Mind)

It makes no sense to default to an arbitrary path. In addition, it in fact works correctly when it is empty. In that case, the toolchain will be searched in PATH.

Update the help text to explain the above, and also that the compiler is supposed to be in the bin subdirectory.

Signed-off-by Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
 toolchain/toolchain-external/Config.in | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com