buildman updates

PATCH New Trevor Woerner 6 participants 16 messages Nov 21, 2018 – Dec 5, 2018

lore

Share & tools

lore

Trevor Woerner

author

2018-11-21 08:31 UTC #0 lore raw

cover buildman updates

Hi,

This patchset was created to update buildman, and fix a couple small issues that were discovered. kernel.org has a new set of toolchains (8.1.0), and I ran into some issues trying to download them.

I'm not a python guru, so if there are better ways to code these fixes, please let me know.

Best regards,
Trevor

Trevor Woerner (3):
buildman/toolchain.py: update versions buildman/toolchain.py: fix toolchain directory buildman/toolchain.py: handle inconsistent tarball names

 tools/buildman/toolchain.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

Signature · 1 line

2.17.0.582.gccdcbd54c

Trevor Woerner

author

2018-11-21 08:31 UTC #1 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

On kernel.org, a newer 8.1.0 version of the toolchains is available. Also, update the kernel.org version list to include some of the older versions so more architectures can be supported.

Signed-off-by Trevor Woerner <[email protected]>

---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

tools/buildman/toolchain.py +1 -1

index 4b35f400e9..442a7f977a 100644
@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
         links = []
         for version in versions:
             url = '%s/%s/%s/' % (base, arch, version)

Signature · 1 line

2.17.0.582.gccdcbd54c

Simon Glass

2018-11-27 01:02 UTC #2 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

Quoted 9 lines · Trevor Woerner

On kernel.org, a newer 8.1.0 version of the toolchains is available. Also, update the kernel.org version list to include some of the older versions so more architectures can be supported.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by Simon Glass <[email protected]>

Daniel Schwierzeck

2018-11-29 01:35 UTC #3 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

In reply to Trevor Woerner

Am 21.11.18 um 09:31 schrieb Trevor Woerner:

Quoted 14 lines

On kernel.org, a newer 8.1.0 version of the toolchains is available. Also, update the kernel.org version list to include some of the older versions so more architectures can be supported.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/buildman/toolchain.py +0 -0
index 4b35f400e9..442a7f977a 100644

tools/buildman/toolchain.py class Toolchains: · line 433

Quoted 5 lines

tools/buildman/toolchain.py +1 -1

@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']

with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and unneeded versions. So it doesn't make sense to add them again.

Also did you a test run on Travis CI to verify that all archs still compile where a 8.1.0 toolchain is available? Travis CI will automatically use the latest avaiable toolchain for each arch. Possible new warnings or errors need to be fixed before doing the switch to 8.1.0.

Quoted 4 lines

tools/buildman/toolchain.py +0 -0

         links = []
         for version in versions:
             url = '%s/%s/%s/' % (base, arch, version)

Signature · 9 lines

- Daniel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181129/76f94910/attachment.sig>

Tom Rini

2018-11-29 19:03 UTC #4 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

Quoted 26 lines · Daniel Schwierzeck

Am 21.11.18 um 09:31 schrieb Trevor Woerner:
Quoted 14 lines
On kernel.org, a newer 8.1.0 version of the toolchains is available. Also, update the kernel.org version list to include some of the older versions so more architectures can be supported.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/buildman/toolchain.py +0 -0
index 4b35f400e9..442a7f977a 100644
tools/buildman/toolchain.py class Toolchains: · line 433
Quoted 5 lines
tools/buildman/toolchain.py +1 -1
@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4', '4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and unneeded versions. So it doesn't make sense to add them again.

Quoted 4 lines

Also did you a test run on Travis CI to verify that all archs still compile where a 8.1.0 toolchain is available? Travis CI will automatically use the latest avaiable toolchain for each arch. Possible new warnings or errors need to be fixed before doing the switch to 8.1.0.

Agreed. Are you using buildman outside of U-Boot where you have a usecase for adding in old versions / other architectures? I do want to move up to 8.1.0 but I'm sure we have warnings to fix before we can do that. I _think_ I even posted a link to what was failing a while ago and fixed a few things, but there's some harder to solve problems left.

Signature · 8 lines

Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181129/7c3d1b85/attachment.sig>

Trevor Woerner

2018-11-30 11:48 UTC #5 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

Quoted 31 lines · Tom Rini

Quoted 4 lines · Daniel Schwierzeck
Am 21.11.18 um 09:31 schrieb Trevor Woerner:
Quoted 1 line
On kernel.org, a newer 8.1.0 version of the toolchains is available.
Also,
Quoted 1 line
Quoted 1 line
update the kernel.org version list to include some of the older
versions so
Quoted 7 lines
Quoted 7 lines
more architectures can be supported.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 10 lines
tools/buildman/toolchain.py class Toolchains: · line 433
Quoted 10 lines
tools/buildman/toolchain.py +1 -1
index 4b35f400e9..442a7f977a 100644
@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4',
'4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
Quoted 4 lines
with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and unneeded versions. So it doesn't make sense to add them again.

Quoted 7 lines

Quoted 4 lines
Also did you a test run on Travis CI to verify that all archs still compile where a 8.1.0 toolchain is available? Travis CI will automatically use the latest avaiable toolchain for each arch. Possible new warnings or errors need to be fixed before doing the switch to 8.1.0.
Agreed. Are you using buildman outside of U-Boot where you have a usecase for adding in old versions / other architectures?

Yes. I was using it as a general-purpose toolchain-downloading tool.

Quoted 5 lines

I do want to
move up to 8.1.0 but I'm sure we have warnings to fix before we can do that. I _think_ I even posted a link to what was failing a while ago and fixed a few things, but there's some harder to solve problems left.

Okay, no problem.

In any case I'm partway through doing a test with and without my patches. I'll post my results for the sake of completeness.

I'm also interested in exploring whether it might be possible to have buildman download pre-built toolchains from bootlin. Maybe that would be of some interest?

Tom Rini

2018-11-30 12:18 UTC #6 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

Quoted 51 lines · Trevor Woerner

Quoted 31 lines · Tom Rini
Quoted 4 lines · Daniel Schwierzeck
Am 21.11.18 um 09:31 schrieb Trevor Woerner:
Quoted 1 line
On kernel.org, a newer 8.1.0 version of the toolchains is available.
Also,
Quoted 1 line
Quoted 1 line
update the kernel.org version list to include some of the older
versions so
Quoted 7 lines
Quoted 7 lines
more architectures can be supported.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 10 lines
tools/buildman/toolchain.py class Toolchains: · line 433
Quoted 10 lines
tools/buildman/toolchain.py +1 -1
index 4b35f400e9..442a7f977a 100644
@@ -430,7 +430,7 @@ class Toolchains:
         """
         arch = command.OutputOneLine('uname', '-m')
         base = 'https://www.kernel.org/pub/tools/crosstool/files/bin'
-        versions = ['7.3.0', '6.4.0', '4.9.4']
+        versions = ['8.1.0', '7.3.0', '6.4.0', '5.5.0', '4.9.4',
'4.9.0', '4.8.5', '4.8.0', '4.7.3', '4.6.3', '4.6.2', '4.6.1', '4.5.1', '4.2.4']
Quoted 4 lines
with 4c58d273e0f2cc21821c1de32494abd5de411d9b we removed the older and unneeded versions. So it doesn't make sense to add them again.
Quoted 7 lines
Quoted 4 lines
Also did you a test run on Travis CI to verify that all archs still compile where a 8.1.0 toolchain is available? Travis CI will automatically use the latest avaiable toolchain for each arch. Possible new warnings or errors need to be fixed before doing the switch to 8.1.0.
Agreed. Are you using buildman outside of U-Boot where you have a usecase for adding in old versions / other architectures?
Yes. I was using it as a general-purpose toolchain-downloading tool.
Quoted 5 lines
I do want to
move up to 8.1.0 but I'm sure we have warnings to fix before we can do that. I _think_ I even posted a link to what was failing a while ago and fixed a few things, but there's some harder to solve problems left.

Quoted 8 lines

Okay, no problem.
In any case I'm partway through doing a test with and without my patches. I'll post my results for the sake of completeness.
I'm also interested in exploring whether it might be possible to have buildman download pre-built toolchains from bootlin. Maybe that would be of some interest?

We had thought about that for a bit, for a while, but Arnd is keeping the kernel.org ones more up to date than Bootlin was, when we had need of easy access to a specific new version. Things may have changed since then since it was 1-2 years ago, at this point. But yes, a patch to add an option to download from elsewhere (perhaps by base URL?) would be useful I think. I can see cases of pointing buildman at an internal mirror instead for example.

Signature · 8 lines

Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20181130/cec37fb7/attachment.sig>

Trevor Woerner

author

2018-12-01 01:41 UTC #7 lore raw

PATCH 1/3 buildman/toolchain.py: update versions

In reply to Trevor Woerner

Quoted 2 lines · Trevor Woerner

In any case I'm partway through doing a test with and without my patches. I'll post my results for the sake of completeness.

Using a relatively recent pull, I have the "master" test, and the "twoerner" test which simply adds my patches on top of "master":

	$ git show-branch --sha1-name master contrib/twoerner/buildman
	! [master] Merge branch '2018-11-28-master-imports'
	 * [contrib/twoerner/buildman] buildman/toolchain.py: handle inconsistent tarball names
	--
	 * [11028a5d2a] buildman/toolchain.py: handle inconsistent tarball names
	 * [ab4e8c6d2c] buildman/toolchain.py: fix toolchain directory
	 * [e8b8be4d32] buildman/toolchain.py: update versions
	-- [e16c888fab] Merge branch '2018-11-28-master-imports'

Cleaning out ~/.buildman and ~/.buildman-toolchains between each test, here are my results after fetching the toolchains:

	$ ./tools/buildman/buildman --fetch-arch all

	$ diff -u master/toolchains twoerner/toolchains 
	--- master/toolchains   2018-11-30 20:32:33.593062110 -0500
	+++ twoerner/toolchains 2018-11-30 20:33:20.025307799 -0500
	@@ -1,14 +1,17 @@
	-List of available toolchains (35):
	+List of available toolchains (41):
	 aarch64
	 alpha
	 am33_2.0
	 arc
	 arm
	+avr32
	 bfin
	 c6x
	 cris
	+crisv32
	 frv
	 h8300
	+hexagon
	 hppa
	 hppa64
	 i386
	@@ -18,14 +21,17 @@
	 microblaze
	 mips
	 mips64
	+nds32le
	 nios2
	 or1k
	+or32
	 powerpc
	 powerpc64
	 ppc64le
	 riscv32
	 riscv64
	 s390
	+s390x
	 sh2
	 sh4
	 sparc

including version information:

	$ diff -u master/toolchains-versions twoerner/toolchains-versions 
	--- master/toolchains-versions  2018-11-30 20:31:56.328864954 -0500
	+++ twoerner/toolchains-versions        2018-11-30 20:31:51.088837231 -0500
	@@ -1,36 +1,42 @@
	-List of available toolchains (35):
	-aarch64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/aarch64-linux/bin/aarch64-linux-gcc
	-alpha     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/alpha-linux/bin/alpha-linux-gcc
	+List of available toolchains (41):
	+aarch64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/aarch64-linux/bin/aarch64-linux-gcc
	+alpha     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/alpha-linux/bin/alpha-linux-gcc
	 am33_2.0  : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/am33_2.0-linux/bin/am33_2.0-linux-gcc
	-arc       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/arc-elf/bin/arc-elf-gcc
	-arm       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-gcc
	+arc       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/arc-linux/bin/arc-linux-gcc
	+arm       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/arm-linux-gnueabi/bin/arm-linux-gnueabi-gcc
	+avr32     : /home/trevor/.buildman-toolchains/gcc-4.2.4-nolibc/avr32-linux/bin/avr32-linux-gcc
	 bfin      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/bfin-uclinux/bin/bfin-uclinux-gcc
	-c6x       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/c6x-elf/bin/c6x-elf-gcc
	+c6x       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/c6x-elf/bin/c6x-elf-gcc
	 cris      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/cris-linux/bin/cris-linux-gcc
	+crisv32   : /home/trevor/.buildman-toolchains/gcc-4.6.3-nolibc/crisv32-linux/bin/crisv32-linux-gcc
	 frv       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/frv-linux/bin/frv-linux-gcc
	-h8300     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/h8300-linux/bin/h8300-linux-gcc
	-hppa      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/hppa-linux/bin/hppa-linux-gcc
	-hppa64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/hppa64-linux/bin/hppa64-linux-gcc
	-i386      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/i386-linux/bin/i386-linux-gcc
	-ia64      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/ia64-linux/bin/ia64-linux-gcc
	+h8300     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/h8300-linux/bin/h8300-linux-gcc
	+hexagon   : /home/trevor/.buildman-toolchains/gcc-4.6.1-nolibc/hexagon-linux/bin/hexagon-linux-gcc
	+hppa      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/hppa-linux/bin/hppa-linux-gcc
	+hppa64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/hppa64-linux/bin/hppa64-linux-gcc
	+i386      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/i386-linux/bin/i386-linux-gcc
	+ia64      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/ia64-linux/bin/ia64-linux-gcc
	 m32r      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/m32r-linux/bin/m32r-linux-gcc
	-m68k      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/m68k-linux/bin/m68k-linux-gcc
	-microblaze: /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/microblaze-linux/bin/microblaze-linux-gcc
	-mips      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/mips-linux/bin/mips-linux-gcc
	-mips64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/mips64-linux/bin/mips64-linux-gcc
	-nios2     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/nios2-linux/bin/nios2-linux-gcc
	+m68k      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/m68k-linux/bin/m68k-linux-gcc
	+microblaze: /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/microblaze-linux/bin/microblaze-linux-gcc
	+mips      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/mips-linux/bin/mips-linux-gcc
	+mips64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/mips64-linux/bin/mips64-linux-gcc
	+nds32le   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/nds32le-elf/bin/nds32le-elf-gcc
	+nios2     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/nios2-linux/bin/nios2-linux-gcc
	 or1k      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/or1k-linux/bin/or1k-linux-gcc
	-powerpc   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/powerpc-linux/bin/powerpc-linux-gcc
	-powerpc64 : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
	+or32      : /home/trevor/.buildman-toolchains/gcc-4.5.1-nolibc/or32-linux/bin/or32-linux-gcc
	+powerpc   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/powerpc-linux/bin/powerpc-linux-gcc
	+powerpc64 : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/powerpc64-linux/bin/powerpc64-linux-gcc
	 ppc64le   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/ppc64le-linux/bin/ppc64le-linux-gcc
	-riscv32   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/riscv32-linux/bin/riscv32-linux-gcc
	-riscv64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc
	-s390      : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/s390-linux/bin/s390-linux-gcc
	-sh2       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sh2-linux/bin/sh2-linux-gcc
	-sh4       : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sh4-linux/bin/sh4-linux-gcc
	-sparc     : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sparc-linux/bin/sparc-linux-gcc
	-sparc64   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/sparc64-linux/bin/sparc64-linux-gcc
	+riscv32   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/riscv32-linux/bin/riscv32-linux-gcc
	+riscv64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/riscv64-linux/bin/riscv64-linux-gcc
	+s390      : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/s390-linux/bin/s390-linux-gcc
	+s390x     : /home/trevor/.buildman-toolchains/gcc-4.9.0-nolibc/s390x-linux/bin/s390x-linux-gcc
	+sh2       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sh2-linux/bin/sh2-linux-gcc
	+sh4       : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sh4-linux/bin/sh4-linux-gcc
	+sparc     : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sparc-linux/bin/sparc-linux-gcc
	+sparc64   : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/sparc64-linux/bin/sparc64-linux-gcc
	 tilegx    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/tilegx-linux/bin/tilegx-linux-gcc
	 tilepro   : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/tilepro-linux/bin/tilepro-linux-gcc
	-x86_64    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/x86_64-linux/bin/x86_64-linux-gcc
	-xtensa    : /home/trevor/.buildman-toolchains/gcc-7.3.0-nolibc/xtensa-linux/bin/xtensa-linux-gcc
	+x86_64    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/x86_64-linux/bin/x86_64-linux-gcc
	+xtensa    : /home/trevor/.buildman-toolchains/gcc-8.1.0-nolibc/xtensa-linux/bin/xtensa-linux-gcc

Getting buildman to build the head revision for everything that it can, the command I used is:

	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/master/output-dir 2>&1 | tee ~/tmp/u-boot-buildman/master/build.log
and
	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/twoerner/output-dir 2>&1 | tee ~/tmp/u-boot-buildman/twoerner/build.log

Then, for results:

	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/master/output-dir -s > ~/tmp/u-boot-buildman/master/build-summary
and
	$ ./tools/buildman/buildman --output-dir ~/tmp/u-boot-buildman/twoerner/output-dir -s > ~/tmp/u-boot-buildman/twoerner/build-summary

I editied these result files so that each build target was listed on its own line (which makes the comparison easier). Comparing between these two builds yields:

	$ diff -u master/build-summary.clean twoerner/build-summary.clean
	--- master/build-summary.clean  2018-11-30 20:10:33.590069536 -0500
	+++ twoerner/build-summary.clean        2018-11-30 20:09:16.325652908 -0500
	@@ -1,5 +1,6 @@
	 boards.cfg is up to date. Nothing to do.
	 Summary of current source for 1305 boards (20 threads, 1 job per thread)
	+       x86:  +    efi-x86_app
	    aarch64:  w+   khadas-vim
			   pine_h64
			   pinebook
	@@ -34,7 +35,13 @@
			   sopine_baseboard
			   orangepi_zero_plus2
			   mvebu_db-88f3720
	-       arc:  w+   hsdk
	+   powerpc:  w+   MPC8641HPCN_36BIT
	+                  MPC8610HPCD
	+                  MPC8641HPCN
	+                  xpedite517x
	+                  MCR3000
	+                  sbc8641d
	+       arc:  +    iot_devkit
	      nios2:  w+   10m50
	       mips:  w+   imgtec_xilfpga
			   bcm968380gerg_ram
	@@ -65,6 +72,7 @@
			   mk802_a10s
			   am43xx_hs_evm
			   ls1021aqds_nor_lpuart
	+                  h2200
			   mx7ulp_evk_plugin
			   Linksprite_pcDuino
			   display5
	@@ -101,6 +109,7 @@
			   q8_a23_tablet_800x480
			   A20-OLinuXino-Lime
			   imx6qdl_icore_mipi
	+                  zipitz2
			   ls1021atwr_nor_lpuart
			   k2e_hs_evm
			   ls1021atwr_sdcard_qspi
	@@ -139,6 +148,7 @@
			   Bananapro
			   Cubieboard4
			   ls1021aiot_sdcard
	+                  colibri_pxa270
			   ls1021aiot_qspi
			   Yones_Toptech_BS1078_V2
			   imx6qdl_icore_nand
	@@ -272,3 +282,5 @@
			   q8_a13_tablet
			   A13-OLinuXinoM
			   +   openrd_base openrd_client openrd_ultimate
	+     nds32:  +    adp-ag101p
	+                  adp-ae3xx

So we see 3 who new architectures are added: x86, powerpc, and nds32. And even with the existing architectures, 3 new targets are built, but arc switches from "hsdk" to "iot_devkit".

Is this meaningful?

Trevor Woerner

author

2018-11-21 08:31 UTC #8 lore raw

PATCH 2/3 buildman/toolchain.py: fix toolchain directory

In reply to Trevor Woerner

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in a way that is different from most toolchains. The first entry when unpacking most toolchain tarballs is:

gcc-<version>-nolib/<targetarch>-<system>

e.g.:

	gcc-8.1.0-nolibc/aarch64-linux/

The first entry of the hexagon toolchain, however, is:

	gcc-4.6.1-nolibc/

This causes the buildman logic in toolchain.py::ScanPath() to not be able to find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin} instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman tries to download a set of toolchains that includes hexagon, the script fails.

This update takes the second line of the tarball unpacking (which works for all the toolchains I've tested from kernel.org) and parses it to take the first two elements, separated by '/'. It makes this logic a bit more robust.

Signed-off-by Trevor Woerner <[email protected]>

---
 tools/buildman/toolchain.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

tools/buildman/toolchain.py +2 -1

index 442a7f977a..326c609a41 100644
@@ -502,7 +502,8 @@ class Toolchains:
             trailing /
         """
         stdout = command.Output('tar', 'xvfJ', fname, '-C', dest)
-        return stdout.splitlines()[0][:-1]
+        dirs = stdout.splitlines()[1].split('/')[:2]
+        return '/'.join(dirs)
 
     def TestSettingsHasPath(self, path):
         """Check if buildman will find this toolchain

Signature · 1 line

2.17.0.582.gccdcbd54c

Simon Glass

2018-11-27 01:02 UTC #9 lore raw

PATCH 2/3 buildman/toolchain.py: fix toolchain directory

Quoted 20 lines · Trevor Woerner

The hexagon toolchain (4.6.1) from kernel.org, for example, was packaged in a way that is different from most toolchains. The first entry when unpacking most toolchain tarballs is:
gcc-<version>-nolib/<targetarch>-<system>
e.g.:
        gcc-8.1.0-nolibc/aarch64-linux/
The first entry of the hexagon toolchain, however, is:
        gcc-4.6.1-nolibc/
This causes the buildman logic in toolchain.py::ScanPath() to not be able to find the "*gcc" executable since it looks in gcc-4.6.1-nolib/{.|bin|usr/bin} instead of gcc-4.6.1/hexagon-linux/{.|bin|usr/bin}. Therefore when buildman tries to download a set of toolchains that includes hexagon, the script fails.

Quoted 8 lines

This update takes the second line of the tarball unpacking (which works for all the toolchains I've tested from kernel.org) and parses it to take the first two elements, separated by '/'. It makes this logic a bit more robust.
Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by Simon Glass <[email protected]>

BTW I really appreciate your great commit msg.

- Simon

Trevor Woerner

author

2018-11-21 08:31 UTC #10 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

In reply to Trevor Woerner

Unfortunately, for some releases the kernel.org toolchain tarball names adhere to the following pattern:

	<hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz

e.g.:

	x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz

while others use the following pattern:

	<hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz

e.g.:

	x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz

Notice that the first pattern has dashes throughout, while the second has dashes throughout except just before the target architecture which has an underscore.

The "dash throughout" versions from kernel.org are:

	8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1

while the "dash and underscore" versions from kernel.org are:

	7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4

This tweak allows the code to handle both versions. Note that this tweak also causes the architecture parsing to get confused and find the following two bogus architectures, "2.0" and "64", which are explicitly checked for, and removed.

Signed-off-by Trevor Woerner <[email protected]>

---
 tools/buildman/toolchain.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

tools/buildman/toolchain.py +3 -2

index 326c609a41..4db70572e1 100644
@@ -517,13 +517,14 @@ class Toolchains:
     def ListArchs(self):
         """List architectures with available toolchains to download"""
         host_arch, archives = self.LocateArchUrl('list')
-        re_arch = re.compile('[-a-z0-9.]*_([^-]*)-.*')
+        re_arch = re.compile('[-a-z0-9.]*[-_]([^-]*)-.*')
         arch_set = set()
         for archive in archives:
             # Remove the host architecture from the start
             arch = re_arch.match(archive[len(host_arch):])
             if arch:
-                arch_set.add(arch.group(1))
+                if arch.group(1) != "2.0" and arch.group(1) != "64":
+                    arch_set.add(arch.group(1))
         return sorted(arch_set)
 
     def FetchAndInstall(self, arch):

Signature · 1 line

2.17.0.582.gccdcbd54c

Simon Glass

2018-11-27 01:02 UTC #11 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

Quoted 34 lines · Trevor Woerner

Unfortunately, for some releases the kernel.org toolchain tarball names adhere to the following pattern:
        <hostarch>-gcc-<ver>-nolib-<targetarch>-<type>.tar.xz
e.g.:
        x86_64-gcc-8.1.0-nolibc-aarch64-linux.tar.xz
while others use the following pattern:
        <hostarch>-gcc-<ver>-nolib_<targetarch>-<type>.tar.xz
e.g.:
        x86_64-gcc-7.3.0-nolibc_aarch64-linux.tar.xz
Notice that the first pattern has dashes throughout, while the second has dashes throughout except just before the target architecture which has an underscore.
The "dash throughout" versions from kernel.org are:
        8.1.0, 6.4.0, 5.5.0, 4.9.4, 4.8.5, 4.6.1
while the "dash and underscore" versions from kernel.org are:
        7.3.0, 4.9.0, 4.8.0, 4.7.3, 4.6.3, 4.6.2, 4.5.1, 4.2.4
This tweak allows the code to handle both versions. Note that this tweak also causes the architecture parsing to get confused and find the following two bogus architectures, "2.0" and "64", which are explicitly checked for, and removed.

Quoted 4 lines

Signed-off-by Trevor Woerner <[email protected]>
---
 tools/buildman/toolchain.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Reviewed-by Simon Glass <[email protected]>

But please use single quotes for strings.

Trevor Woerner

2018-11-28 00:51 UTC #12 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

Quoted 3 lines · Simon Glass

Reviewed-by Simon Glass <[email protected]>
But please use single quotes for strings.

I can send a v2 if you like?

Simon Glass

2018-11-28 19:38 UTC #13 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

Hi Trevor,

Quoted 8 lines · Trevor Woerner

Quoted 3 lines · Simon Glass
Reviewed-by Simon Glass <[email protected]>
But please use single quotes for strings.
I can send a v2 if you like?

If you like, but I can fix it up when applying.

- Simon

Trevor Woerner

2018-11-28 19:54 UTC #14 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

Quoted 6 lines · Simon Glass

Hi Trevor,
Quoted 3 lines · Trevor Woerner
Quoted 1 line · Simon Glass

Quoted 9 lines

Quoted 5 lines
Quoted 2 lines
Reviewed-by Simon Glass <[email protected]>
But please use single quotes for strings.
I can send a v2 if you like?
If you like, but I can fix it up when applying.

Yes, please go ahead and fix it up.

sjg at google.com

2018-12-05 23:11 UTC #15 lore raw

PATCH 3/3 buildman/toolchain.py: handle inconsistent tarball names

Quoted 6 lines · Simon Glass

Hi Trevor,
Quoted 3 lines · Trevor Woerner
Quoted 1 line · Simon Glass

Quoted 9 lines

Quoted 5 lines
Quoted 2 lines
Reviewed-by Simon Glass <[email protected]>
But please use single quotes for strings.
I can send a v2 if you like?
If you like, but I can fix it up when applying.

Yes, please go ahead and fix it up.

Applied to u-boot-dm/master, thanks!