Skip to contentSkip to conversation

RISC-V: build: Allow LTO to be selected

PATCH In review Wende Tan 5 participants 10 messages May 12, 2022 – Oct 28, 2022 RISC-V

lore
Share & tools
lore

Wende Tan

author
#0 lore raw

PATCH v2 1/1 RISC-V: build: Allow LTO to be selected

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -0
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
 	select ARCH_SUPPORTS_ATOMIC_RMW
 	select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
 	select ARCH_SUPPORTS_HUGETLBFS if MMU
+	# LLD >= 14:
+	# - https://github.com/llvm/llvm-project/issues/50505
+	select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+	select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
 	select ARCH_USE_MEMTEST
 	select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
 	select ARCH_WANT_FRAME_POINTERS
Signature · 1 line
2.25.1
Tested-by on v2 1/1
#1 lore raw
Quoted 11 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

WDYT?

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Quoted 33 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -1
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
author
#2 lore raw
Quoted 33 lines · Nick Desaulniers
Quoted 10 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

Quoted 6 lines

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

WDYT?

It seems that when LTO is enabled, the compiler (clang) only generates bitcode, and LLD collects all the bitcode, does LTO, and generates machine code (assembly) and ELF object files? Actually, the error message in #50505 was originally reported by LLD. So I'd like to check LLD_VERSION here.

Quoted 5 lines

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Aha, that patch is for enabling LTO for allyesconfig (especially, with xfs and overlayfs). I think that patch is not mandatory for enabling LTO for RISC-V.

Quoted 4 lines

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Thanks,
Wende

Quoted 32 lines
Quoted 32 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -1
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

#3 lore raw
Quoted 40 lines · On Fri, May 13, 2022 at 11:28 AM twd2 <[email protected]> wrote:
Quoted 38 lines · Nick Desaulniers
Quoted 10 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

Quoted 4 lines
Quoted 1 line

WDYT?

It seems that when LTO is enabled, the compiler (clang) only generates bitcode, and LLD collects all the bitcode, does LTO, and generates machine code (assembly) and ELF object files? Actually, the error message in #50505 was originally reported by LLD. So I'd like to check LLD_VERSION here.

lld calls into llvm/lib/LTO which in turn uses llvm/lib/MC to write a relocatable object file.
In this sense, it's the lld version which matters, but I think use either clang/lld version is ok.

Technically one can use clang 13 and do LTO with lld 14 thanks to bitcode compatibility. In this sense the lib/MC specific change only requires lld version.

Quoted 52 lines
Quoted 5 lines

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Aha, that patch is for enabling LTO for allyesconfig (especially, with xfs and overlayfs). I think that patch is not mandatory for enabling LTO for RISC-V.

Quoted 4 lines

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Thanks,
Wende

Quoted 32 lines
Quoted 32 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -1
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

Signature · 1 line
宋方睿
Reviewed-by
#4 lore raw
In reply to twd2
Quoted 40 lines · On Fri, May 13, 2022 at 11:28 AM twd2 <[email protected]> wrote:
Quoted 38 lines · Nick Desaulniers
Quoted 10 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

Quoted 4 lines
Quoted 1 line

WDYT?

It seems that when LTO is enabled, the compiler (clang) only generates bitcode, and LLD collects all the bitcode, does LTO, and generates machine code (assembly) and ELF object files? Actually, the error message in #50505 was originally reported by LLD. So I'd like to check LLD_VERSION here.

Ah, that's right. Ok, LGTM.

Quoted 54 lines
Quoted 7 lines

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Aha, that patch is for enabling LTO for allyesconfig (especially, with xfs and overlayfs). I think that patch is not mandatory for enabling LTO for RISC-V.

Quoted 4 lines

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Thanks,
Wende

Quoted 32 lines
Quoted 32 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +4 -1
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
Reviewed-by
#5 lore raw
Quoted 42 lines · Nick Desaulniers
Quoted 40 lines · On Fri, May 13, 2022 at 11:28 AM twd2 <[email protected]> wrote:
Quoted 38 lines · Nick Desaulniers
Quoted 10 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

WDYT?

It seems that when LTO is enabled, the compiler (clang) only generates bitcode, and LLD collects all the bitcode, does LTO, and generates machine code (assembly) and ELF object files? Actually, the error message in #50505 was originally reported by LLD. So I'd like to check LLD_VERSION here.

Ah, that's right. Ok, LGTM.

Quoted 60 lines
Quoted 46 lines
Quoted 7 lines

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Aha, that patch is for enabling LTO for allyesconfig (especially, with xfs and overlayfs). I think that patch is not mandatory for enabling LTO for RISC-V.

Quoted 4 lines

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Thanks,
Wende

Quoted 24 lines
Quoted 24 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +2 -0
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
arch/riscv/Kconfig +2 -1
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

--
Thanks,
~Nick Desaulniers

Signature · 1 line
宋方睿
author
#6 lore raw

Hi Palmer,

ping?

Thanks
Wende

Quoted 107 lines · Fāng-ruì Sòng
Quoted 40 lines · Nick Desaulniers
Quoted 39 lines · On Fri, May 13, 2022 at 11:28 AM twd2 <[email protected]> wrote:
Quoted 38 lines · Nick Desaulniers
Quoted 10 lines · Wende Tan

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server,

Haha! That's ok, allyesconfig is not expected to boot for any architectures AFAIK. For CI, we simply verify we can build them; we boot test everything but allyesconfig and allmodconfig (and architectures which don't yet have qemu ports). It helps detect when assembler sources don't use enough encoding space when referenced external symbols are too far away for larger images, IME.

That's a long time; first time I've seen a number from someone trying to LTO allyesconfig!

Quoted 7 lines

successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

So looking at that change, it doesn't look like it touches LLD at all. To me it touches both the assembler and the object streamer, which is used by the compiler (and assembler) to stream instructions directly into an object file (without the need for an external assembler).

That makes me think that
CLANG_VERSION >= 140000
would be more appropriate than
LLD_VERSION >= 140000

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

WDYT?

It seems that when LTO is enabled, the compiler (clang) only generates bitcode, and LLD collects all the bitcode, does LTO, and generates machine code (assembly) and ELF object files? Actually, the error message in #50505 was originally reported by LLD. So I'd like to check LLD_VERSION here.

Ah, that's right. Ok, LGTM.

Quoted 55 lines
Quoted 42 lines
Quoted 5 lines

Also, I'm curious if the LLVM patch you had me commit for you recently is at all related or necessary for this? If so, then the version check should probably be against clang-15, not clang-14.
https://github.com/llvm/llvm-project/commit/6baaad740a5abb4bfcff022a8114abb4eea66a2d

Aha, that patch is for enabling LTO for allyesconfig (especially, with xfs and overlayfs). I think that patch is not mandatory for enabling LTO for RISC-V.

Quoted 4 lines

Anyways, I just did a build+boot (in qemu) test of defconfig+thinlto and defconfig+lto. LGTM

Thanks,
Wende

Quoted 24 lines
Quoted 24 lines

---
v2:
- Some textual changes suggested by Nick.
- Drop the changes to arch/riscv/Makefile, since the LLVM issue is filed and resolved.
- Drop the unnecessary changes to arch/riscv/kernel/vdso/Makefile.

v1: https://lore.kernel.org/linux-riscv/[email protected]/

---
 arch/riscv/Kconfig | 4 ++++
 1 file changed, 4 insertions(+)
arch/riscv/Kconfig +2 -0
index 00fd9c548f26..c55f6b95e5af 100644
@@ -38,6 +38,10 @@ config RISCV
        select ARCH_SUPPORTS_ATOMIC_RMW
        select ARCH_SUPPORTS_DEBUG_PAGEALLOC if MMU
        select ARCH_SUPPORTS_HUGETLBFS if MMU
+       # LLD >= 14:
+       # - https://github.com/llvm/llvm-project/issues/50505
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
arch/riscv/Kconfig +2 -1
+       select ARCH_SUPPORTS_LTO_CLANG if LLD_VERSION >= 140000
+       select ARCH_SUPPORTS_LTO_CLANG_THIN if LLD_VERSION >= 140000
        select ARCH_USE_MEMTEST
        select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU
        select ARCH_WANT_FRAME_POINTERS
--

2.25.1

--
Thanks,
~Nick Desaulniers

Signature · 4 lines
Best wishes!
Sincerely,
twd2
https://twd2.me
maintainer
#7 lore raw
In reply to Wende Tan
Quoted 15 lines

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

Quoted 5 lines

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

Sorry for missing this, the v2 never made it to my inbox. Not sure exactly what happened, but an off-list ping made it through. I've put this on for-next, I don't have any way to test it because I don't have clang setup yet.

author
#8 lore raw

Never mind. Thanks for your reply!

But I find Nick's Tested-by tag is missing?
https://lore.kernel.org/all/CAKwvOdmJ26h4L=3+42xeWG0_mZmZXkye=+mNWgFOHbo59ZVz1w@mail.gmail.com/

Quoted 26 lines · Palmer Dabbelt
Quoted 16 lines

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

Quoted 5 lines

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

Sorry for missing this, the v2 never made it to my inbox. Not sure exactly what happened, but an off-list ping made it through. I've put this on for-next, I don't have any way to test it because I don't have clang setup yet.

#9 lore raw
In reply to Palmer Dabbelt
Quoted 18 lines · Palmer Dabbelt
Quoted 18 lines

Allow LTO to be selected for RISC-V, only when LLD >= 14, since there is an issue [1] in prior LLD versions that prevents LLD to generate proper machine code for RISC-V when writing nops.

I have tested enabling LTO for defconfig. The LLD took ~2m21s and ~3GiB on our Intel Xeon Gold 6140 server and produced an 18MiB Image. The image can boot to shell using an archriscv rootfs on QEMU.

I have also tested it for allyesconfig without COMPILE_TEST, FTRACE, KASAN, and GCOV. The LLD took ~7h03m and ~335GiB on the server, successfully producing a 1.7GiB Image. Unfortunately, we cannot boot this image because the create_kernel_page_table() -> alloc_pmd_early() -> BUG_ON() logic limits the image to be < 1GiB. Maybe we can fix it in a separate patch further.

[1] https://github.com/llvm/llvm-project/issues/50505, resolved by LLVM commit e63455d5e0e5 ("[MC] Use local MCSubtargetInfo in writeNops")

Quoted 8 lines
Quoted 2 lines

Sorry for missing this, the v2 never made it to my inbox. Not sure exactly what happened, but an off-list ping made it through. I've put this on for-next, I don't have any way to test it because I don't have clang setup yet.

Did this get dropped by accident? I do not see it in your tree.

Cheers,
Nathan