Skip to contentSkip to conversation

x86/purgatory: Remove LTO flags

PATCH Superseded Song Liu 2 participants 6 messages Sep 8, 2023 – Sep 14, 2023

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Song Liu

author
#0 lore raw

PATCH v2 1/1 x86/purgatory: Remove LTO flags

With LTO enabled, ld.lld generates multiple .text sections for purgatory.ro:

$ readelf -S purgatory.ro  | grep " .text"
  [ 1] .text             PROGBITS         0000000000000000  00000040
  [ 7] .text.purgatory   PROGBITS         0000000000000000  000020e0
  [ 9] .text.warn        PROGBITS         0000000000000000  000021c0
  [13] .text.sha256_upda PROGBITS         0000000000000000  000022f0
  [15] .text.sha224_upda PROGBITS         0000000000000000  00002be0
  [17] .text.sha256_fina PROGBITS         0000000000000000  00002bf0
  [19] .text.sha224_fina PROGBITS         0000000000000000  00002cc0

This cause WARNING from kexec_purgatory_setup_sechdrs():

WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
kexec_load_purgatory+0x37f/0x390

Fix this by disabling LTO for purgatory.

Fixes 8652d44f466a ("kexec: support purgatories with .text.hot sections")
Cc Ricardo Ribalda <[email protected]>
Cc Sami Tolvanen <[email protected]>

---
AFAICT, x86 is the only arch that supports LTO and purgatory.

Changes in v2:
1. Use CC_FLAGS_LTO instead of hardcode -flto. (Nick Desaulniers)

---
 arch/x86/purgatory/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/purgatory/Makefile +4 -0
index c2a29be35c01..08aa0f25f12a 100644
@@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY
 # optimization flags.
 KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
 
+# When LTO is enabled, llvm emits many text sections, which is not supported
+# by kexec. Remove -flto=* flags.
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
Signature · 1 line
2.34.1
#1 lore raw
Quoted 13 lines · Song Liu

With LTO enabled, ld.lld generates multiple .text sections for purgatory.ro:

$ readelf -S purgatory.ro  | grep " .text"
  [ 1] .text             PROGBITS         0000000000000000  00000040
  [ 7] .text.purgatory   PROGBITS         0000000000000000  000020e0
  [ 9] .text.warn        PROGBITS         0000000000000000  000021c0
  [13] .text.sha256_upda PROGBITS         0000000000000000  000022f0
  [15] .text.sha224_upda PROGBITS         0000000000000000  00002be0
  [17] .text.sha256_fina PROGBITS         0000000000000000  00002bf0
  [19] .text.sha224_fina PROGBITS         0000000000000000  00002cc0
Quoted 6 lines

This cause WARNING from kexec_purgatory_setup_sechdrs():

WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
kexec_load_purgatory+0x37f/0x390

Fix this by disabling LTO for purgatory.

Thanks for the v2!

Quoted 2 lines
Fixes 8652d44f466a ("kexec: support purgatories with .text.hot sections")

Dunno that this fixes tag is precise. I think perhaps

Fixes b33fff07e3e3 ("x86, build: allow LTO to be selected")

would be more precise.

Quoted 22 lines

---
AFAICT, x86 is the only arch that supports LTO and purgatory.

Changes in v2:
1. Use CC_FLAGS_LTO instead of hardcode -flto. (Nick Desaulniers)

---
 arch/x86/purgatory/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/purgatory/Makefile +0 -0
index c2a29be35c01..08aa0f25f12a 100644

arch/x86/purgatory/Makefile CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY · line 23

Quoted 5 lines
arch/x86/purgatory/Makefile +2 -0
@@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY
 # optimization flags.
 KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
 
+# When LTO is enabled, llvm emits many text sections, which is not supported
+# by kexec. Remove -flto=* flags.

-flto* in LLVM implies -ffunction-sections, which creates a .text.<fn name> section per function definition to facilitate -Wl,--gc-sections.

Overall the question here is "do we really need to optimize kexec?"

If the answer is yes, then this patch AND 8652d44f466a are both pessimizing kexec (though having it work at all is strictly better than not at all). The best fix IMO would be to provide a linker script for this purgatory image that rejoins the text sections back into one .text. For example:

commit eff8728fe698 ("vmlinux.lds.h: Add PGO and AutoFDO input sections")

I assume people do care about the time to kexec, hence the raison d'etre for projects like kpatch.

I'm fine to sign off on this approach if we don't really care, or want to care more later, but we can do better here.

Quoted 8 lines
arch/x86/purgatory/Makefile +2 -1
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
--

2.34.1

Signature · 2 lines
Thanks,
~Nick Desaulniers
author
#2 lore raw
Quoted 22 lines · Nick Desaulniers
Quoted 13 lines · Song Liu

With LTO enabled, ld.lld generates multiple .text sections for purgatory.ro:

$ readelf -S purgatory.ro  | grep " .text"
  [ 1] .text             PROGBITS         0000000000000000  00000040
  [ 7] .text.purgatory   PROGBITS         0000000000000000  000020e0
  [ 9] .text.warn        PROGBITS         0000000000000000  000021c0
  [13] .text.sha256_upda PROGBITS         0000000000000000  000022f0
  [15] .text.sha224_upda PROGBITS         0000000000000000  00002be0
  [17] .text.sha256_fina PROGBITS         0000000000000000  00002bf0
  [19] .text.sha224_fina PROGBITS         0000000000000000  00002cc0
Quoted 6 lines

This cause WARNING from kexec_purgatory_setup_sechdrs():

WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
kexec_load_purgatory+0x37f/0x390

Fix this by disabling LTO for purgatory.

Quoted 8 lines

Thanks for the v2!

Quoted 2 lines
Fixes 8652d44f466a ("kexec: support purgatories with .text.hot sections")

Dunno that this fixes tag is precise. I think perhaps

Fixes b33fff07e3e3 ("x86, build: allow LTO to be selected")

Thanks for the correction!

Quoted 42 lines

would be more precise.

Quoted 22 lines

---
AFAICT, x86 is the only arch that supports LTO and purgatory.

Changes in v2:
1. Use CC_FLAGS_LTO instead of hardcode -flto. (Nick Desaulniers)

---
 arch/x86/purgatory/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/purgatory/Makefile +0 -0
index c2a29be35c01..08aa0f25f12a 100644

arch/x86/purgatory/Makefile CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY · line 23

Quoted 5 lines
arch/x86/purgatory/Makefile +2 -0
@@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY
 # optimization flags.
 KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
 
+# When LTO is enabled, llvm emits many text sections, which is not supported
+# by kexec. Remove -flto=* flags.

-flto* in LLVM implies -ffunction-sections, which creates a .text.<fn name> section per function definition to facilitate -Wl,--gc-sections.

Overall the question here is "do we really need to optimize kexec?"

If the answer is yes, then this patch AND 8652d44f466a are both pessimizing kexec (though having it work at all is strictly better than not at all). The best fix IMO would be to provide a linker script for this purgatory image that rejoins the text sections back into one .text. For example:

Quoted 4 lines

commit eff8728fe698 ("vmlinux.lds.h: Add PGO and AutoFDO input sections")

I assume people do care about the time to kexec, hence the raison d'etre for projects like kpatch.

AFAICT, optimizations like LTO and PGO can give a few % of improvement, which is probably not important for kexec. The benefit is in the order of seconds (or less?). The benefit of kpatch is that we can keep the workload running while fixing the kernel bug. Based on our experience at Meta, it may take hours to graceful shutdown the application to run kexec. In this case, a few seconds of improvement (via LTO/PGO purgatory) doesn't save us much.

Thanks,
Song

Quoted 17 lines

I'm fine to sign off on this approach if we don't really care, or want to care more later, but we can do better here.

Quoted 8 lines
arch/x86/purgatory/Makefile +2 -1
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
--

2.34.1

--
Thanks,
~Nick Desaulniers

author
#3 lore raw
In reply to Song Liu

Hi folks,

Quoted 20 lines · Song Liu

With LTO enabled, ld.lld generates multiple .text sections for purgatory.ro:

$ readelf -S purgatory.ro  | grep " .text"
  [ 1] .text             PROGBITS         0000000000000000  00000040
  [ 7] .text.purgatory   PROGBITS         0000000000000000  000020e0
  [ 9] .text.warn        PROGBITS         0000000000000000  000021c0
  [13] .text.sha256_upda PROGBITS         0000000000000000  000022f0
  [15] .text.sha224_upda PROGBITS         0000000000000000  00002be0
  [17] .text.sha256_fina PROGBITS         0000000000000000  00002bf0
  [19] .text.sha224_fina PROGBITS         0000000000000000  00002cc0

This cause WARNING from kexec_purgatory_setup_sechdrs():

WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
kexec_load_purgatory+0x37f/0x390

Fix this by disabling LTO for purgatory.

Quoted 8 lines
Fixes 8652d44f466a ("kexec: support purgatories with .text.hot sections")
Cc Ricardo Ribalda <[email protected]>
Cc Sami Tolvanen <[email protected]>

What would be the next step for this work? We hope to back port it to our production kernel soon.

Thanks,
Song

Quoted 28 lines

---
AFAICT, x86 is the only arch that supports LTO and purgatory.

Changes in v2:
1. Use CC_FLAGS_LTO instead of hardcode -flto. (Nick Desaulniers)

---
 arch/x86/purgatory/Makefile | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/purgatory/Makefile +4 -1
index c2a29be35c01..08aa0f25f12a 100644
@@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS -D__NO_FORTIFY
 # optimization flags.
 KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
 
+# When LTO is enabled, llvm emits many text sections, which is not supported
+# by kexec. Remove -flto=* flags.
+KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO),$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib
--

2.34.1

#4 lore raw
Quoted 3 lines · Song Liu

What would be the next step for this work? We hope to back port it to our production kernel soon.

Please send a v3 with the fixes tag updated. I wouldn't mind if you added a comment to the commit message to the effect of:

We could also add the use of an explicit linker script to rejoin .text.* sections back into .text. Simply disable the production of more .text.* sections for now; -flto* implies -ffunction-sections.

Signature · 2 lines
Thanks,
~Nick Desaulniers
author
#5 lore raw
Quoted 6 lines · Nick Desaulniers
Quoted 3 lines · Song Liu

What would be the next step for this work? We hope to back port it to our production kernel soon.

Quoted 6 lines

Please send a v3 with the fixes tag updated. I wouldn't mind if you added a comment to the commit message to the effect of:

We could also add the use of an explicit linker script to rejoin .text.* sections back into .text. Simply disable the production of more .text.* sections for now; -flto* implies -ffunction-sections.

Got it. I just sent v3 with these changes.

Thanks,
Song