Skip to contentSkip to conversation

build/xen: fix symbol generation with LLVM LD

PATCH In review Roger Pau Monne 3 participants 9 messages May 5, 2022 – Jul 6, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 build/xen: fix symbol generation with LLVM LD

Current LLVM LD implementation will turn global hidden symbols in object files into local ones when generating the .symtab of the Xen binary image.

This is different from GNU ld implementation, that will only do the conversion (or remove the symbols) when generation .dynsym but not .symtab. Such conversion breaks the processing of symbols done by tools/symbols.

Use protected symbol visibility instead of hidden, as that preserves the symbol binding while not generating GOT or PLT indirections that are not compatible with some of the inline assembly constructs currently used.

While there also make the visibility setting compiler support
non-optional: compilers not supporting it won't be able to build Xen anyway, and will just throw a compiler error sooner rather than later during the build.

---
 xen/Kconfig                | 4 ----
 xen/include/xen/compiler.h | 9 +++++----
 2 files changed, 5 insertions(+), 8 deletions(-)
xen/Kconfig +0 -4
index 134e6e68ad..a9182fb13d 100644
@@ -29,10 +29,6 @@ config LD_IS_GNU
 config LD_IS_LLVM
 	def_bool $(success,$(LD) --version | head -n 1 | grep -q "^LLD")
 
-# -fvisibility=hidden reduces -fpic cost, if it's available
-config CC_HAS_VISIBILITY_ATTRIBUTE
-	def_bool $(cc-option,-fvisibility=hidden)
-
 # Use -f{function,data}-sections compiler parameters
 config CC_SPLIT_SECTIONS
 	bool
xen/include/xen/compiler.h +5 -4
index 933aec09a9..c144b17217 100644
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )
Signature · 1 line
2.36.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 14 lines · Roger Pau Monne

Current LLVM LD implementation will turn global hidden symbols in object files into local ones when generating the .symtab of the Xen binary image.

This is different from GNU ld implementation, that will only do the conversion (or remove the symbols) when generation .dynsym but not .symtab. Such conversion breaks the processing of symbols done by tools/symbols.

Use protected symbol visibility instead of hidden, as that preserves the symbol binding while not generating GOT or PLT indirections that are not compatible with some of the inline assembly constructs currently used.

Quoted 6 lines

While there also make the visibility setting compiler support
non-optional: compilers not supporting it won't be able to build Xen anyway, and will just throw a compiler error sooner rather than later during the build.

#2 lore raw
In reply to Roger Pau Monne
Quoted 8 lines · Roger Pau Monne
xen/include/xen/compiler.h +0 -2
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */

xen/include/xen/compiler.h line 135

Quoted 10 lines
xen/include/xen/compiler.h +5 -2
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Jan

#3 lore raw
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Roger Pau Monne
xen/include/xen/compiler.h +5 -4
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
Quoted 6 lines

xen/include/xen/compiler.h line 135

Quoted 2 lines
xen/include/xen/compiler.h +0 -0
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Hm, great. I guess I will have to use protected with clang and hidden with gcc then, for lack of a better solution.

I'm slightly confused as to why my godbolt example:

https://godbolt.org/z/chTnMWxeP

Seems to work with gcc 11 then. I will have to investigate a bit I think.

Thanks, and sorry for the trouble.

#4 lore raw
Quoted 28 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Roger Pau Monne
xen/include/xen/compiler.h +5 -4
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
Quoted 6 lines

xen/include/xen/compiler.h line 135

Quoted 2 lines
xen/include/xen/compiler.h +0 -0
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Hm, great. I guess I will have to use protected with clang and hidden with gcc then, for lack of a better solution.

Quoted 6 lines

I'm slightly confused as to why my godbolt example:

https://godbolt.org/z/chTnMWxeP

Seems to work with gcc 11 then. I will have to investigate a bit I think.

So it seems the problem is explicitly with constructs like:

void (*foo)(void);

void test(void)

{
    asm volatile (".long [addr]" :: [addr] "i" (&(foo)));
}

See:

https://godbolt.org/z/TYqeGdWsn

AFAICT gcc will consider the function pointer foo to go through the GOT/PLT redirection table, while clang will not. I think gcc behavior is correct because in theory foo could be set from a different module? protect only guarantees that references to local functions cannot be overwritten, but not external ones.

I don't really see a good way to fix this, rather that setting different visibilities based on the compiler. clang would use protected and gcc would use hidden. I think it's unlikely to have a toolstack setup to use gcc as the compiler and LLVM LD as the linker, which would be the problematic configuration, and even in that case it's kind of a cosmetic issue with symbol resolution, binary output from the linker would still be correct.

Let me know if that seems acceptable.

Thanks, Roger.

#5 lore raw
Quoted 47 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Roger Pau Monne
xen/include/xen/compiler.h +5 -4
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
Quoted 6 lines

xen/include/xen/compiler.h line 135

Quoted 2 lines
xen/include/xen/compiler.h +0 -0
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Hm, great. I guess I will have to use protected with clang and hidden with gcc then, for lack of a better solution.

Quoted 6 lines

I'm slightly confused as to why my godbolt example:

https://godbolt.org/z/chTnMWxeP

Seems to work with gcc 11 then. I will have to investigate a bit I think.

So it seems the problem is explicitly with constructs like:

void (*foo)(void);

void test(void)

{
    asm volatile (".long [addr]" :: [addr] "i" (&(foo)));
}

See:

Quoted 7 lines

https://godbolt.org/z/TYqeGdWsn

AFAICT gcc will consider the function pointer foo to go through the GOT/PLT redirection table, while clang will not. I think gcc behavior is correct because in theory foo could be set from a different module? protect only guarantees that references to local functions cannot be overwritten, but not external ones.

Right, since there's no way to tell the compiler that the symbol will be resolved in the same "module".

Quoted 3 lines

I don't really see a good way to fix this, rather that setting different visibilities based on the compiler. clang would use protected and gcc would use hidden.

If gcc's behavior is indeed correct, then moving to protected with clang would set us up for going through GOT/PLT there - either right away (if the implement this like gcc), or once they correct their behavior. I don't think we want that. Therefore I think we want to alter visibility between compilation and linking (i.e. presumably right in prelink.o), going from compile-time hidden to link-time protected. That would likely be closer to what your original patch did (sadly there's no "convert <visibility1> to <visibility2> option to objcopy, and making it have one wouldn't really help us here; it's also not clear to me whether llvm comes with its own objcopy, or whether they re-use GNU's).

Quoted 5 lines

I think it's unlikely to have a
toolstack setup to use gcc as the compiler and LLVM LD as the linker, which would be the problematic configuration, and even in that case it's kind of a cosmetic issue with symbol resolution, binary output from the linker would still be correct.

While likely moot with the above, I agree that we could make such an assumption if need be.

Jan

Quoted 4 lines

Let me know if that seems acceptable.

Thanks, Roger.

#6 lore raw
Quoted 63 lines · Jan Beulich
Quoted 47 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Roger Pau Monne
xen/include/xen/compiler.h +5 -4
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
Quoted 6 lines

xen/include/xen/compiler.h line 135

Quoted 2 lines
xen/include/xen/compiler.h +0 -0
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Hm, great. I guess I will have to use protected with clang and hidden with gcc then, for lack of a better solution.

Quoted 6 lines

I'm slightly confused as to why my godbolt example:

https://godbolt.org/z/chTnMWxeP

Seems to work with gcc 11 then. I will have to investigate a bit I think.

So it seems the problem is explicitly with constructs like:

void (*foo)(void);

void test(void)

{
    asm volatile (".long [addr]" :: [addr] "i" (&(foo)));
}

See:

Quoted 7 lines

https://godbolt.org/z/TYqeGdWsn

AFAICT gcc will consider the function pointer foo to go through the GOT/PLT redirection table, while clang will not. I think gcc behavior is correct because in theory foo could be set from a different module? protect only guarantees that references to local functions cannot be overwritten, but not external ones.

Right, since there's no way to tell the compiler that the symbol will be resolved in the same "module".

Quoted 3 lines

I don't really see a good way to fix this, rather that setting different visibilities based on the compiler. clang would use protected and gcc would use hidden.

Quoted 11 lines

If gcc's behavior is indeed correct, then moving to protected with clang would set us up for going through GOT/PLT there - either right away (if the implement this like gcc), or once they correct their behavior. I don't think we want that. Therefore I think we want to alter visibility between compilation and linking (i.e. presumably right in prelink.o), going from compile-time hidden to link-time protected. That would likely be closer to what your original patch did (sadly there's no "convert <visibility1> to <visibility2> option to objcopy, and making it have one wouldn't really help us here; it's also not clear to me whether llvm comes with its own objcopy, or whether they re-use GNU's).

So I've raised the difference in protected behavior between gcc and clang:

https://discourse.llvm.org/t/gcc-vs-clang-differences-in-protected-visibility-implementation

It's no clear to me whether clang would switch it's implementation, but it also seems fragile to rely on global protected function pointers not going through the GOT.

Do you have any recommendation as to how to change symbol visibility? I've been looking at objcopy, but I don't seem to find a way to do it.

Thanks, Roger.

#7 lore raw
Quoted 76 lines · Roger Pau Monné
Quoted 63 lines · Jan Beulich
Quoted 47 lines · Roger Pau Monné
Quoted 28 lines · Roger Pau Monné
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Roger Pau Monne
xen/include/xen/compiler.h +5 -4
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a), typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
Quoted 6 lines

xen/include/xen/compiler.h line 135

Quoted 2 lines
xen/include/xen/compiler.h +0 -0
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */
 #define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )

This has failed my pre-push build test, with massive amounts of errors about asm() constraints in the alternative call infrastructure. This was with gcc 11.3.0.

Hm, great. I guess I will have to use protected with clang and hidden with gcc then, for lack of a better solution.

Quoted 6 lines

I'm slightly confused as to why my godbolt example:

https://godbolt.org/z/chTnMWxeP

Seems to work with gcc 11 then. I will have to investigate a bit I think.

So it seems the problem is explicitly with constructs like:

void (*foo)(void);

void test(void)

{
    asm volatile (".long [addr]" :: [addr] "i" (&(foo)));
}

See:

Quoted 7 lines

https://godbolt.org/z/TYqeGdWsn

AFAICT gcc will consider the function pointer foo to go through the GOT/PLT redirection table, while clang will not. I think gcc behavior is correct because in theory foo could be set from a different module? protect only guarantees that references to local functions cannot be overwritten, but not external ones.

Right, since there's no way to tell the compiler that the symbol will be resolved in the same "module".

Quoted 3 lines

I don't really see a good way to fix this, rather that setting different visibilities based on the compiler. clang would use protected and gcc would use hidden.

Quoted 11 lines

If gcc's behavior is indeed correct, then moving to protected with clang would set us up for going through GOT/PLT there - either right away (if the implement this like gcc), or once they correct their behavior. I don't think we want that. Therefore I think we want to alter visibility between compilation and linking (i.e. presumably right in prelink.o), going from compile-time hidden to link-time protected. That would likely be closer to what your original patch did (sadly there's no "convert <visibility1> to <visibility2> option to objcopy, and making it have one wouldn't really help us here; it's also not clear to me whether llvm comes with its own objcopy, or whether they re-use GNU's).

Quoted 8 lines

So I've raised the difference in protected behavior between gcc and clang:

https://discourse.llvm.org/t/gcc-vs-clang-differences-in-protected-visibility-implementation

It's no clear to me whether clang would switch it's implementation, but it also seems fragile to rely on global protected function pointers not going through the GOT.

I agree, and I don't view it as likely that they would change their code. GNU objcopy offers a separate step for that conversion (--localize-hidden), but ..

Quoted 3 lines

Do you have any recommendation as to how to change symbol visibility? I've been looking at objcopy, but I don't seem to find a way to do it.

... kind of unexpectedly doesn't offer means to alter visibility. So I guess you/we need to turn back to your original RFC approach, no matter that it wasn't really pretty.

Jan

#8 lore raw
In reply to Roger Pau Monne

Hi,

It seems that this patch has been stale for more than 2 months and from the discussion in this thread I think some actions need to be taken from the author. So I am sending this email as a gentle reminder.

Kind regards,
Henry

Quoted 67 lines

-----Original Message-----
Subject: [PATCH] build/xen: fix symbol generation with LLVM LD

Current LLVM LD implementation will turn global hidden symbols in object files into local ones when generating the .symtab of the Xen binary image.

This is different from GNU ld implementation, that will only do the conversion (or remove the symbols) when generation .dynsym but not .symtab. Such conversion breaks the processing of symbols done by tools/symbols.

Use protected symbol visibility instead of hidden, as that preserves the symbol binding while not generating GOT or PLT indirections that are not compatible with some of the inline assembly constructs currently used.

While there also make the visibility setting compiler support
non-optional: compilers not supporting it won't be able to build Xen anyway, and will just throw a compiler error sooner rather than later during the build.

---
 xen/Kconfig                | 4 ----
 xen/include/xen/compiler.h | 9 +++++----
 2 files changed, 5 insertions(+), 8 deletions(-)
xen/Kconfig +0 -4
index 134e6e68ad..a9182fb13d 100644
@@ -29,10 +29,6 @@ config LD_IS_GNU
 config LD_IS_LLVM
 	def_bool $(success,$(LD) --version | head -n 1 | grep -q "^LLD")
 
-# -fvisibility=hidden reduces -fpic cost, if it's available
-config CC_HAS_VISIBILITY_ATTRIBUTE
-	def_bool $(cc-option,-fvisibility=hidden)
-
 # Use -f{function,data}-sections compiler parameters
 config CC_SPLIT_SECTIONS
 	bool
xen/include/xen/compiler.h +5 -4
index 933aec09a9..c144b17217 100644
@@ -125,10 +125,11 @@
 #define __must_be_array(a) \
   BUILD_BUG_ON_ZERO(__builtin_types_compatible_p(typeof(a),
 typeof(&a[0])))
 
-#ifdef CONFIG_CC_HAS_VISIBILITY_ATTRIBUTE
-/* Results in more efficient PIC code (no indirections through GOT or PLT). */
-#pragma GCC visibility push(hidden)
-#endif
+/*
+ * Results in more efficient PIC code (no indirections through GOT or PLT)
+ * and is also required by some of the assembly constructs.
+ */
+#pragma GCC visibility push(protected)
 
 /* Make the optimizer believe the variable can be manipulated arbitrarily. */

#define OPTIMIZER_HIDE_VAR(var) __asm__ ( "" : "+g" (var) )
--
2.36.0