Skip to contentSkip to conversation

ARM: move __fixup_smp out of init section

PATCH New Rob Herring 3 participants 8 messages Feb 25, 2014 – Feb 26, 2014 Arm (32-bit)

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers don't work before the MMU is enabled. Fix this by moving __fixup_smp to the .head.text section as it is not very big.

---
 arch/arm/kernel/head.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kernel/head.S +1 -1
index 914616e..78e1443 100644
@@ -475,7 +475,7 @@ ENDPROC(__turn_mmu_on)
 
 
 #ifdef CONFIG_SMP_ON_UP
-	__INIT
+	__HEAD
 __fixup_smp:
 	and	r3, r9, #0x000f0000	@ architecture version
 	teq	r3, #0x000f0000		@ CPU ID supported?
Signature · 1 line
1.8.3.2
#1 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 9 lines · On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers don't work before the MMU is enabled. Fix this by moving __fixup_smp to the .head.text section as it is not very big.

This looks good to me, but I have some related questions:

* I needed to use -mlong-calls for large kernels. Did you manage without?

* Do you (or anyone) know how to force the use of a veneer? I see a problem with some driver calling __do_div_asm from IIRC an init section, and that creates a link error when the kernel gets too big

* If FUNCTION_TRACER is enabled, we get calls from every function using 'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

* Same thing but simpler for svc_preempt calling preempt_schedule_irq and lookup_processor_type calling __lookup_processor_type. In those cases I guess we should be able to trivially rewrite the assembly to jump through an extra register.

The complete patch I have is at http://pastebin.com/5yVaUC4u, but it's likely that it's completely broken ;-)

Arnd

#2 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 9 lines · Arnd Bergmann
Quoted 8 lines · On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers don't work before the MMU is enabled. Fix this by moving __fixup_smp to the .head.text section as it is not very big.

Quoted 5 lines
Quoted 1 line

This looks good to me, but I have some related questions:

* I needed to use -mlong-calls for large kernels. Did you manage without?

Yes, but I started with allyesconfig and then disabled things until it would link. This was mainly PCI, various debug, and filesystems. This particular failure you don't see until you try to boot.

I also saw intermittent inconsistent kallsym errors which KALLSYMS_EXTRA_PASS=1 did not fix.

Quoted 3 lines

* Do you (or anyone) know how to force the use of a veneer? I see a problem with some driver calling __do_div_asm from IIRC an init section, and that creates a link error when the kernel gets too big

That is surprising given that I saw veneers be inserted for some asm to asm calls. Maybe it is something I turned off or something to do with the types of sections.

Quoted 2 lines

* If FUNCTION_TRACER is enabled, we get calls from every function using 'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

Put it in the middle of the image? ;)

Quoted 4 lines

* Same thing but simpler for svc_preempt calling preempt_schedule_irq and lookup_processor_type calling __lookup_processor_type. In those cases I guess we should be able to trivially rewrite the assembly to jump through an extra register.

I did see that lookup_processor_type had a veneer in my build.

Rob

#3 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 14 lines · On Tuesday 25 February 2014 15:05:01 Rob Herring wrote:
Quoted 13 lines · Arnd Bergmann
Quoted 9 lines · On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers don't work before the MMU is enabled. Fix this by moving __fixup_smp to the .head.text section as it is not very big.

This looks good to me, but I have some related questions:

Quoted 8 lines
Quoted 1 line

* I needed to use -mlong-calls for large kernels. Did you manage without?

Yes, but I started with allyesconfig and then disabled things until it would link. This was mainly PCI, various debug, and filesystems. This particular failure you don't see until you try to boot.

I also saw intermittent inconsistent kallsym errors which KALLSYMS_EXTRA_PASS=1 did not fix.

Yes, I saw those too. With XIP_KERNEL enabled, I've seen a randconfig case that needed 12 extra passes before converging. Without that option, it's much better but there are still cases that need more than 3 passes and I haven't gotten to the bottom of that. It seems to be the same configuration that refuses to link at all with ld.gold.

Quoted 11 lines
Quoted 3 lines

* Do you (or anyone) know how to force the use of a veneer? I see a problem with some driver calling __do_div_asm from IIRC an init section, and that creates a link error when the kernel gets too big

That is surprising given that I saw veneers be inserted for some asm to asm calls. Maybe it is something I turned off or something to do with the types of sections.

Quoted 2 lines

* If FUNCTION_TRACER is enabled, we get calls from every function using 'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

Quoted 8 lines

Put it in the middle of the image? ;)

Quoted 4 lines

* Same thing but simpler for svc_preempt calling preempt_schedule_irq and lookup_processor_type calling __lookup_processor_type. In those cases I guess we should be able to trivially rewrite the assembly to jump through an extra register.

I did see that lookup_processor_type had a veneer in my build.

Strange. Could it be that -mlong-calls turns off veneers?

Which gcc and binutils version do you use?

Arnd

#4 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 48 lines · Arnd Bergmann
Quoted 14 lines · On Tuesday 25 February 2014 15:05:01 Rob Herring wrote:
Quoted 13 lines · Arnd Bergmann
Quoted 9 lines · On Tuesday 25 February 2014 12:16:33 Rob Herring wrote:

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers don't work before the MMU is enabled. Fix this by moving __fixup_smp to the .head.text section as it is not very big.

This looks good to me, but I have some related questions:

Quoted 8 lines
Quoted 1 line

* I needed to use -mlong-calls for large kernels. Did you manage without?

Yes, but I started with allyesconfig and then disabled things until it would link. This was mainly PCI, various debug, and filesystems. This particular failure you don't see until you try to boot.

I also saw intermittent inconsistent kallsym errors which KALLSYMS_EXTRA_PASS=1 did not fix.

Yes, I saw those too. With XIP_KERNEL enabled, I've seen a randconfig case that needed 12 extra passes before converging. Without that option, it's much better but there are still cases that need more than 3 passes and I haven't gotten to the bottom of that. It seems to be the same configuration that refuses to link at all with ld.gold.

Quoted 18 lines
Quoted 3 lines

* Do you (or anyone) know how to force the use of a veneer? I see a problem with some driver calling __do_div_asm from IIRC an init section, and that creates a link error when the kernel gets too big

That is surprising given that I saw veneers be inserted for some asm to asm calls. Maybe it is something I turned off or something to do with the types of sections.

Quoted 2 lines

* If FUNCTION_TRACER is enabled, we get calls from every function using 'bl __gnu_mcount_nc'. Do you think it's possible to fix those?

Put it in the middle of the image? ;)

Quoted 4 lines

* Same thing but simpler for svc_preempt calling preempt_schedule_irq and lookup_processor_type calling __lookup_processor_type. In those cases I guess we should be able to trivially rewrite the assembly to jump through an extra register.

Quoted 5 lines
Quoted 1 line

I did see that lookup_processor_type had a veneer in my build.

Strange. Could it be that -mlong-calls turns off veneers?

Which gcc and binutils version do you use?

It's the arm cross compiler in ubuntu 13.10:

$ arm-linux-gnueabihf-ld -v
GNU ld (GNU Binutils for Ubuntu) 2.23.52.20130913
$ arm-linux-gnueabihf-gcc -v
Using built-in specs.
COLLECT_GCC=arm-linux-gnueabihf-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc-cross/arm-linux-gnueabihf/4.8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro
4.8.1-10ubuntu7'
--with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs
--enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr
--program-suffix=-4.8 --enable-shared --enable-linker-build-id
--libexecdir=/usr/lib --without-included-gettext
--enable-threads=posix
--with-gxx-include-dir=/usr/arm-linux-gnueabihf/include/c++/4.8.1
--libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu
--enable-libstdcxx-debug --enable-libstdcxx-time=yes
--enable-gnu-unique-object --disable-libitm --disable-libquadmath
--enable-plugin --with-system-zlib --disable-browser-plugin
--enable-java-awt=gtk --enable-gtk-cairo
--with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross/jre
--enable-java-home
--with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-armhf-cross
--with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-armhf-cross
--with-arch-directory=arm
--with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libgcj
--enable-objc-gc --enable-multiarch --enable-multilib
--disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16
--with-float=hard --with-mode=thumb --disable-werror
--enable-checking=release --build=x86_64-linux-gnu
--host=x86_64-linux-gnu --target=arm-linux-gnueabihf
--program-prefix=arm-linux-gnueabihf-
--includedir=/usr/arm-linux-gnueabihf/include
Thread model: posix
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7)

Rob

#5 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 8 lines · On Tuesday 25 February 2014 15:46:15 Rob Herring wrote:

It's the arm cross compiler in ubuntu 13.10:

$ arm-linux-gnueabihf-ld -v
GNU ld (GNU Binutils for Ubuntu) 2.23.52.20130913
$ arm-linux-gnueabihf-gcc -v
gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu7)

Ok, I'm using the exact same one here.

Arnd

#6 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

In reply to Rob Herring
Quoted 5 lines · Rob Herring

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers

Don't you mean the linker? The compiler isn't involved here. Can you show the disassembly of the veneer please?

I'm confused why the linker would add a veneer for this, but not for the stuff built by the compiler.

Signature · 2 lines
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly
improving, and getting towards what was expected from it.
#7 lore raw

PATCH 1/1 ARM: move __fixup_smp out of init section

Quoted 9 lines · Russell King - ARM Linux
Quoted 5 lines · Rob Herring

From: Rob Herring <[email protected]>

With large kernel builds such as allyesconfig exceeding maximum relative branch offsets, the init section will be too far away to branch to directly. This causes veneers to be added by the compiler, but veneers

Don't you mean the linker? The compiler isn't involved here. Can you show the disassembly of the veneer please?

Here is the call to fixup_smp:

c0008054:       eb000058        bl      c00081bc <__vet_atags>
c0008058:       eb00006a        bl      c0008208 <____fixup_smp_veneer>
c000805c:       eb000042        bl      c000816c <__fixup_pv_table>
c0008060:       eb000007        bl      c0008084 <__create_page_tables>

Here's the veneer:

c0008208 <____fixup_smp_veneer>:
c0008208: e51ff004 ldr pc, [pc, #-4] ; c000820c
<____fixup_smp_veneer+0x4>
c000820c: c2659260 .word 0xc2659260

Quoted 3 lines

I'm confused why the linker would add a veneer for this, but not for the stuff built by the compiler.

There are 1413 other symbols with veneer in my image. So it is not that the linker can't create them, but why it cannot for some as the image gets bigger. Does "text" in a section name have any functional impact on the linker? It seems veneers between .head.text and and init.text are created, but between *.text and .fixup, .proc.info, etc. they don't get created.

Rob