ARM: option for loading modules into vmalloc area

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

arch/arm/mm/mmu.c +1 -1

index 9d2cdda..9e0c4f4 100644
@@ -1161,16 +1161,15 @@ void __init sanity_check_meminfo(void)
 
 static inline void prepare_page_table(void)
 {
-	unsigned long addr;
+	unsigned long addr = 0;
 	phys_addr_t end;
 
 	/*
 	 * Clear out all the mappings below the kernel image.

Quoted 14 lines · Konstantin Khlebnikov

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Quoted 3 lines

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

It also isn't compatible with the older architectures which don't have "blx".

Quoted 12 lines

arch/arm/mm/mmu.c +1 -1

index 9d2cdda..9e0c4f4 100644
@@ -1161,16 +1161,15 @@ void __init sanity_check_meminfo(void)
 
  static inline void prepare_page_table(void)
  {
-	unsigned long addr;
+	unsigned long addr = 0;
  	phys_addr_t end;
 

arch/arm/mm/mmu.c void __init sanity_check_meminfo(void) · line 1175

Quoted 12 lines

arch/arm/mm/mmu.c +2 -3

  	/*
  	 * Clear out all the mappings below the kernel image.
  	 */
-	for (addr = 0; addr < MODULES_VADDR; addr += PMD_SIZE)
-		pmd_clear(pmd_off_k(addr));
-
  #ifdef CONFIG_XIP_KERNEL
+	for ( ; addr < MODULES_VADDR; addr += PMD_SIZE)
+		pmd_clear(pmd_off_k(addr));
  	/* The XIP kernel is mapped in the module area -- skip over it */
  	addr = ((unsigned long)_etext + PMD_SIZE - 1) & PMD_MASK;
  #endif

This seems to be an unrelated change. In any case, it looks wrong to me. Please describe what you are trying to achieve here.

Quoted 15 lines · Russell King - ARM Linux

Quoted 14 lines · Konstantin Khlebnikov

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Quoted 5 lines

Quoted 3 lines

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

It also isn't compatible with the older architectures which don't have "blx".

Ok, I'll add "depends on CPU_V6 || CPU_V7" I don't think that it is necessary for older cpus.

Quoted 19 lines · On Tuesday 18 November 2014 21:13:56 Konstantin Khlebnikov wrote:

Quoted 18 lines · Russell King - ARM Linux

Quoted 17 lines · Konstantin Khlebnikov

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

Quoted 5 lines

Quoted 2 lines

It also isn't compatible with the older architectures which don't have "blx".

Ok, I'll add "depends on CPU_V6 || CPU_V7" I don't think that it is necessary for older cpus.

Why not just use a different branch instruction for the older CPUs?

Quoted 20 lines · Arnd Bergmann

Quoted 19 lines · On Tuesday 18 November 2014 21:13:56 Konstantin Khlebnikov wrote:

Quoted 18 lines · Russell King - ARM Linux

Quoted 17 lines · Konstantin Khlebnikov

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

It also isn't compatible with the older architectures which don't have "blx".

Ok, I'll add "depends on CPU_V6 || CPU_V7" I don't think that it is necessary for older cpus.

Why not just use a different branch instruction for the older CPUs?

ARMv6 doesn't support movw/movt so this will only work on v7.

What about doing 'mov lr, pc; ldr pc,=symbol' instead? You clearly don't care about performance in this case, so the performance hit (due to the dcache access and interfering with the return stack predictors) should be tolerable. The only thing to be careful about is thumb2
kernels: you would need to set the thumb bit in lr manually but only if the call is made /from/ thumb. You would probably be better off just depending on !THUMB2_KERNEL.

Ard.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Quoted 32 lines · Ard Biesheuvel

Quoted 20 lines · Arnd Bergmann

Quoted 19 lines · On Tuesday 18 November 2014 21:13:56 Konstantin Khlebnikov wrote:

Quoted 18 lines · Russell King - ARM Linux

Quoted 17 lines · Konstantin Khlebnikov

Usually modules are loaded into small area prior to the kernel text because they are linked with the kernel using short calls. Compile-time instrumentation like GCOV or KASAN bloats code a lot, and as a result huge modules no longer fit into reserved area.

This patch adds option CONFIG_MODULES_USE_VMALLOC which lifts limitation on amount of loaded modules. It links modules using long-calls (option -mlong-calls) and loads them into vmalloc area.

In few places exported symbols are called from inline assembly. This patch adds macro for such call sites: __asmbl and __asmbl_clobber. Call turns into single 'bl' or sequence 'movw; movt; blx' depending on context and state of config option.

Unfortunately this option isn't compatible with CONFIG_FUNCTION_TRACER. Compiler emits short calls to profiling function despite of -mlong-calls. This is a bug in GCC, but ftrace anyway needs an update to handle this.

Quoted 8 lines

Quoted 5 lines

Quoted 2 lines

It also isn't compatible with the older architectures which don't have "blx".

Ok, I'll add "depends on CPU_V6 || CPU_V7" I don't think that it is necessary for older cpus.

Why not just use a different branch instruction for the older CPUs?

ARMv6 doesn't support movw/movt so this will only work on v7.

Quoted 7 lines

What about doing 'mov lr, pc; ldr pc,=symbol' instead? You clearly don't care about performance in this case, so the performance hit (due to the dcache access and interfering with the return stack predictors) should be tolerable. The only thing to be careful about is thumb2
kernels: you would need to set the thumb bit in lr manually but only if the call is made /from/ thumb. You would probably be better off just depending on !THUMB2_KERNEL.

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Quoted 18 lines · Ard Biesheuvel

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

Quoted 27 lines · Russell King - ARM Linux

Quoted 18 lines · Ard Biesheuvel

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Quoted 6 lines

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

That applies to any inline asm statement in general: the compiler assumes that the expanded size will not interfere with its ability to emit literals after the function's return instruction. Sometimes it will put a literal pool in the middle of the function if it is very large, and I am not sure if an inline asm by itself would ever trigger that heuristic to kick in.

But by the same logic, i.e., due to the fact that GCC manages its own literals, the literal pool at the assembly level is unlikely to be so large that you will actually hit this condition.

Quoted 4 lines

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

We could at least add some labels instead of doing explicit pc arithmetic, i.e.,

adr lr, 1f
ldr pc, 0f
0: .long symbol
1:

Quoted 54 lines · Ard Biesheuvel

Quoted 27 lines · Russell King - ARM Linux

Quoted 18 lines · Ard Biesheuvel

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Quoted 6 lines

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

That applies to any inline asm statement in general: the compiler assumes that the expanded size will not interfere with its ability to emit literals after the function's return instruction. Sometimes it will put a literal pool in the middle of the function if it is very large, and I am not sure if an inline asm by itself would ever trigger that heuristic to kick in.

But by the same logic, i.e., due to the fact that GCC manages its own literals, the literal pool at the assembly level is unlikely to be so large that you will actually hit this condition.

Quoted 4 lines

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

We could at least add some labels instead of doing explicit pc arithmetic, i.e.,

Quoted 4 lines

adr lr, 1f
ldr pc, 0f
0: .long symbol
1:

I think we need some unique prefix here, this macro is used inside bigger inline assembly constructions and probably another macro.

Numbers are disambiguated by the f and b suffixes, so they can be reused in the same .s file. So as long as you use a strictly numerical prefix, you can deal correctly with the case where, for instance, do_div() is called twice in the same compilation unit, and still not clash with other inline asm

Quoted 5 lines · Ard Biesheuvel

Numbers are disambiguated by the f and b suffixes, so they can be reused in the same .s file. So as long as you use a strictly numerical prefix, you can deal correctly with the case where, for instance, do_div() is called twice in the same compilation unit, and still not clash with other inline asm

What's not particularly nice though is to hide these in a macro, which itself may be part of a larger macro or code fragment also using small numbers.

We probably ought to be a bit more inteligent about how we choose these numbers inside macros, rather than just randomly picking some and hoping that they don't clash.

Quoted 7 lines · Russell King - ARM Linux

On Wed, Nov 19, 2014 at 05:02:40PM +0100, Ard Biesheuvel wrote:
Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

That applies to any inline asm statement in general: the compiler assumes that the expanded size will not interfere with its ability to emit literals after the function's return instruction. Sometimes it will put a literal pool in the middle of the function if it is very large, and I am not sure if an inline asm by itself would ever trigger that heuristic to kick in.

Quoted 3 lines

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

We could at least add some labels instead of doing explicit pc arithmetic, i.e.,

adr lr, 1f
ldr pc, 0f
0: .long symbol
1:

Quoted 18 lines · Ard Biesheuvel

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

Quoted 33 lines · Russell King - ARM Linux

Quoted 18 lines · Ard Biesheuvel

Quoted 7 lines · Konstantin Khlebnikov

Do you mean ldr pc, =symbol ?

In this case I get this error:

/tmp/ccAHtONU.s: Assembler messages:
/tmp/ccAHtONU.s:220: Error: invalid literal constant: pool needs to be closer

Quoted 9 lines

Probably constant pool doesn't work well in inline assembly.

Something like this seems work:

add lr, pc, #4
ldr pc, [pc, #-4]
.long symbol

Quoted 7 lines

You can add a '.ltorg' instruction which tells the assembler to dump the literal pool, but you still need to jump over it, i.e.,

adr lr, 0f
ldr pc, =symbol
.ltorg
0:

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

Quoted 3 lines

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

Modules could be built without far calls by default, and then the module linker would only have to redirect those calls whose destination is too far away to a dynamically created trampoline table.

If I remember correctly you even posted some patches to that effect a couple years ago. Maybe those could be salvaged?

I would largely recommend a solution where the link process could deal with it automatically and as needed rather than sprinkling yet more manually maintained macros into assembly code.

Quoted 9 lines · Russell King - ARM Linux

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

Modules could be built without far calls by default, and then the module linker would only have to redirect those calls whose destination is too far away to a dynamically created trampoline table.

If I remember correctly you even posted some patches to that effect a couple years ago. Maybe those could be salvaged?

Quoted 11 lines · Nicolas Pitre

Quoted 9 lines · Russell King - ARM Linux

Which is not a good idea either, because the compiler needs to know how far away its own manually generated literal pool is from the instructions which reference it. The .ltorg statement can end up emitting any number of literals at that point, which makes it indeterminant how many words are contained within the asm() statement.

Yes, it isn't desirable to waste an entire data cache line per indirect call like the original quote above, but I don't see a practical alternative.

Quoted 6 lines

Modules could be built without far calls by default, and then the module linker would only have to redirect those calls whose destination is too far away to a dynamically created trampoline table.

If I remember correctly you even posted some patches to that effect a couple years ago. Maybe those could be salvaged?

I don't think I ever did, because its pretty much impossible to do as I explained in a follow up to this thread.

We _used_ to do this with the userspace insmod methods, but since we got this kernel-side linker, it's been pretty much impossible to do without rewriting the module code. That's not going to happen on account of one quirky architecture which Linus doesn't particularly like.

Quoted 7 lines · Russell King - ARM Linux

I don't think I ever did, because its pretty much impossible to do as I explained in a follow up to this thread.

We _used_ to do this with the userspace insmod methods, but since we got this kernel-side linker, it's been pretty much impossible to do without rewriting the module code. That's not going to happen on account of one quirky architecture which Linus doesn't particularly like.

Still... We could try adding a hook in the generic module linker code for a pre-relocation pass. Maybe only ARM would use it, but if the need to load big modules is real then I imagine Linus could be amenable to a compromise.

Quoted 9 lines · Nicolas Pitre

Quoted 7 lines · Russell King - ARM Linux

I don't think I ever did, because its pretty much impossible to do as I explained in a follow up to this thread.

We _used_ to do this with the userspace insmod methods, but since we got this kernel-side linker, it's been pretty much impossible to do without rewriting the module code. That's not going to happen on account of one quirky architecture which Linus doesn't particularly like.

Quoted 4 lines

Still... We could try adding a hook in the generic module linker code for a pre-relocation pass. Maybe only ARM would use it, but if the need to load big modules is real then I imagine Linus could be amenable to a compromise.

So, how big a table would you allocate for the trampolines, based upon not knowing anything about the module being loaded? 4K? 8K? 64K?

Quoted 15 lines · Russell King - ARM Linux

Quoted 13 lines · Nicolas Pitre

Quoted 7 lines · Russell King - ARM Linux

I don't think I ever did, because its pretty much impossible to do as I explained in a follow up to this thread.

We _used_ to do this with the userspace insmod methods, but since we got this kernel-side linker, it's been pretty much impossible to do without rewriting the module code. That's not going to happen on account of one quirky architecture which Linus doesn't particularly like.

Still... We could try adding a hook in the generic module linker code for a pre-relocation pass. Maybe only ARM would use it, but if the need to load big modules is real then I imagine Linus could be amenable to a compromise.

Quoted 2 lines

So, how big a table would you allocate for the trampolines, based upon not knowing anything about the module being loaded? 4K? 8K? 64K?

The idea of a pre-relocation pass is to determine that. That could be something similar to calling apply_relocate() twice: once to determine the number of trampoline entries, and a second time to perform the actual relocation.

Quoted 17 lines · Nicolas Pitre

Quoted 15 lines · Russell King - ARM Linux

Quoted 13 lines · Nicolas Pitre

Quoted 7 lines · Russell King - ARM Linux

I don't think I ever did, because its pretty much impossible to do as I explained in a follow up to this thread.

We _used_ to do this with the userspace insmod methods, but since we got this kernel-side linker, it's been pretty much impossible to do without rewriting the module code. That's not going to happen on account of one quirky architecture which Linus doesn't particularly like.

Still... We could try adding a hook in the generic module linker code for a pre-relocation pass. Maybe only ARM would use it, but if the need to load big modules is real then I imagine Linus could be amenable to a compromise.

Quoted 8 lines

Quoted 2 lines

So, how big a table would you allocate for the trampolines, based upon not knowing anything about the module being loaded? 4K? 8K? 64K?

The idea of a pre-relocation pass is to determine that. That could be something similar to calling apply_relocate() twice: once to determine the number of trampoline entries, and a second time to perform the actual relocation.

Well, the veneers shouldn't take more than 3 words each, right?

ldr ip, [pc]
bx ip
.long symbol

and you would need at most one veneer per unique external symbol referenced by one or more R_ARM_CALL relocations. Is there no way to just add that to the static mem footprint as padding, and let the loader populate it as needed at module relocation time?