arm64: vdso: getcpu() support
Subsystems (3)
v2 cover arm64: vdso: getcpu() support
This series is a rebase of the previously posted getcpu() support with some additional patches 5-10 added which try to do some cleanups and clarifications of the vDSO code and extend it to multi-page support. Those patches are currently drafts and haven't been fully tested or considered, they're posted as there was some discussion of other applications of the per-CPU data so it seemed useful to share this in progress work.
Some applications, especially tracing ones, benefit from avoiding the syscall overhead for getcpu() so it is common for architectures to have vDSO implementations. Add one for arm64, using TPIDRRO_EL0 to pass a pointer to per-CPU data rather than just store the immediate value in order to allow for future extensibility.
It is questionable if something TPIDRRO_EL0 based is worthwhile at all on current kernels, since v4.18 we have had support for restartable sequences which can be used to provide a sched_getcpu() implementation with generally better performance than the vDSO approach on architectures which have that[1]. Work is ongoing to implement this for glibc:
https://lore.kernel.org/lkml/20200527185130.5604-3-mathieu.desnoyers@efficio
+s.com/
but is not yet merged and will need similar work for other userspaces. The main advantages for the vDSO implementation are the node parameter (though this is a static mapping to CPU number so could be looked up separately when processing data if it's needed, it shouldn't need to be in the hot path) and ease of implementation for users.
This is currently not compatible with KPTI due to the use of TPIDRRO_EL0 by the KPTI trampoline, this could be addressed by reinitializing that system register in the return path but I have found it hard to justify adding that overhead for all users for something that is essentially a profiling optimization which is likely to get superceeded by a more modern implementation - if there are other uses for the per-CPU data then the balance might change here.
There is some overlap with an in flight patch series from Andrei Vagin supporting time namespaces in the vDSO, there shouldn't be a fundamental issue integrating the two serieses.
This builds on work done by Kristina Martsenko some time ago but is a new implementation.
v2:
- Rebase on v5.8-rc3.
- Add further cleanup patches & a first draft of multi-page support.
Mark Brown (11):
arm64: vdso: Provide a define when building the vDSO
arm64: vdso: Add per-CPU data
arm64: vdso: Initialise the per-CPU vDSO data
arm64: vdso: Add getcpu() implementation
arm64: vdso: Remove union in declaration of the data store
arm64: vdso: Document and verify alignment of vDSO text
arm64: vdso: Rename vdso_pages to vdso_text_pages
arm64: vdso: Simplify pagelist allocation
arm64: vdso: Parameterise vDSO data length assumptions in code
arm64: vdso: Support multiple pages of vDSO data
selftests: vdso: Support arm64 in getcpu() test
arch/arm64/include/asm/processor.h | 12 +- arch/arm64/include/asm/vdso.h | 11 ++ arch/arm64/include/asm/vdso/datapage.h | 54 +++++++++ arch/arm64/kernel/process.c | 26 +++- arch/arm64/kernel/vdso.c | 112 ++++++++++++------ arch/arm64/kernel/vdso/Makefile | 4 +- arch/arm64/kernel/vdso/vdso.lds.S | 3 +- arch/arm64/kernel/vdso/vgetcpu.c | 48 ++++++++ .../testing/selftests/vDSO/vdso_test_getcpu.c | 10 ++ 9 files changed, 229 insertions(+), 51 deletions(-) create mode 100644 arch/arm64/include/asm/vdso/datapage.h create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c
base-commit: 9ebcfadb0610322ac537dd7aa5d9cbc2b2894c68
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/11 arm64: vdso: Provide a define when building the vDSO
Provide a define identifying if code is being built for the vDSO to help with writing headers that are shared between the kernel and the vDSO.
--- arch/arm64/kernel/vdso/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \ $(btildflags-y) -T ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18 -ccflags-y += -DDISABLE_BRANCH_PROFILING +ccflags-y += -DDISABLE_BRANCH_PROFILING -D__VDSO__ CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS) KBUILD_CFLAGS += $(DISABLE_LTO)
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/11 arm64: vdso: Add per-CPU data
In reply to Mark BrownIn order to support a vDSO getcpu() implementation add per-CPU data to the vDSO data page. Do this by wrapping the generic vdso_data struct in an arm64 specific one with an array of per-CPU data. The offset of the per-CPU data applying to a CPU will be stored in TPIDRRO_EL0, this allows us to get to the per-CPU data without doing any multiplications.
Since we currently only map a single data page for the vDSO but support very large numbers of CPUs TPIDRRO may be set to zero for CPUs which don't fit in the data page. This will also happen when KPTI is active since kernel_ventry uses TPIDRRO_EL0 as a scratch register in that case, add a comment to the code explaining this.
Acessors for the data are provided in the header since they will be needed in multiple files and it seems neater to keep things together.
--- arch/arm64/include/asm/processor.h | 12 +----- arch/arm64/include/asm/vdso/datapage.h | 54 ++++++++++++++++++++++++++ arch/arm64/kernel/process.c | 26 ++++++++++++- arch/arm64/kernel/vdso.c | 5 ++- 4 files changed, 83 insertions(+), 14 deletions(-) create mode 100644 arch/arm64/include/asm/vdso/datapage.h
arch/arm64/include/asm/processor.h +1 -11
@@ -207,17 +207,7 @@ static inline void set_compat_ssbs_bit(struct pt_regs *regs) regs->pstate |=PSR_AA32_SSBS_BIT ; } -static inline void start_thread (struct pt_regs *regs,unsigned long pc, -unsigned long sp) -{ -start_thread_common (regs, pc); - regs->pstate = PSR_MODE_EL0t; - -if (arm64_get_ssbd_state () !=ARM64_SSBD_FORCE_ENABLE ) -set_ssbs_bit (regs); - - regs->sp = sp; -} +void start_thread (struct pt_regs *regs,unsigned long pc,unsigned long sp);static inline bool is_ttbr0_addr (unsigned long addr) {
arch/arm64/include/asm/vdso/datapage.h new file +54 -0
54 lines not on this page · raw message
arch/arm64/kernel/process.c +25 -1
@@ -55,6 +55,7 @@#include <asm/processor.h> #include <asm/pointer_auth.h> #include <asm/stacktrace.h> +#include <asm/vdso/datapage.h> #if defined(CONFIG_STACKPROTECTOR ) && !defined(CONFIG_STACKPROTECTOR_PER_TASK )#include <linux/stackprotector.h>
@@ -309,6 +310,28 @@ void show_regs(struct pt_regs * regs)dump_backtrace (regs,NULL ,KERN_DEFAULT ); } +void start_thread (struct pt_regs *regs,unsigned long pc,unsigned long sp) +{ +start_thread_common (regs, pc); + regs->pstate = PSR_MODE_EL0t; + +if (arm64_get_ssbd_state () !=ARM64_SSBD_FORCE_ENABLE ) +set_ssbs_bit (regs); + + regs->sp = sp; + +/* +* Store the vDSO per-CPU offset if supported. Disable +* preemption to make sure we read the CPU offset on the CPU +* we write it on. +*/ +if (!arm64_kernel_unmapped_at_el0 ()) { +preempt_disable (); +write_sysreg (vdso_cpu_offset (), tpidrro_el0); +preempt_enable (); + } +} +static void tls_thread_flush (void ) {write_sysreg (0 , tpidr_el0);
@@ -452,7 +475,8 @@ static void tls_thread_switch(struct task_struct *next)if (is_compat_thread (task_thread_info (next)))write_sysreg (next->thread.uw.tp_value, tpidrro_el0);else if (!arm64_kernel_unmapped_at_el0 ()) -write_sysreg (0 , tpidrro_el0); +/* Used as scratch in KPTI trampoline so don't set here. */ +write_sysreg (vdso_cpu_offset (), tpidrro_el0);write_sysreg (*task_user_tls (next), tpidr_el0); }
arch/arm64/kernel/vdso.c +3 -2
@@ -27,6 +27,7 @@#include <asm/cacheflush.h> #include <asm/signal32.h> #include <asm/vdso.h> +#include <asm/vdso/datapage.h> extern char vdso_start[], vdso_end[];#ifdef CONFIG_COMPAT_VDSO
@@ -70,10 +71,10 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = {* The vDSO data page. */ static union { -struct vdso_data data[CS_BASES ]; +struct arm64_vdso_data data;u8 page[PAGE_SIZE ]; } vdso_data_store __page_aligned_data; -struct vdso_data *vdso_data = vdso_data_store.data; +struct vdso_data *vdso_data = vdso_data_store.data.data; static int __vdso_remap(enum vdso_abi abi, const struct vm_special_mapping *sm,
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 3/11 arm64: vdso: Initialise the per-CPU vDSO data
In reply to Mark BrownRegister with the CPU hotplug system to initialise the per-CPU data for getcpu().
--- arch/arm64/kernel/vdso.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+)
arch/arm64/kernel/vdso.c +28 -0
@@ -9,6 +9,7 @@#include <linux/cache.h> #include <linux/clocksource.h> +#include <linux/cpuhotplug.h> #include <linux/elf.h> #include <linux/err.h> #include <linux/errno.h>
@@ -18,6 +19,7 @@#include <linux/sched.h> #include <linux/signal.h> #include <linux/slab.h> +#include <linux/smp.h> #include <linux/timekeeper_internal.h> #include <linux/vmalloc.h> #include <vdso/datapage.h>
@@ -358,6 +360,26 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp) }#endif /* CONFIG_COMPAT */ +static void vdso_cpu_init (void *p) +{ +struct arm64_vdso_data *data = (struct arm64_vdso_data *)vdso_data; +unsigned int cpu; + +if (vdso_cpu_offset ()) { + cpu =smp_processor_id (); + + data->cpu_data[cpu].cpu = cpu; + data->cpu_data[cpu].node =cpu_to_node (cpu); + } +} + +static int vdso_cpu_online (unsigned int cpu) +{ +smp_call_function_single (cpu, vdso_cpu_init,NULL ,1 ); + +return 0 ; +} +static int vdso_mremap (const struct vm_special_mapping *sm,struct vm_area_struct *new_vma) {
@@ -384,6 +406,12 @@ static int __init vdso_init(void) vdso_info[VDSO_ABI_AA64 ].dm = &aarch64_vdso_maps[AA64_MAP_VVAR ]; vdso_info[VDSO_ABI_AA64 ].cm = &aarch64_vdso_maps[AA64_MAP_VDSO ]; +/* +* Initialize per-CPU data, callback runs for all current and +* future CPUs. +*/ +cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"vdso" , vdso_cpu_online,NULL ); +return __vdso_init (VDSO_ABI_AA64 ); }arch_initcall (vdso_init);
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 4/11 arm64: vdso: Add getcpu() implementation
In reply to Mark BrownSome applications, especially trace ones, benefit from avoiding the syscall overhead on getcpu() calls so provide a vDSO implementation of it.
--- arch/arm64/kernel/vdso/Makefile | 2 +- arch/arm64/kernel/vdso/vdso.lds.S | 1 + arch/arm64/kernel/vdso/vgetcpu.c | 48 +++++++++++++++++++++++++++++++ 3 files changed, 50 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kernel/vdso/vgetcpu.c
arch/arm64/kernel/vdso/Makefile +1 -1
@@ -11,7 +11,7 @@ ARCH_REL_TYPE_ABS := R_AARCH64_JUMP_SLOT|R_AARCH64_GLOB_DAT|R_AARCH64_ABS64 include $(srctree)/lib/vdso/Makefile -obj-vdso := vgettimeofday.o note.o sigreturn.o +obj-vdso := vgettimeofday.o note.o sigreturn.o vgetcpu.o # Build rules targets := $(obj-vdso) vdso.so vdso.so.dbg
arch/arm64/kernel/vdso/vdso.lds.S +1 -0
@@ -77,6 +77,7 @@ VERSION __kernel_gettimeofday; __kernel_clock_gettime; __kernel_clock_getres; + __kernel_getcpu; local: *; }; }
arch/arm64/kernel/vdso/vgetcpu.c new file +48 -0
48 lines not on this page · raw message
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 5/11 arm64: vdso: Remove union in declaration of the data store
In reply to Mark BrownCurrently the vDSO data store is declared as part of a union with a page of char for no documented reason, it *appears* that this has been done for alignment purposes but we also have a __page_aligned_data annotation so it's not 100% clear why and confusing when looking at the code. Since it seems to be redundant with current toolchains remove the union.
--- arch/arm64/kernel/vdso.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
arch/arm64/kernel/vdso.c +2 -8
@@ -69,14 +69,8 @@ static struct vdso_abi_info vdso_info[] __ro_after_init = {#endif /* CONFIG_COMPAT_VDSO */ }; -/* -* The vDSO data page. -*/ -static union { -struct arm64_vdso_data data; -u8 page[PAGE_SIZE ]; -} vdso_data_store __page_aligned_data; -struct vdso_data *vdso_data = vdso_data_store.data.data; +struct arm64_vdso_data vdso_data_store __page_aligned_data; +struct vdso_data *vdso_data = vdso_data_store.data; static int __vdso_remap(enum vdso_abi abi, const struct vm_special_mapping *sm,
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 6/11 arm64: vdso: Document and verify alignment of vDSO text
In reply to Mark BrownWe do no rounding when working out the number of vDSO text pages because we ensure in vdso.S that the vDSO text is an exact number of pages long. This is not immediately apparent when looking at the code that does the vDSO mapping in vdso.c so add a comment explaining what's going on and check that this assumption is true.
Ideally we'd have a BUILD_BUG_ON() but since the start and end come from another compilation unit it needs to be a runtime check, add one which prints a warning and errors out similarly to the existing check for the vDSO not being an ELF object.
--- arch/arm64/kernel/vdso.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/arm64/kernel/vdso.c +9 -0
@@ -99,6 +99,15 @@ static int __vdso_init(enum vdso_abi abi)return -EINVAL ; } +/* +* We ensure that the vDSO text is page aligned and an exact +* number of pages in vdso.S so don't need to round here. +*/ +if (!PAGE_ALIGNED (vdso_info[abi].vdso_code_start) || + !PAGE_ALIGNED (vdso_info[abi].vdso_code_end)) { +pr_err ("vDSO is not page aligned\n" ); +return -EINVAL ; + } vdso_info[abi].vdso_pages = ( vdso_info[abi].vdso_code_end - vdso_info[abi].vdso_code_start) >>
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 7/11 arm64: vdso: Rename vdso_pages to vdso_text_pages
In reply to Mark BrownIn preparation for having multiple data pages rename the current vdso_pages to vdso_text_pages.
--- arch/arm64/kernel/vdso.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
arch/arm64/kernel/vdso.c +5 -5
@@ -47,7 +47,7 @@ struct vdso_abi_info {const char *name;const char *vdso_code_start;const char *vdso_code_end; -unsigned long vdso_pages; +unsigned long vdso_text_pages;/* Data Mapping */ struct vm_special_mapping *dm;/* Code Mapping */
@@ -108,13 +108,13 @@ static int __vdso_init(enum vdso_abi abi)pr_err ("vDSO is not page aligned\n" );return -EINVAL ; } - vdso_info[abi].vdso_pages = ( + vdso_info[abi].vdso_text_pages = ( vdso_info[abi].vdso_code_end - vdso_info[abi].vdso_code_start) >>PAGE_SHIFT ;/* Allocate the vDSO pagelist, plus a page for the data. */ - vdso_pagelist =kcalloc (vdso_info[abi].vdso_pages +1 , + vdso_pagelist =kcalloc (vdso_info[abi].vdso_text_pages +1 ,sizeof (struct page *),GFP_KERNEL );if (vdso_pagelist ==NULL )
@@ -127,7 +127,7 @@ static int __vdso_init(enum vdso_abi abi)/* Grab the vDSO code pages. */ pfn =sym_to_pfn (vdso_info[abi].vdso_code_start); -for (i =0 ; i < vdso_info[abi].vdso_pages; i++) +for (i =0 ; i < vdso_info[abi].vdso_text_pages; i++) vdso_pagelist[i +1 ] =pfn_to_page (pfn + i); vdso_info[abi].dm->pages = &vdso_pagelist[0 ];
@@ -145,7 +145,7 @@ static int __setup_additional_pages(enum vdso_abi abi,unsigned long gp_flags =0 ;void *ret; - vdso_text_len = vdso_info[abi].vdso_pages <<PAGE_SHIFT ; + vdso_text_len = vdso_info[abi].vdso_text_pages <<PAGE_SHIFT ;/* Be sure to map the data page */ vdso_mapping_len = vdso_text_len +PAGE_SIZE ;
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 8/11 arm64: vdso: Simplify pagelist allocation
In reply to Mark BrownCurrently we allocate a single array for the struct page * for the vDSO and then do the lookup for code and data in slightly different ways. Make this more regular by factoring out the array allocation and struct page * lookup into a function and using it for both. We don't currently rely on everything being in one array so there should be no functional change.
--- arch/arm64/kernel/vdso.c | 42 ++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 21 deletions(-)
arch/arm64/kernel/vdso.c +21 -21
@@ -88,12 +88,23 @@ static int __vdso_remap(enum vdso_abi abi,return 0 ; } -static int __vdso_init (enum vdso_abi abi) +static struct page **vdso_get_pages (unsigned long pfn,size_t pages) { +struct page **pagelist;int i; -struct page **vdso_pagelist; -unsigned long pfn; + pagelist =kcalloc (pages,sizeof (struct page *),GFP_KERNEL ); +if (!pagelist) +return NULL ; + +for (i =0 ; i < pages; i++) + pagelist[i] =pfn_to_page (pfn + i); + +return pagelist; +} + +static int __vdso_init (enum vdso_abi abi) +{if (memcmp (vdso_info[abi].vdso_code_start,"\177ELF" ,4 )) {pr_err ("vDSO is not a valid ELF object!\n" );return -EINVAL ;
@@ -113,25 +124,14 @@ static int __vdso_init(enum vdso_abi abi) vdso_info[abi].vdso_code_start) >>PAGE_SHIFT ; -/* Allocate the vDSO pagelist, plus a page for the data. */ - vdso_pagelist =kcalloc (vdso_info[abi].vdso_text_pages +1 , -sizeof (struct page *), -GFP_KERNEL ); -if (vdso_pagelist ==NULL ) -return -ENOMEM ; - -/* Grab the vDSO data page. */ - vdso_pagelist[0 ] =phys_to_page (__pa_symbol (vdso_data)); - + vdso_info[abi].dm->pages = +vdso_get_pages (sym_to_pfn (vdso_data),1 ); + vdso_info[abi].cm->pages = +vdso_get_pages (sym_to_pfn (vdso_info[abi].vdso_code_start), + vdso_info[abi].vdso_text_pages); -/* Grab the vDSO code pages. */ - pfn =sym_to_pfn (vdso_info[abi].vdso_code_start); - -for (i =0 ; i < vdso_info[abi].vdso_text_pages; i++) - vdso_pagelist[i +1 ] =pfn_to_page (pfn + i); - - vdso_info[abi].dm->pages = &vdso_pagelist[0 ]; - vdso_info[abi].cm->pages = &vdso_pagelist[1 ]; +if (!vdso_info[abi].cm->pages || !vdso_info[abi].dm->pages) +return -ENOMEM ;return 0 ; }
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 9/11 arm64: vdso: Parameterise vDSO data length assumptions in code
In reply to Mark BrownIn preparation for adding per-CPU data for the vDSO factor out the assumptions about the vDSO having a single data page so that we use a function vdso_data_size() to determine the actual length and a vdso_data_pages to determine the number of pages mapped for data. The actual data size and runtime behaviour are currently unaffected.
--- arch/arm64/include/asm/vdso.h | 7 +++++++ arch/arm64/kernel/vdso.c | 18 ++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-)
arch/arm64/include/asm/vdso.h +7 -0
@@ -12,6 +12,13 @@*/ #define VDSO_LBASE 0x0 +/* +* Hard code magic numbers since we need these in vdso.lds.S, there +* are BUILD_BUG_ON() checks in vdso.c to make sure these are correct. +*/ +#define VDSO_BASE_DATA_SIZE 240 +#define VDSO_CS_BASES 2 +#ifndef __ASSEMBLY__#include <generated/vdso-offsets.h>
arch/arm64/kernel/vdso.c +12 -6
@@ -48,6 +48,7 @@ struct vdso_abi_info {const char *vdso_code_start;const char *vdso_code_end;unsigned long vdso_text_pages; +unsigned long vdso_data_pages;/* Data Mapping */ struct vm_special_mapping *dm;/* Code Mapping */
@@ -110,6 +111,8 @@ static int __vdso_init(enum vdso_abi abi)return -EINVAL ; } +BUILD_BUG_ON (sizeof (struct vdso_data ) !=VDSO_BASE_DATA_SIZE ); +/* * We ensure that the vDSO text is page aligned and an exact * number of pages in vdso.S so don't need to round here.
@@ -123,9 +126,11 @@ static int __vdso_init(enum vdso_abi abi) vdso_info[abi].vdso_code_end - vdso_info[abi].vdso_code_start) >>PAGE_SHIFT ; + vdso_info[abi].vdso_data_pages =VDSO_DATA_PAGES ; vdso_info[abi].dm->pages = -vdso_get_pages (sym_to_pfn (vdso_data),1 ); +vdso_get_pages (sym_to_pfn (vdso_data), + vdso_info[abi].vdso_data_pages); vdso_info[abi].cm->pages =vdso_get_pages (sym_to_pfn (vdso_info[abi].vdso_code_start), vdso_info[abi].vdso_text_pages);
@@ -141,13 +146,14 @@ static int __setup_additional_pages(enum vdso_abi abi, struct linux_binprm *bprm, int uses_interp) { -unsigned long vdso_base, vdso_text_len, vdso_mapping_len; +unsigned long vdso_base, vdso_mapping_len; +unsigned long vdso_text_len, vdso_data_len;unsigned long gp_flags =0 ;void *ret; vdso_text_len = vdso_info[abi].vdso_text_pages <<PAGE_SHIFT ; -/* Be sure to map the data page */ - vdso_mapping_len = vdso_text_len +PAGE_SIZE ; + vdso_data_len = vdso_info[abi].vdso_data_pages <<PAGE_SHIFT ; + vdso_mapping_len = vdso_text_len + vdso_data_len; vdso_base =get_unmapped_area (NULL ,0 , vdso_mapping_len,0 ,0 );if (IS_ERR_VALUE (vdso_base)) {
@@ -155,7 +161,7 @@ static int __setup_additional_pages(enum vdso_abi abi, goto up_fail; } - ret =_install_special_mapping (mm, vdso_base,PAGE_SIZE , + ret =_install_special_mapping (mm, vdso_base, vdso_data_len,VM_READ |VM_MAYREAD , vdso_info[abi].dm);if (IS_ERR (ret))
@@ -164,7 +170,7 @@ static int __setup_additional_pages(enum vdso_abi abi,if (IS_ENABLED (CONFIG_ARM64_BTI_KERNEL ) &&system_supports_bti ()) gp_flags =VM_ARM64_BTI ; - vdso_base +=PAGE_SIZE ; + vdso_base += vdso_data_len; mm->context.vdso = (void *)vdso_base; ret = _install_special_mapping(mm, vdso_base, vdso_text_len, VM_READ|VM_EXEC|gp_flags|
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 10/11 arm64: vdso: Support multiple pages of vDSO data
In reply to Mark BrownIn order to provide per-CPU data for systems with larger numbers of CPUs we need multiple data pages for the vDSO, exactly how many will depend on the page size and the maximum number of CPUs the kernel is configured to support.
Since the vDSO references the data via a symbol defined in the linker script we need to be able to reference the sizes of structures defined in C which appears not to be something we have sensible support for. Handle this by defining magic numbers and using BUILD_BUG_ON() which checks that the numbers are accurate, this is rather distasteful but works. Currently the patch duplicates CS_BASES, with the hard coding of struct sizes it felt like more trouble than it was worth to move it into an assembler clean header.
--- arch/arm64/include/asm/vdso.h | 4 ++++ arch/arm64/kernel/vdso.c | 2 ++ arch/arm64/kernel/vdso/vdso.lds.S | 2 +- 3 files changed, 7 insertions(+), 1 deletion(-)
arch/arm64/include/asm/vdso.h +4 -0
@@ -18,6 +18,10 @@*/ #define VDSO_BASE_DATA_SIZE 240#define VDSO_CS_BASES 2 +#define VDSO_PER_CPU_SIZE 8 +#define VDSO_DATA_SIZE ((VDSO_PER_CPU_SIZE * CONFIG_NR_CPUS) \ + + (VDSO_BASE_DATA_SIZE * VDSO_CS_BASES)) +#define VDSO_DATA_PAGES ((VDSO_DATA_SIZE / PAGE_SIZE) + 1)#ifndef __ASSEMBLY__
arch/arm64/kernel/vdso.c +2 -0
@@ -112,6 +112,8 @@ static int __vdso_init(enum vdso_abi abi) }BUILD_BUG_ON (sizeof (struct vdso_data ) !=VDSO_BASE_DATA_SIZE ); +BUILD_BUG_ON (CS_BASES !=VDSO_CS_BASES ); +BUILD_BUG_ON (sizeof (struct vdso_cpu_data ) !=VDSO_PER_CPU_SIZE );/* * We ensure that the vDSO text is page aligned and an exact
arch/arm64/kernel/vdso/vdso.lds.S +1 -1
@@ -17,7 +17,7 @@ OUTPUT_ARCH(aarch64) SECTIONS { - PROVIDE(_vdso_data = . - PAGE_SIZE); + PROVIDE(_vdso_data = . - (VDSO_DATA_PAGES * PAGE_SIZE)); . = VDSO_LBASE + SIZEOF_HEADERS; .hash : { *(.hash) } :text
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 11/11 selftests: vdso: Support arm64 in getcpu() test
In reply to Mark Brownarm64 exports the vDSO ABI with a version of LINUX_2.6.39 and symbols prefixed with __kernel rather than __vdso. Update the getcpu() test to handle this.
--- tools/testing/selftests/vDSO/vdso_test_getcpu.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
tools/testing/selftests/vDSO/vdso_test_getcpu.c +10 -0
@@ -14,8 +14,18 @@#include "../kselftest.h" #include "parse_vdso.h" +/* +* ARM64's vDSO exports its getcpu() implementation with a different +* name and version from other architectures, so we need to handle it +* as a special case. +*/ +#if defined(__aarch64__) +const char *version ="LINUX_2.6.39" ; +const char *name ="__kernel_getcpu" ; +#else const char *version ="LINUX_2.6" ;const char *name ="__vdso_getcpu" ; +#endif struct getcpu_cache ; typedef long (*getcpu_t)(unsigned int *, unsigned int *,
Signature · 7 lines
2.20.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel