ARM: VDSO

arch/arm/include/asm/vdso_datapage.h line 36

Quoted 6 lines · Nathan Lynch

arch/arm/include/asm/vdso_datapage.h new file +6 -0

+struct vdso_data {
+       u32 seq_count;          /* sequence count - odd during updates */
+       u16 use_syscall;        /* whether to fall back to syscalls */
+       u16 cs_shift;           /* clocksource shift */
+       u32 xtime_coarse_sec;   /* coarse time */
+       u32 xtime_coarse_nsec;

Note that we will at some point have to introduce 64-bit time_t in user space in some form. We will have to add new system calls for anything dealing with time on the kernel boundary, but it would be good to be prepared here.

Is vdso_data an ABI-relevant data structure? If so, I think all you need here is to make xtime_coarse_sec a u64 type for now, so we can add the other functions later. The code can of course for now access only the lower half (keeping endianess in mind), so there should not be any performance impact.

If the structure is not part of the ABI, there probably isn't much we can do here.

Quoted 12 lines · Arnd Bergmann

arch/arm/include/asm/vdso_datapage.h line 36

Quoted 6 lines · Nathan Lynch

arch/arm/include/asm/vdso_datapage.h new file +6 -0

+struct vdso_data {
+       u32 seq_count;          /* sequence count - odd during updates */
+       u16 use_syscall;        /* whether to fall back to syscalls */
+       u16 cs_shift;           /* clocksource shift */
+       u32 xtime_coarse_sec;   /* coarse time */
+       u32 xtime_coarse_nsec;

Note that we will at some point have to introduce 64-bit time_t in user space in some form. We will have to add new system calls for anything dealing with time on the kernel boundary, but it would be good to be prepared here.

Quoted 7 lines

Is vdso_data an ABI-relevant data structure? If so, I think all you need here is to make xtime_coarse_sec a u64 type for now, so we can add the other functions later. The code can of course for now access only the lower half (keeping endianess in mind), so there should not be any performance impact.

If the structure is not part of the ABI, there probably isn't much we can do here.

In short, no, vdso_data is not ABI-relevant.

The vdso_data structure may be accessed legitimately only through the entry points (gettimeofday, clock_gettime) that the VDSO provides, and it is my intent that we should be able to change the structure as needed, just like other architectures. vdso_data should not show up in any exported header and I'll double-check to make sure it does not.

So, when the time comes, we should be able to change it to accommodate 64-bit time_t by (I imagine) making any field that stores seconds 64 bits and adding new entry points (clock_gettime64 or whatever).

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Of particular note is that a post-processing step ("vdsomunge") is necessary to produce a shared object which is architecturally allowed to be used by both soft- and hard-float EABI programs.

The 2012 edition of the ARM ABI defines Tag_ABI_VFP_args = 3 "Code is compatible with both the base and VFP variants; the user did not permit non-variadic functions to pass FP parameters/results." Unfortunately current toolchains do not support this tag, which is ideally what we would use.

The best available option is to ensure that both EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD are unset in the ELF header's e_flags, indicating that the shared object is "old" and should be accepted for backward compatibility's sake. While binutils < 2.24 appear to produce a vdso.so with both flags clear, 2.24 always sets EF_ARM_ABI_FLOAT_SOFT, with no way to inhibit this behavior. So we have to fix things up with a custom post-processing step.

In fact, the VDSO code in glibc does much less validation (including checking these flags) than the code for handling conventional file-backed shared libraries, so this is a bit moot unless glibc's VDSO code becomes more strict.

Signed-off-by Nathan Lynch <[email protected]>

---
 arch/arm/kernel/asm-offsets.c        |   5 +
 arch/arm/kernel/vdso/.gitignore      |   1 +
 arch/arm/kernel/vdso/Makefile        |  59 +++++++
 arch/arm/kernel/vdso/checkundef.sh   |   9 +
 arch/arm/kernel/vdso/datapage.S      |  15 ++
 arch/arm/kernel/vdso/vdso.S          |  35 ++++
 arch/arm/kernel/vdso/vdso.lds.S      |  88 ++++++++++
 arch/arm/kernel/vdso/vdsomunge.c     | 193 +++++++++++++++++++++
 arch/arm/kernel/vdso/vgettimeofday.c | 320 +++++++++++++++++++++++++++++++++++
 9 files changed, 725 insertions(+)

 create mode 100644 arch/arm/kernel/vdso/.gitignore
 create mode 100644 arch/arm/kernel/vdso/Makefile
 create mode 100755 arch/arm/kernel/vdso/checkundef.sh
 create mode 100644 arch/arm/kernel/vdso/datapage.S
 create mode 100644 arch/arm/kernel/vdso/vdso.S
 create mode 100644 arch/arm/kernel/vdso/vdso.lds.S
 create mode 100644 arch/arm/kernel/vdso/vdsomunge.c
 create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

Quoted 45 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Of particular note is that a post-processing step ("vdsomunge") is necessary to produce a shared object which is architecturally allowed to be used by both soft- and hard-float EABI programs.

The 2012 edition of the ARM ABI defines Tag_ABI_VFP_args = 3 "Code is compatible with both the base and VFP variants; the user did not permit non-variadic functions to pass FP parameters/results." Unfortunately current toolchains do not support this tag, which is ideally what we would use.

The best available option is to ensure that both EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD are unset in the ELF header's e_flags, indicating that the shared object is "old" and should be accepted for backward compatibility's sake. While binutils < 2.24 appear to produce a vdso.so with both flags clear, 2.24 always sets EF_ARM_ABI_FLOAT_SOFT, with no way to inhibit this behavior. So we have to fix things up with a custom post-processing step.

In fact, the VDSO code in glibc does much less validation (including checking these flags) than the code for handling conventional file-backed shared libraries, so this is a bit moot unless glibc's VDSO code becomes more strict.

Signed-off-by Nathan Lynch <[email protected]>

---
 arch/arm/kernel/asm-offsets.c        |   5 +
 arch/arm/kernel/vdso/.gitignore      |   1 +
 arch/arm/kernel/vdso/Makefile        |  59 +++++++
 arch/arm/kernel/vdso/checkundef.sh   |   9 +
 arch/arm/kernel/vdso/datapage.S      |  15 ++
 arch/arm/kernel/vdso/vdso.S          |  35 ++++
 arch/arm/kernel/vdso/vdso.lds.S      |  88 ++++++++++
 arch/arm/kernel/vdso/vdsomunge.c     | 193 +++++++++++++++++++++
 arch/arm/kernel/vdso/vgettimeofday.c | 320 +++++++++++++++++++++++++++++++++++
 9 files changed, 725 insertions(+)

Quoted 8 lines

 create mode 100644 arch/arm/kernel/vdso/.gitignore
 create mode 100644 arch/arm/kernel/vdso/Makefile
 create mode 100755 arch/arm/kernel/vdso/checkundef.sh
 create mode 100644 arch/arm/kernel/vdso/datapage.S
 create mode 100644 arch/arm/kernel/vdso/vdso.S
 create mode 100644 arch/arm/kernel/vdso/vdso.lds.S
 create mode 100644 arch/arm/kernel/vdso/vdsomunge.c
 create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

One change I would like to see (to stop the directory tree getting soo deep) is to move this to arch/arm/vdso - just like x86 is arch/x86/vdso. Was there a pressing reason to have it in arch/arm/kernel ?

Quoted 55 lines · Russell King - ARM Linux

Quoted 45 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Of particular note is that a post-processing step ("vdsomunge") is necessary to produce a shared object which is architecturally allowed to be used by both soft- and hard-float EABI programs.

The 2012 edition of the ARM ABI defines Tag_ABI_VFP_args = 3 "Code is compatible with both the base and VFP variants; the user did not permit non-variadic functions to pass FP parameters/results." Unfortunately current toolchains do not support this tag, which is ideally what we would use.

The best available option is to ensure that both EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD are unset in the ELF header's e_flags, indicating that the shared object is "old" and should be accepted for backward compatibility's sake. While binutils < 2.24 appear to produce a vdso.so with both flags clear, 2.24 always sets EF_ARM_ABI_FLOAT_SOFT, with no way to inhibit this behavior. So we have to fix things up with a custom post-processing step.

In fact, the VDSO code in glibc does much less validation (including checking these flags) than the code for handling conventional file-backed shared libraries, so this is a bit moot unless glibc's VDSO code becomes more strict.

Signed-off-by Nathan Lynch <[email protected]>

---
  arch/arm/kernel/asm-offsets.c        |   5 +
  arch/arm/kernel/vdso/.gitignore      |   1 +
  arch/arm/kernel/vdso/Makefile        |  59 +++++++
  arch/arm/kernel/vdso/checkundef.sh   |   9 +
  arch/arm/kernel/vdso/datapage.S      |  15 ++
  arch/arm/kernel/vdso/vdso.S          |  35 ++++
  arch/arm/kernel/vdso/vdso.lds.S      |  88 ++++++++++
  arch/arm/kernel/vdso/vdsomunge.c     | 193 +++++++++++++++++++++
  arch/arm/kernel/vdso/vgettimeofday.c | 320 +++++++++++++++++++++++++++++++++++
  9 files changed, 725 insertions(+)

Quoted 8 lines

create mode 100644 arch/arm/kernel/vdso/.gitignore create mode 100644 arch/arm/kernel/vdso/Makefile create mode 100755 arch/arm/kernel/vdso/checkundef.sh create mode 100644 arch/arm/kernel/vdso/datapage.S create mode 100644 arch/arm/kernel/vdso/vdso.S create mode 100644 arch/arm/kernel/vdso/vdso.lds.S create mode 100644 arch/arm/kernel/vdso/vdsomunge.c create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

Quoted 3 lines

One change I would like to see (to stop the directory tree getting soo deep) is to move this to arch/arm/vdso - just like x86 is arch/x86/vdso. Was there a pressing reason to have it in arch/arm/kernel ?

It also looks like there's something missing for vdso_install to work. arch/x86 has this:

PHONY += vdso_install
vdso_install:

         $(Q)$(MAKE) $(build)=arch/x86/vdso $@

but doesn't list it in the arch help. I'm sure we can do better on ARM. :)

Quoted 45 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Of particular note is that a post-processing step ("vdsomunge") is necessary to produce a shared object which is architecturally allowed to be used by both soft- and hard-float EABI programs.

The 2012 edition of the ARM ABI defines Tag_ABI_VFP_args = 3 "Code is compatible with both the base and VFP variants; the user did not permit non-variadic functions to pass FP parameters/results." Unfortunately current toolchains do not support this tag, which is ideally what we would use.

The best available option is to ensure that both EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD are unset in the ELF header's e_flags, indicating that the shared object is "old" and should be accepted for backward compatibility's sake. While binutils < 2.24 appear to produce a vdso.so with both flags clear, 2.24 always sets EF_ARM_ABI_FLOAT_SOFT, with no way to inhibit this behavior. So we have to fix things up with a custom post-processing step.

In fact, the VDSO code in glibc does much less validation (including checking these flags) than the code for handling conventional file-backed shared libraries, so this is a bit moot unless glibc's VDSO code becomes more strict.

Signed-off-by Nathan Lynch <[email protected]>

---
 arch/arm/kernel/asm-offsets.c        |   5 +
 arch/arm/kernel/vdso/.gitignore      |   1 +
 arch/arm/kernel/vdso/Makefile        |  59 +++++++
 arch/arm/kernel/vdso/checkundef.sh   |   9 +
 arch/arm/kernel/vdso/datapage.S      |  15 ++
 arch/arm/kernel/vdso/vdso.S          |  35 ++++
 arch/arm/kernel/vdso/vdso.lds.S      |  88 ++++++++++
 arch/arm/kernel/vdso/vdsomunge.c     | 193 +++++++++++++++++++++
 arch/arm/kernel/vdso/vgettimeofday.c | 320 +++++++++++++++++++++++++++++++++++
 9 files changed, 725 insertions(+)

Quoted 8 lines

 create mode 100644 arch/arm/kernel/vdso/.gitignore
 create mode 100644 arch/arm/kernel/vdso/Makefile
 create mode 100755 arch/arm/kernel/vdso/checkundef.sh
 create mode 100644 arch/arm/kernel/vdso/datapage.S
 create mode 100644 arch/arm/kernel/vdso/vdso.S
 create mode 100644 arch/arm/kernel/vdso/vdso.lds.S
 create mode 100644 arch/arm/kernel/vdso/vdsomunge.c
 create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

One change I would like to see (to stop the directory tree getting soo deep) is to move this to arch/arm/vdso - just like x86 is arch/x86/vdso. Was there a pressing reason to have it in arch/arm/kernel ?

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Quoted 6 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

Quoted 3 lines

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Okay, how is this used by userspace?

It seems that on ARM, we generate a dso with these symbols:

DYNAMIC SYMBOL TABLE:

000001e4 l    d  .eh_frame_hdr	00000000              .eh_frame_hdr
00000000 g    DO *ABS*	00000000  LINUX_3.17  LINUX_3.17
000003c4 g    DF .text	00000080  LINUX_3.17  __kernel_clock_getres
00000444 g    DF .text	0000002c  LINUX_3.17  __kernel_gettimeofday
00000298 g    DF .text	0000012c  LINUX_3.17  __kernel_clock_gettime

whereas, x86-64 generates this:

DYNAMIC SYMBOL TABLE:

ffffffffff700330 l    d  .eh_frame_hdr	0000000000000000              .eh_frame_hdr
ffffffffff700600  w   DF .text	00000000000002b9  LINUX_2.6   clock_gettime
0000000000000000 g    DO *ABS*	0000000000000000  LINUX_2.6   LINUX_2.6
ffffffffff7008c0 g    DF .text	000000000000015a  LINUX_2.6   __vdso_gettimeofday
ffffffffff700a40 g    DF .text	000000000000003d  LINUX_2.6   __vdso_getcpu
ffffffffff7008c0  w   DF .text	000000000000015a  LINUX_2.6   gettimeofday
ffffffffff700a20  w   DF .text	0000000000000016  LINUX_2.6   time
ffffffffff700a40  w   DF .text	000000000000003d  LINUX_2.6   getcpu
ffffffffff700600 g    DF .text	00000000000002b9  LINUX_2.6   __vdso_clock_gettime
ffffffffff700a20 g    DF .text	0000000000000016  LINUX_2.6   __vdso_time

hence references to gettimeofday automatically get resolved to the vdso version without needing C library updates.

Or... do we need a glibc update to make use of this?

Quoted 11 lines · Russell King - ARM Linux

Quoted 9 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Quoted 1 line

Okay, how is this used by userspace?

Primarily it's intended to be transparent to userspace applications; glibc -- with a patch -- detects the VDSO at startup and binds the appropriate routines to the VDSO's.

However, even an unpatched glibc makes the VDSO symbols available for lookup via dlsym, which has been convenient for testing.

Quoted 2 lines

hence references to gettimeofday automatically get resolved to the vdso version without needing C library updates.

Hmm, I'm not sure. I had wondered why x86-64's vdso has the aliases (clock_gettime --> __vdso_clock_gettime). Most other architectures' VDSOs don't.

Quoted 1 line

Or... do we need a glibc update to make use of this?

That is the idea, yes. You don't have to rebuild applications but glibc needs to be updated to dispatch its gettimeofday and clock_gettime to the VDSO.

Quoted 12 lines · Nathan Lynch

Quoted 11 lines · Russell King - ARM Linux

Quoted 9 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Quoted 8 lines

Quoted 1 line

Okay, how is this used by userspace?

Primarily it's intended to be transparent to userspace applications; glibc -- with a patch -- detects the VDSO at startup and binds the appropriate routines to the VDSO's.

However, even an unpatched glibc makes the VDSO symbols available for lookup via dlsym, which has been convenient for testing.

It's a shame that the vDSO provided symbols aren't automatically picked up in place of the glibc versions, because it means probably many years before the vDSO seriously gets used.

I know that I'm probably many years away from it, even if Ubuntu adopt it for something recent - I'm rather stuck with 12.04 due to the move towards requiring 3D GPU acceleration in later versions. It seems that the choices on ARM now are to either have a crippled distro through lack of open GPU support, or stick with an old distro that doesn't require GPUs.

Quoted 6 lines

Quoted 2 lines

hence references to gettimeofday automatically get resolved to the vdso version without needing C library updates.

Hmm, I'm not sure. I had wondered why x86-64's vdso has the aliases (clock_gettime --> __vdso_clock_gettime). Most other architectures' VDSOs don't.

Yes, and providing them seems to have no useful benefit either, because they aren't picked up automatically by the linker.

Quoted 5 lines

Quoted 1 line

Or... do we need a glibc update to make use of this?

That is the idea, yes. You don't have to rebuild applications but glibc needs to be updated to dispatch its gettimeofday and clock_gettime to the VDSO.

Yea, many people aren't going to rebuild glibc just because of this, me included. If only there was a way for automatic vdso support.

So, I'm going to have to rely on you to say "yes it works" and to keep an eye on it in the future in case something breaks.

Quoted 6 lines · Russell King - ARM Linux

I know that I'm probably many years away from it, even if Ubuntu adopt it for something recent - I'm rather stuck with 12.04 due to the move towards requiring 3D GPU acceleration in later versions. It seems that the choices on ARM now are to either have a crippled distro through lack of open GPU support, or stick with an old distro that doesn't require GPUs.

Based on other messages from you on the ML I'm assuming you have an i.MX6 system in mind? (Forgive me if I'm wrong.) On any Cortex-A9 or -A8 the VDSO isn't going to be any great benefit anyway. Those CPUs don't implement the generic timers extension, so only clock_gettime() with low-resolution clock ids will see a speedup. And few programs use those, to my knowledge.

A15, A7, and I'm pretty sure A12 and A17 all implement the timer extension and will see full benefit. Some Qualcomm CPUs too, I think.

Quoted 15 lines · Nathan Lynch

Quoted 6 lines · Russell King - ARM Linux

I know that I'm probably many years away from it, even if Ubuntu adopt it for something recent - I'm rather stuck with 12.04 due to the move towards requiring 3D GPU acceleration in later versions. It seems that the choices on ARM now are to either have a crippled distro through lack of open GPU support, or stick with an old distro that doesn't require GPUs.

Based on other messages from you on the ML I'm assuming you have an i.MX6 system in mind? (Forgive me if I'm wrong.) On any Cortex-A9 or -A8 the VDSO isn't going to be any great benefit anyway. Those CPUs don't implement the generic timers extension, so only clock_gettime() with low-resolution clock ids will see a speedup. And few programs use those, to my knowledge.

Quoted 2 lines

A15, A7, and I'm pretty sure A12 and A17 all implement the timer extension and will see full benefit. Some Qualcomm CPUs too, I think.

Here's the results from vdsotest... yes on iMX6 as that's the most advanced bootable system I have:

clock-gettime-monotonic system calls per second: 1601780
clock-gettime-monotonic vdso calls per second: 1600862 (1.00x speedup)
clock-getres-monotonic system calls per second: 3001396
clock-getres-monotonic vdso calls per second: 52163259 (17.38x speedup)
clock-gettime-monotonic-coarse system calls per second: 2824631
clock-gettime-monotonic-coarse vdso calls per second: 13760820 (4.87x speedup)
clock-getres-monotonic-coarse system calls per second: 3166932
clock-getres-monotonic-coarse vdso calls per second: 52111805 (16.45x speedup)
clock-gettime-realtime system calls per second: 1596863
clock-gettime-realtime vdso calls per second: 1590424 (1.00x speedup)
clock-getres-realtime system calls per second: 2993748
clock-getres-realtime vdso calls per second: 52123753 (17.41x speedup)
clock-gettime-realtime-coarse system calls per second: 3007894
clock-gettime-realtime-coarse vdso calls per second: 17081617 (5.68x speedup)
clock-getres-realtime-coarse system calls per second: 3072073

clock-getres-realtime-coarse vdso calls per second: 49401452 (16.08x speedup)
Note: vDSO version of getcpu not found getcpu system calls per second: 4902323
getcpu vdso calls per second: 4972290 (1.01x speedup)
Note: vDSO version of getcpu not found
Note: vDSO version of getcpu not found gettimeofday system calls per second: 1637099
gettimeofday vdso calls per second: 1637573 (1.00x speedup)

That's consistent with my results on iMX6. The reported 1.00x "speedup" for clock-gettime-monotonic etc indicates the VDSO is falling back to syscall.

Thanks for testing.

Quoted 5 lines · Nathan Lynch

That's consistent with my results on iMX6. The reported 1.00x "speedup" for clock-gettime-monotonic etc indicates the VDSO is falling back to syscall.

Thanks for testing.

Here's another issue which cropped up when I ran this patch set through the autobuilder last night - allnoconfig's now fail with:

mm/memory.c: In function 'gate_vma_init':
mm/memory.c:3410:22: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3411:20: error: 'FIXADDR_USER_END' undeclared (first use in this function) mm/memory.c: In function 'in_gate_area_no_mm':
mm/memory.c:3432:15: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3432:46: error: 'FIXADDR_USER_END' undeclared (first use in this function) make[2]: *** [mm/memory.o] Error 1

Quoted 10 lines · Russell King - ARM Linux

Quoted 5 lines · Nathan Lynch

That's consistent with my results on iMX6. The reported 1.00x "speedup" for clock-gettime-monotonic etc indicates the VDSO is falling back to syscall.

Thanks for testing.

Here's another issue which cropped up when I ran this patch set through the autobuilder last night - allnoconfig's now fail with:

Quoted 7 lines

mm/memory.c: In function 'gate_vma_init':
mm/memory.c:3410:22: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3411:20: error: 'FIXADDR_USER_END' undeclared (first use in this function) mm/memory.c: In function 'in_gate_area_no_mm':
mm/memory.c:3432:15: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3432:46: error: 'FIXADDR_USER_END' undeclared (first use in this function) make[2]: *** [mm/memory.o] Error 1

arch/arm/include/page.h:
#ifdef CONFIG_KUSER_HELPERS
#define __HAVE_ARCH_GATE_AREA 1
#endif

mm/memory.c:
#if !defined(__HAVE_ARCH_GATE_AREA)

#if defined(AT_SYSINFO_EHDR)
static struct vm_area_struct gate_vma;

static int __init gate_vma_init(void)

{
	gate_vma.vm_mm = NULL;
	gate_vma.vm_start = FIXADDR_USER_START;
	gate_vma.vm_end = FIXADDR_USER_END;

...

The vdso patches add an ARM definition for AT_SYSINFO_EHDR. So when CONFIG_KUSER_HELPERS=n, this code is pulled in now...

Not sure what the fix would be right now. I don't understand why there is this relationship between AT_SYSINFO_EHDR and gate vma code.

Quoted 40 lines · Nathan Lynch

Quoted 10 lines · Russell King - ARM Linux

Quoted 5 lines · Nathan Lynch

That's consistent with my results on iMX6. The reported 1.00x "speedup" for clock-gettime-monotonic etc indicates the VDSO is falling back to syscall.

Thanks for testing.

Here's another issue which cropped up when I ran this patch set through the autobuilder last night - allnoconfig's now fail with:

Quoted 7 lines

mm/memory.c: In function 'gate_vma_init':
mm/memory.c:3410:22: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3411:20: error: 'FIXADDR_USER_END' undeclared (first use in this function) mm/memory.c: In function 'in_gate_area_no_mm':
mm/memory.c:3432:15: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3432:46: error: 'FIXADDR_USER_END' undeclared (first use in this function) make[2]: *** [mm/memory.o] Error 1

arch/arm/include/page.h:
#ifdef CONFIG_KUSER_HELPERS
#define __HAVE_ARCH_GATE_AREA 1
#endif

mm/memory.c:
#if !defined(__HAVE_ARCH_GATE_AREA)

#if defined(AT_SYSINFO_EHDR)
static struct vm_area_struct gate_vma;

static int __init gate_vma_init(void)

{
	gate_vma.vm_mm = NULL;
	gate_vma.vm_start = FIXADDR_USER_START;
	gate_vma.vm_end = FIXADDR_USER_END;

...

The vdso patches add an ARM definition for AT_SYSINFO_EHDR. So when CONFIG_KUSER_HELPERS=n, this code is pulled in now...

Quoted 2 lines

Not sure what the fix would be right now. I don't understand why there is this relationship between AT_SYSINFO_EHDR and gate vma code.

Me neither. It looks like changing those tests for AT_SYSINFO_EHDR to something like __HAVE_GATE_VMA or CONFIG_HAVE_GATE_VMA would be a good step, so we can keep this disabled on ARM. I don't see a need for the gate VMA stuff just because we have a vDSO.

From 4ad05345625d6f6a045ebe5ed355e3a2c2272fd4 Mon Sep 17 00:00:00 2001

Quoted 40 lines · Nathan Lynch

Quoted 10 lines · Russell King - ARM Linux

Quoted 5 lines · Nathan Lynch

That's consistent with my results on iMX6. The reported 1.00x "speedup" for clock-gettime-monotonic etc indicates the VDSO is falling back to syscall.

Thanks for testing.

Here's another issue which cropped up when I ran this patch set through the autobuilder last night - allnoconfig's now fail with:

Quoted 7 lines

mm/memory.c: In function 'gate_vma_init':
mm/memory.c:3410:22: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3411:20: error: 'FIXADDR_USER_END' undeclared (first use in this function) mm/memory.c: In function 'in_gate_area_no_mm':
mm/memory.c:3432:15: error: 'FIXADDR_USER_START' undeclared (first use in this function) mm/memory.c:3432:46: error: 'FIXADDR_USER_END' undeclared (first use in this function) make[2]: *** [mm/memory.o] Error 1

arch/arm/include/page.h:
#ifdef CONFIG_KUSER_HELPERS
#define __HAVE_ARCH_GATE_AREA 1
#endif

mm/memory.c:
#if !defined(__HAVE_ARCH_GATE_AREA)

#if defined(AT_SYSINFO_EHDR)
static struct vm_area_struct gate_vma;

static int __init gate_vma_init(void)

{
	gate_vma.vm_mm = NULL;
	gate_vma.vm_start = FIXADDR_USER_START;
	gate_vma.vm_end = FIXADDR_USER_END;

...

The vdso patches add an ARM definition for AT_SYSINFO_EHDR. So when CONFIG_KUSER_HELPERS=n, this code is pulled in now...

Quoted 2 lines

Not sure what the fix would be right now. I don't understand why there is this relationship between AT_SYSINFO_EHDR and gate vma code.

Me neither. It looks like changing those tests for AT_SYSINFO_EHDR to something like __HAVE_GATE_VMA or CONFIG_HAVE_GATE_VMA would be a good step, so we can keep this disabled on ARM. I don't see a need for the gate VMA stuff just because we have a vDSO.

Quoted 9 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Okay, how is this used by userspace?

It seems that on ARM, we generate a dso with these symbols:

DYNAMIC SYMBOL TABLE:

000001e4 l    d  .eh_frame_hdr	00000000              .eh_frame_hdr
00000000 g    DO *ABS*	00000000  LINUX_3.17  LINUX_3.17
000003c4 g    DF .text	00000080  LINUX_3.17  __kernel_clock_getres
00000444 g    DF .text	0000002c  LINUX_3.17  __kernel_gettimeofday
00000298 g    DF .text	0000012c  LINUX_3.17  __kernel_clock_gettime

Quoted 13 lines · Russell King - ARM Linux

Quoted 9 lines · Nathan Lynch

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Okay, how is this used by userspace?

Quoted 8 lines

It seems that on ARM, we generate a dso with these symbols:

DYNAMIC SYMBOL TABLE:

000001e4 l    d  .eh_frame_hdr    00000000              .eh_frame_hdr
00000000 g    DO *ABS*    00000000  LINUX_3.17  LINUX_3.17
000003c4 g    DF .text    00000080  LINUX_3.17  __kernel_clock_getres
00000444 g    DF .text    0000002c  LINUX_3.17  __kernel_gettimeofday
00000298 g    DF .text    0000012c  LINUX_3.17  __kernel_clock_gettime

Sorry, late to the thread.

I think that, if your function signatures match, you should give them the same names and versions as for x86 (i.e. LINUX_2.6, __vdso_clock_gettime). Userspace will thank you.

Don't do the weak clock_gettime, etc aliases, though. That was never a good idea to begin with.

Place vdso-related user-space code in arch/arm/kernel/vdso/.

It is almost completely written in C with some assembly helpers to load the data page address, sample the counter, and fall back to system calls when necessary.

If CONFIG_ARM_ARCH_TIMER is not enabled, the vdso cannot service high-resolution clocks and falls back to syscalls. Low-resolution clocks e.g. CLOCK_REALTIME_COARSE can be serviced regardless.

Of particular note is that a post-processing step ("vdsomunge") is necessary to produce a shared object which is architecturally allowed to be used by both soft- and hard-float EABI programs.

The 2012 edition of the ARM ABI defines Tag_ABI_VFP_args = 3 "Code is compatible with both the base and VFP variants; the user did not permit non-variadic functions to pass FP parameters/results." Unfortunately current toolchains do not support this tag, which is ideally what we would use.

The best available option is to ensure that both EF_ARM_ABI_FLOAT_SOFT and EF_ARM_ABI_FLOAT_HARD are unset in the ELF header's e_flags, indicating that the shared object is "old" and should be accepted for backward compatibility's sake. While binutils < 2.24 appear to produce a vdso.so with both flags clear, 2.24 always sets EF_ARM_ABI_FLOAT_SOFT, with no way to inhibit this behavior. So we have to fix things up with a custom post-processing step.

In fact, the VDSO code in glibc does much less validation (including checking these flags) than the code for handling conventional file-backed shared libraries, so this is a bit moot unless glibc's VDSO code becomes more strict.

Signed-off-by Nathan Lynch <[email protected]>

arch/arm/kernel/vdso/Makefile new file +1 -0

new file mode 100644
index 000000000000..ceb712a7626c
@@ -0,0 +1,59 @@
+hostprogs-y := vdsomunge

arch/arm/kernel/vdso/checkundef.sh new file +0 -0

new file mode 100755
index 000000000000..185c30da202b

arch/arm/kernel/vdso/datapage.S new file +5 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+	.align 2
+.L_vdso_data_ptr:

arch/arm/kernel/vdso/datapage.S new file +19 -0

+
+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+#define HOST_ORDER ELFDATA2LSB
+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+#define HOST_ORDER ELFDATA2MSB
+#endif
+
+static const char *outfile;
+
+static void cleanup(void)
+{
+	if (error_message_count > 0 && outfile != NULL)
+		unlink(outfile);
+}
+
+static Elf32_Word read_elf_word(Elf32_Word word, bool swap)
+{
+	return swap ? bswap_32(word) : word;
+}

Quoted 15 lines · Nathan Lynch

arch/arm/kernel/vdso/datapage.S new file +9 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+	.align 2
+.L_vdso_data_ptr:
+	.long	_start - . - VDSO_DATA_SIZE
+
+ENTRY(__get_datapage)
+	.cfi_startproc

arch/arm/kernel/vdso/datapage.S line 15

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+	adr	r0, .L_vdso_data_ptr
+	ldr	r1, [r0]
+	add	r0, r0, r1
+	bx	lr
+	.cfi_endproc
+ENDPROC(__get_datapage)

Can you translate this into English for the non-ARM-speakers here?

Quoted 16 lines · Andy Lutomirski

Quoted 15 lines · Nathan Lynch

arch/arm/kernel/vdso/datapage.S new file +9 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+	.align 2
+.L_vdso_data_ptr:
+	.long	_start - . - VDSO_DATA_SIZE
+
+ENTRY(__get_datapage)
+	.cfi_startproc

Quoted 8 lines

arch/arm/kernel/vdso/datapage.S line 15

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+	adr	r0, .L_vdso_data_ptr
+	ldr	r1, [r0]
+	add	r0, r0, r1
+	bx	lr
+	.cfi_endproc
+ENDPROC(__get_datapage)

Can you translate this into English for the non-ARM-speakers here?

Also, I'm not sure .cfi directives are the right things to use for AArch32. ARM has special .fnstart, .fnend, .movsp diretives for its own unwind format.

Maybe GDB works with both -- have you tried unwinding out of this with anything?

Quoted 30 lines · Will Deacon

Quoted 16 lines · Andy Lutomirski

Quoted 15 lines · Nathan Lynch

arch/arm/kernel/vdso/datapage.S new file +9 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+   .align 2
+.L_vdso_data_ptr:
+   .long   _start - . - VDSO_DATA_SIZE
+
+ENTRY(__get_datapage)
+   .cfi_startproc

Quoted 8 lines

arch/arm/kernel/vdso/datapage.S line 15

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+   adr     r0, .L_vdso_data_ptr
+   ldr     r1, [r0]
+   add     r0, r0, r1
+   bx      lr
+   .cfi_endproc
+ENDPROC(__get_datapage)

Can you translate this into English for the non-ARM-speakers here?

Also, I'm not sure .cfi directives are the right things to use for AArch32. ARM has special .fnstart, .fnend, .movsp diretives for its own unwind format.

Quoted 2 lines

Maybe GDB works with both -- have you tried unwinding out of this with anything?

Not since it stopped crashing, which was months ago :-)

I think it worked with GDB 7.5 at least.

arch/arm/kernel/vdso/Makefile line 11

Quoted 8 lines · Nathan Lynch

arch/arm/kernel/vdso/Makefile new file +8 -0

+
+# Build rules
+targets := $(obj-vdso) vdso.so vdso.so.dbg vdso.so.raw vdso.lds
+obj-vdso := $(addprefix $(obj)/, $(obj-vdso))
+
+ccflags-y := -shared -fPIC -fno-common -fno-builtin -fno-stack-protector
+ccflags-y += -nostdlib -Wl,-soname=linux-vdso.so.1 \
+		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)

Does this need -DDISABLE_BRANCH_PROFILING?

Quoted 6 lines

arch/arm/kernel/vdso/checkundef.sh new file +0 -0

new file mode 100755
index 000000000000..185c30da202b

arch/arm/kernel/vdso/checkundef.sh line 9

Quoted 9 lines

arch/arm/kernel/vdso/checkundef.sh new file +9 -0

@@ -0,0 +1,9 @@
+#!/bin/sh
+nm="$1"
+file="$2"
+"$nm" -u "$file" | ( ret=0; while read discard symbol
+do
+    echo "$file: undefined symbol $symbol"
+    ret=1
+done ; exit $ret )
+exit $?

This is just as buggy as the x86 version. make;make malfunctions.

Can you rely on a new enough toolchain to use -Wl,--no-undefined?

Quoted 15 lines

arch/arm/kernel/vdso/datapage.S new file +9 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+	.align 2
+.L_vdso_data_ptr:
+	.long	_start - . - VDSO_DATA_SIZE
+
+ENTRY(__get_datapage)
+	.cfi_startproc

arch/arm/kernel/vdso/datapage.S line 15

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+	adr	r0, .L_vdso_data_ptr
+	ldr	r1, [r0]
+	add	r0, r0, r1
+	bx	lr
+	.cfi_endproc
+ENDPROC(__get_datapage)

Can you translate this into English for the non-ARM-speakers here?

Quoted 22 lines

arch/arm/kernel/vdso/datapage.S new file +22 -0

+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+#define HOST_ORDER ELFDATA2LSB
+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+#define HOST_ORDER ELFDATA2MSB
+#endif
+
+static const char *outfile;
+
+static void cleanup(void)
+{
+	if (error_message_count > 0 && outfile != NULL)
+		unlink(outfile);
+}
+
+static Elf32_Word read_elf_word(Elf32_Word word, bool swap)
+{
+	return swap ? bswap_32(word) : word;
+}
+
+static Elf32_Half read_elf_half(Elf32_Half half, bool swap)
+{
+	return swap ? bswap_16(half) : half;

arch/arm/kernel/vdso/datapage.S

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+}
+
+static void write_elf_word(Elf32_Word val, Elf32_Word *dst, bool swap)
+{
+	*dst = swap ? bswap_32(val) : val;
+}

The macros in arch/x86/vdso/vdso2c.c are IMO much nicer.

Quoted 11 lines · Andy Lutomirski

arch/arm/kernel/vdso/Makefile line 11

Quoted 8 lines · Nathan Lynch

arch/arm/kernel/vdso/Makefile new file +8 -0

+
+# Build rules
+targets := $(obj-vdso) vdso.so vdso.so.dbg vdso.so.raw vdso.lds
+obj-vdso := $(addprefix $(obj)/, $(obj-vdso))
+
+ccflags-y := -shared -fPIC -fno-common -fno-builtin -fno-stack-protector
+ccflags-y += -nostdlib -Wl,-soname=linux-vdso.so.1 \
+            $(call cc-ldoption, -Wl$(comma)--hash-style=sysv)

Does this need -DDISABLE_BRANCH_PROFILING?

Yes, thanks.

Quoted 18 lines

Quoted 6 lines

arch/arm/kernel/vdso/checkundef.sh new file +0 -0

new file mode 100755
index 000000000000..185c30da202b

arch/arm/kernel/vdso/checkundef.sh line 9

Quoted 9 lines

arch/arm/kernel/vdso/checkundef.sh new file +9 -0

@@ -0,0 +1,9 @@
+#!/bin/sh
+nm="$1"
+file="$2"
+"$nm" -u "$file" | ( ret=0; while read discard symbol
+do
+    echo "$file: undefined symbol $symbol"
+    ret=1
+done ; exit $ret )
+exit $?

This is just as buggy as the x86 version. make;make malfunctions.

Quoted 1 line

Can you rely on a new enough toolchain to use -Wl,--no-undefined?

Looks like relying on --no-undefined should be okay.

Quoted 15 lines

Quoted 15 lines

arch/arm/kernel/vdso/datapage.S new file +9 -0

new file mode 100644
index 000000000000..fbf36d75da06
@@ -0,0 +1,15 @@
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+
+    .align 2
+.L_vdso_data_ptr:
+    .long   _start - . - VDSO_DATA_SIZE
+
+ENTRY(__get_datapage)
+    .cfi_startproc

Quoted 8 lines

arch/arm/kernel/vdso/datapage.S line 15

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+    adr     r0, .L_vdso_data_ptr
+    ldr     r1, [r0]
+    add     r0, r0, r1
+    bx      lr
+    .cfi_endproc
+ENDPROC(__get_datapage)

Can you translate this into English for the non-ARM-speakers here?

It's a PC-relative load of the data page. If someone knows how to make this happen in the C portion of the vdso, it would might get us a little speedup.

Quoted 22 lines

Quoted 22 lines

arch/arm/kernel/vdso/datapage.S new file +22 -0

+#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
+#define HOST_ORDER ELFDATA2LSB
+#elif __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+#define HOST_ORDER ELFDATA2MSB
+#endif
+
+static const char *outfile;
+
+static void cleanup(void)
+{
+    if (error_message_count > 0 && outfile != NULL)
+            unlink(outfile);
+}
+
+static Elf32_Word read_elf_word(Elf32_Word word, bool swap)
+{
+    return swap ? bswap_32(word) : word;
+}
+
+static Elf32_Half read_elf_half(Elf32_Half half, bool swap)
+{
+    return swap ? bswap_16(half) : half;

Quoted 8 lines

arch/arm/kernel/vdso/datapage.S

Quoted 6 lines

arch/arm/kernel/vdso/datapage.S new file +6 -0

+}
+
+static void write_elf_word(Elf32_Word val, Elf32_Word *dst, bool swap)
+{
+    *dst = swap ? bswap_32(val) : val;
+}

The macros in arch/x86/vdso/vdso2c.c are IMO much nicer.

Respectfully, I think I'd disagree, but more importantly, vdso2c.c seems to assume a little-endian target.

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

Signed-off-by Nathan Lynch <[email protected]>

---
 arch/arm/kernel/process.c |  13 +++-
 arch/arm/kernel/vdso.c    | 168 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 179 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/kernel/vdso.c

arch/arm/kernel/process.c +11 -2

index 40445fb71ac9..39b0d68aa068 100644
@@ -520,6 +520,7 @@ extern struct page *get_signal_page(void);
 int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 {
 	struct mm_struct *mm = current->mm;
+	unsigned long npages;
 	unsigned long addr;
 	unsigned long hint;
 	int ret;

@@ -529,9 +530,11 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 	if (!signal_page)
 		return -ENOMEM;
 
+	npages = (vdso_mapping_len >> PAGE_SHIFT) + 1;
+
 	down_write(&mm->mmap_sem);
-	hint = vdso_addr(mm, 1);
-	addr = get_unmapped_area(NULL, hint, PAGE_SIZE, 0, 0);
+	hint = vdso_addr(mm, npages);
+	addr = get_unmapped_area(NULL, hint, npages, 0, 0);
 	if (IS_ERR_VALUE(addr)) {
 		ret = addr;
 		goto up_fail;

@@ -544,6 +547,12 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 	if (ret == 0)
 		mm->context.sigpage = addr;
 
+	/* Unlike the sigpage, failure to install the vdso is unlikely
+	 * to be fatal to the process, so no error check needed
+	 * here.
+	 */
+	arm_install_vdso(mm, addr + PAGE_SIZE);
+
  up_fail:
 	up_write(&mm->mmap_sem);
 	return ret;

arch/arm/kernel/vdso.c new file +85 -0

new file mode 100644
index 000000000000..0cfd25f09adf
@@ -0,0 +1,168 @@
+/*
+ * Adapted from arm64 version.
+ *
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/err.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/slab.h>
+#include <linux/timekeeper_internal.h>
+#include <linux/vmalloc.h>
+
+#include <asm/barrier.h>
+#include <asm/cacheflush.h>
+#include <asm/page.h>
+#include <asm/vdso.h>
+#include <asm/vdso_datapage.h>
+
+static struct page **vdso_pagelist;
+
+unsigned long vdso_mapping_len __read_mostly;
+
+/*
+ * The vDSO data page.
+ */
+static union vdso_data_store vdso_data_store __page_aligned_data;
+static struct vdso_data *vdso_data = &vdso_data_store.data;
+
+static int __init vdso_init(void)
+{
+	unsigned long vdso_pages;
+	int i;
+
+	if (memcmp(&vdso_start, "\177ELF", 4)) {
+		pr_err("vDSO is not a valid ELF object!\n");
+		return -ENOEXEC;
+	}
+
+	vdso_pages = (&vdso_end - &vdso_start) >> PAGE_SHIFT;
+	pr_debug("vdso: %ld code pages at base %p\n", vdso_pages, &vdso_start);
+
+	/* Allocate the vDSO pagelist, plus a page for the data. */
+	vdso_pagelist = kcalloc(vdso_pages + 1, sizeof(struct page *),
+				GFP_KERNEL);
+	if (vdso_pagelist == NULL)
+		return -ENOMEM;
+
+	/* Grab the vDSO data page. */
+	vdso_pagelist[0] = virt_to_page(vdso_data);
+
+	/* Grab the vDSO code pages. */
+	for (i = 0; i < vdso_pages; i++)
+		vdso_pagelist[i + 1] = virt_to_page(&vdso_start + i * PAGE_SIZE);
+
+	/* Precompute the mapping size */
+	vdso_mapping_len = (vdso_pages + 1) << PAGE_SHIFT;
+
+	return 0;
+}
+arch_initcall(vdso_init);
+
+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+	int ret;
+
+	mm->context.vdso = ~0UL;
+
+	if (vdso_pagelist == NULL)
+		return;
+
+	/*

Quoted 7 lines · Nathan Lynch

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

Quoted 16 lines

arch/arm/kernel/vdso.c new file +16 -0

+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+	int ret;
+
+	mm->context.vdso = ~0UL;
+
+	if (vdso_pagelist == NULL)
+		return;
+
+	/*
+	 * Put vDSO base into mm struct before calling
+	 * install_special_mapping so the perf counter mmap tracking
+	 * code will recognise it as a vDSO.
+	 */
+	mm->context.vdso = addr;

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+	ret = install_special_mapping(mm, addr, vdso_mapping_len,
+				      VM_READ|VM_EXEC|
+				      VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+				      vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

Quoted 8 lines · Andy Lutomirski

Quoted 7 lines · Nathan Lynch

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

[...]

Quoted 16 lines

Quoted 16 lines

arch/arm/kernel/vdso.c new file +16 -0

+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+	int ret;
+
+	mm->context.vdso = ~0UL;
+
+	if (vdso_pagelist == NULL)
+		return;
+
+	/*
+	 * Put vDSO base into mm struct before calling
+	 * install_special_mapping so the perf counter mmap tracking
+	 * code will recognise it as a vDSO.
+	 */
+	mm->context.vdso = addr;

Quoted 8 lines

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+	ret = install_special_mapping(mm, addr, vdso_mapping_len,
+				      VM_READ|VM_EXEC|
+				      VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+				      vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

Hmm, but how else can we support software breakpoints on the vdso?

Quoted 37 lines · Will Deacon

Quoted 8 lines · Andy Lutomirski

Quoted 7 lines · Nathan Lynch

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

[...]

Quoted 24 lines

Quoted 11 lines

arch/arm/kernel/vdso.c new file +11 -0

+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+	int ret;
+
+	mm->context.vdso = ~0UL;
+
+	if (vdso_pagelist == NULL)
+		return;
+
+	/*

arch/arm/kernel/vdso.c line 95

Quoted 10 lines

arch/arm/kernel/vdso.c new file +10 -0

+	 * Put vDSO base into mm struct before calling
+	 * install_special_mapping so the perf counter mmap tracking
+	 * code will recognise it as a vDSO.
+	 */
+	mm->context.vdso = addr;
+
+	ret = install_special_mapping(mm, addr, vdso_mapping_len,
+				      VM_READ|VM_EXEC|
+				      VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+				      vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

Quoted 1 line

Hmm, but how else can we support software breakpoints on the vdso?

I believe Andy is suggesting separate VMAs (with different VM flags) for the VDSO's data and code. So, breakpoints in code would work, but attempts to modify the data page via ptrace() would fail outright instead of silently COWing.

Quoted 40 lines · Nathan Lynch

Quoted 38 lines · Will Deacon

Quoted 8 lines · Andy Lutomirski

Quoted 7 lines · Nathan Lynch

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

[...]

Quoted 16 lines

Quoted 16 lines

arch/arm/kernel/vdso.c new file +16 -0

+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+ int ret;
+
+ mm->context.vdso = ~0UL;
+
+ if (vdso_pagelist == NULL)
+         return;
+
+ /*
+  * Put vDSO base into mm struct before calling
+  * install_special_mapping so the perf counter mmap tracking
+  * code will recognise it as a vDSO.
+  */
+ mm->context.vdso = addr;

Quoted 8 lines

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+ ret = install_special_mapping(mm, addr, vdso_mapping_len,
+                               VM_READ|VM_EXEC|
+                               VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+                               vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

Hmm, but how else can we support software breakpoints on the vdso?

Quoted 4 lines

I believe Andy is suggesting separate VMAs (with different VM flags) for the VDSO's data and code. So, breakpoints in code would work, but attempts to modify the data page via ptrace() would fail outright instead of silently COWing.

Ah, yes. That makes a lot of sense for the data page -- we should do something similar on arm64 too, since the CoW will break everything for the task being debugged. We could also drop the EXEC flags too.

Quoted 46 lines · Will Deacon

Quoted 44 lines · Nathan Lynch

Quoted 37 lines · Will Deacon

Quoted 8 lines · Andy Lutomirski

Quoted 7 lines · Nathan Lynch

Initialize the vdso page list at boot, install the vdso mapping at exec time, and update the data page during timer ticks. This code is not built if CONFIG_VDSO is not enabled.

Account for the vdso length when randomizing the offset from the stack. The vdso is placed immediately following the sigpage with a separate install_special_mapping call in arm_install_vdso.

[...]

Quoted 16 lines

Quoted 16 lines

arch/arm/kernel/vdso.c new file +16 -0

+/* assumes mmap_sem is write-locked */
+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+ int ret;
+
+ mm->context.vdso = ~0UL;
+
+ if (vdso_pagelist == NULL)
+         return;
+
+ /*
+  * Put vDSO base into mm struct before calling
+  * install_special_mapping so the perf counter mmap tracking
+  * code will recognise it as a vDSO.
+  */
+ mm->context.vdso = addr;

Quoted 8 lines

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+ ret = install_special_mapping(mm, addr, vdso_mapping_len,
+                               VM_READ|VM_EXEC|
+                               VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+                               vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

Quoted 1 line

Hmm, but how else can we support software breakpoints on the vdso?

I believe Andy is suggesting separate VMAs (with different VM flags) for the VDSO's data and code. So, breakpoints in code would work, but attempts to modify the data page via ptrace() would fail outright instead of silently COWing.

Quoted 3 lines

Ah, yes. That makes a lot of sense for the data page -- we should do something similar on arm64 too, since the CoW will break everything for the task being debugged. We could also drop the EXEC flags too.

If you do this, I have a slight preference for the new vma being called "[vvar]" to match x86. It'll make the CRIU people happy if and when they port it to ARM.

Quoted 6 lines · Will Deacon

Quoted 4 lines · Nathan Lynch

I believe Andy is suggesting separate VMAs (with different VM flags) for the VDSO's data and code. So, breakpoints in code would work, but attempts to modify the data page via ptrace() would fail outright instead of silently COWing.

Quoted 3 lines

Ah, yes. That makes a lot of sense for the data page -- we should do something similar on arm64 too, since the CoW will break everything for the task being debugged. We could also drop the EXEC flags too.

If you do this, I have a slight preference for the new vma being called "[vvar]" to match x86. It'll make the CRIU people happy if and when they port it to ARM.

Hi Andy,

Quoted 7 lines · Andy Lutomirski

Quoted 6 lines · Will Deacon

Quoted 4 lines · Nathan Lynch

I believe Andy is suggesting separate VMAs (with different VM flags) for the VDSO's data and code. So, breakpoints in code would work, but attempts to modify the data page via ptrace() would fail outright instead of silently COWing.

Quoted 7 lines

Quoted 3 lines

Ah, yes. That makes a lot of sense for the data page -- we should do something similar on arm64 too, since the CoW will break everything for the task being debugged. We could also drop the EXEC flags too.

If you do this, I have a slight preference for the new vma being called "[vvar]" to match x86. It'll make the CRIU people happy if and when they port it to ARM.

I quickly hacked something (see below) and now I see the following in /proc/$$/maps:

7fa1574000-7fa1575000 r-xp 00000000 00:00 0 [vdso]
7fa1575000-7fa1576000 r--p 00000000 00:00 0 [vvar]

Is that what you're after?

Caveat 2: (major) I'm kind of surprised that this, or the current code, works reliably. You're doing something that I tried briefly for x86_64:

        _end = .;
        PROVIDE(end = .);

        . = ALIGN(PAGE_SIZE);
        PROVIDE(_vdso_data = .);

This sounds great, except that you're assuming that vdso_end - vdso_start == ALIGN(_end, PAGE_SIZE) - (vdso base address).

If you *fully* strip the vdso (eu-strip --strip-sections), then this is true: eu-strip --strip-sections outputs just the PT_LOAD piece of the vdso. But any binutils-generated incompletely stripped ELF image contains a section table and possible non-allocatable sections at the end. If these exceed the amount of unused space in the last PT_LOAD page, then they'll spill into the next page, and _vdso_data in the vdso will no longer match the address at which vdso.c loads it. Boom!

I bet you're getting away with this because the whole arm64 vdso seems to be written in assembly, so it seems extremely unlikely to exceed one page minus a few hundred bytes. But if you start adding complexity, you might get unlucky.

Quoted 21 lines · Andy Lutomirski

Caveat 2: (major) I'm kind of surprised that this, or the current code, works reliably. You're doing something that I tried briefly for x86_64:

        _end = .;
        PROVIDE(end = .);

        . = ALIGN(PAGE_SIZE);
        PROVIDE(_vdso_data = .);

This sounds great, except that you're assuming that vdso_end - vdso_start == ALIGN(_end, PAGE_SIZE) - (vdso base address).

If you *fully* strip the vdso (eu-strip --strip-sections), then this is true: eu-strip --strip-sections outputs just the PT_LOAD piece of the vdso. But any binutils-generated incompletely stripped ELF image contains a section table and possible non-allocatable sections at the end. If these exceed the amount of unused space in the last PT_LOAD page, then they'll spill into the next page, and _vdso_data in the vdso will no longer match the address at which vdso.c loads it. Boom!

Quoted 4 lines

I bet you're getting away with this because the whole arm64 vdso seems to be written in assembly, so it seems extremely unlikely to exceed one page minus a few hundred bytes. But if you start adding complexity, you might get unlucky.

This is why I switched (in v5) the proposed 32-bit ARM VDSO to place the data page before the code -- adding -frecord-gcc-switches to the compiler flags was enough to break it.

I meant to call Will's attention to it at the time for arm64's sake, but I guess it slipped my mind... sorry.

Quoted 31 lines · Nathan Lynch

Quoted 21 lines · Andy Lutomirski

Caveat 2: (major) I'm kind of surprised that this, or the current code, works reliably. You're doing something that I tried briefly for x86_64:

        _end = .;
        PROVIDE(end = .);

        . = ALIGN(PAGE_SIZE);
        PROVIDE(_vdso_data = .);

This sounds great, except that you're assuming that vdso_end - vdso_start == ALIGN(_end, PAGE_SIZE) - (vdso base address).

If you *fully* strip the vdso (eu-strip --strip-sections), then this is true: eu-strip --strip-sections outputs just the PT_LOAD piece of the vdso. But any binutils-generated incompletely stripped ELF image contains a section table and possible non-allocatable sections at the end. If these exceed the amount of unused space in the last PT_LOAD page, then they'll spill into the next page, and _vdso_data in the vdso will no longer match the address at which vdso.c loads it. Boom!

Quoted 4 lines

I bet you're getting away with this because the whole arm64 vdso seems to be written in assembly, so it seems extremely unlikely to exceed one page minus a few hundred bytes. But if you start adding complexity, you might get unlucky.

This is why I switched (in v5) the proposed 32-bit ARM VDSO to place the data page before the code -- adding -frecord-gcc-switches to the compiler flags was enough to break it.

Quoted 2 lines

I meant to call Will's attention to it at the time for arm64's sake, but I guess it slipped my mind... sorry.

Hmm, so I could definitely look at doing the same thing, but I don't know if we actually need to for arm64. As Andy points out, we're written entirely in assembly and we objcopy -S to create the vdso.so. I've dumped the headers below and everything appears to be PT_LOAD.

Quoted 32 lines · Will Deacon

Quoted 31 lines · Nathan Lynch

Quoted 21 lines · Andy Lutomirski

Caveat 2: (major) I'm kind of surprised that this, or the current code, works reliably. You're doing something that I tried briefly for x86_64:

        _end = .;
        PROVIDE(end = .);

        . = ALIGN(PAGE_SIZE);
        PROVIDE(_vdso_data = .);

This sounds great, except that you're assuming that vdso_end - vdso_start == ALIGN(_end, PAGE_SIZE) - (vdso base address).

If you *fully* strip the vdso (eu-strip --strip-sections), then this is true: eu-strip --strip-sections outputs just the PT_LOAD piece of the vdso. But any binutils-generated incompletely stripped ELF image contains a section table and possible non-allocatable sections at the end. If these exceed the amount of unused space in the last PT_LOAD page, then they'll spill into the next page, and _vdso_data in the vdso will no longer match the address at which vdso.c loads it. Boom!

Quoted 4 lines

I bet you're getting away with this because the whole arm64 vdso seems to be written in assembly, so it seems extremely unlikely to exceed one page minus a few hundred bytes. But if you start adding complexity, you might get unlucky.

This is why I switched (in v5) the proposed 32-bit ARM VDSO to place the data page before the code -- adding -frecord-gcc-switches to the compiler flags was enough to break it.

Quoted 7 lines

Quoted 2 lines

I meant to call Will's attention to it at the time for arm64's sake, but I guess it slipped my mind... sorry.

Hmm, so I could definitely look at doing the same thing, but I don't know if we actually need to for arm64. As Andy points out, we're written entirely in assembly and we objcopy -S to create the vdso.so. I've dumped the headers below and everything appears to be PT_LOAD.

Your dump doesn't show the location of the section and section string tables themselves. Try:

eu-readelf -l -h -S whatever.so

Quoted 9 lines · Andy Lutomirski

Quoted 7 lines · Will Deacon

Quoted 1 line · Nathan Lynch

On 07/02/2014 10:54 AM, Andy Lutomirski wrote:

Hmm, so I could definitely look at doing the same thing, but I don't know if we actually need to for arm64. As Andy points out, we're written entirely in assembly and we objcopy -S to create the vdso.so. I've dumped the headers below and everything appears to be PT_LOAD.

Quoted 4 lines

Your dump doesn't show the location of the section and section string tables themselves. Try:

eu-readelf -l -h -S whatever.so

Thanks. See below.

Will

--->8

ELF Header:

  Start of section headers:          1888 (bytes into file)
  Size of section headers:           64 (bytes)
  Number of section headers:         14

Section header string table index: 13

Section Headers:
[13] .shstrtab STRTAB 0000000000000000 000006e8

       0000000000000078  0000000000000000           0     0     1

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000

                 0x00000000000006e8 0x00000000000006e8  R E    10

Quoted 4 lines · Will Deacon

ELF Header:

  Start of section headers:          1888 (bytes into file)
  Size of section headers:           64 (bytes)
  Number of section headers:         14

That's 896 bytes for the section table, starting at offset 1888.

Quoted 7 lines

Section header string table index: 13

Section Headers:
[13] .shstrtab STRTAB 0000000000000000 000006e8

       0000000000000078  0000000000000000           0     0     1

120 bytes of section headers strings, starting at offset 1768 (and not allocatable -- no 'A' here).

Quoted 6 lines

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000

                 0x00000000000006e8 0x00000000000006e8  R E    10

The loadable segment is 1768 bytes long, starting at the beginning of the file (and therefore the beginning of the mapping).

So you have a total of 1016 bytes of non-allocatable stuff at the end that I've accounted for. I assume that the whole file is 2784 bytes.

If you added text or data to bring PT_LOAD to between 3081 and 4095 bytes, then the section headers and/or section string table would cross a page boundary.

Cheers for this, Andy.

Quoted 26 lines · Andy Lutomirski

Quoted 4 lines · Will Deacon

ELF Header:

  Start of section headers:          1888 (bytes into file)
  Size of section headers:           64 (bytes)
  Number of section headers:         14

That's 896 bytes for the section table, starting at offset 1888.

Quoted 7 lines

Section header string table index: 13

Section Headers:
[13] .shstrtab STRTAB 0000000000000000 000006e8

       0000000000000078  0000000000000000           0     0     1

120 bytes of section headers strings, starting at offset 1768 (and not allocatable -- no 'A' here).

Quoted 6 lines

Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000000000 0x0000000000000000 0x0000000000000000

                 0x00000000000006e8 0x00000000000006e8  R E    10

Quoted 5 lines

The loadable segment is 1768 bytes long, starting at the beginning of the file (and therefore the beginning of the mapping).

So you have a total of 1016 bytes of non-allocatable stuff at the end that I've accounted for. I assume that the whole file is 2784 bytes.

Correct.

Quoted 3 lines

If you added text or data to bring PT_LOAD to between 3081 and 4095 bytes, then the section headers and/or section string table would cross a page boundary.

Ok, so that explains why things are working at the moment and it sounds like we have some wiggle room too. I'll look at moving the datapage anyway, since having these artificial limits is error-prone and fragile, but I don't think it's something we need to panic about immediately.

Also, if you get to the bottom of your binutils issues with the section allocation, that might work for us since we don't really have any legacy binutils supporting arm64.

Quoted 5 lines · Will Deacon

Ok, so that explains why things are working at the moment and it sounds like we have some wiggle room too. I'll look at moving the datapage anyway, since having these artificial limits is error-prone and fragile, but I don't think it's something we need to panic about immediately.

Quoted 3 lines

Also, if you get to the bottom of your binutils issues with the section allocation, that might work for us since we don't really have any legacy binutils supporting arm64.

Just in case anyone still cares about this thread, moving the vvar area back before the vdso text is queued up for 3.17. I gave up on reliably keeping binutils happy with my ugly hack.

Quoted 15 lines · Nathan Lynch

arch/arm/kernel/vdso.c new file +15 -0

+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+	int ret;
+
+	mm->context.vdso = ~0UL;
+
+	if (vdso_pagelist == NULL)
+		return;
+
+	/*
+	 * Put vDSO base into mm struct before calling
+	 * install_special_mapping so the perf counter mmap tracking
+	 * code will recognise it as a vDSO.
+	 */
+	mm->context.vdso = addr;

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+	ret = install_special_mapping(mm, addr, vdso_mapping_len,
+				      VM_READ|VM_EXEC|
+				      VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+				      vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

You also seem to
be sticking it *before* the vdso in the same vma. This will severely piss off all the tools that assume that "[vdso]" points to an ELF object.

x86 calls this thing "[vvar]" and sticks it after the vdso. You might not want to have the complexity of sticking it after the vdso (it's distinctly nontrivial), but I see nothing wrong with giving it its own vma just before the vdso. The new _install_special_mapping function makes it considerably easier to do.

Quoted 16 lines · Andy Lutomirski

Quoted 15 lines · Nathan Lynch

arch/arm/kernel/vdso.c new file +15 -0

+void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
+{
+    int ret;
+
+    mm->context.vdso = ~0UL;
+
+    if (vdso_pagelist == NULL)
+            return;
+
+    /*
+     * Put vDSO base into mm struct before calling
+     * install_special_mapping so the perf counter mmap tracking
+     * code will recognise it as a vDSO.
+     */
+    mm->context.vdso = addr;

Quoted 8 lines

arch/arm/kernel/vdso.c line 95

Quoted 5 lines

arch/arm/kernel/vdso.c new file +5 -0

+
+    ret = install_special_mapping(mm, addr, vdso_mapping_len,
+                                  VM_READ|VM_EXEC|
+                                  VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+                                  vdso_pagelist);

Eek. You're mapping the shared data VM_MAYWRITE. This will cause bizarre and confusing failures if ptrace pokes at it.

I'm aware of that. One could argue (as does the author of the equivalent code in powerpc) that this is a "well, don't do that" situation. But I tend to agree that it would be nicer to prevent this failure mode.

Quoted 3 lines

You also seem to
be sticking it *before* the vdso in the same vma. This will severely piss off all the tools that assume that "[vdso]" points to an ELF object.

Hmm, which tools? Shouldn't they be consulting AT_SYSINFO_EHDR in the auxiliary vector instead?

Quoted 6 lines

x86 calls this thing "[vvar]" and sticks it after the vdso. You might not want to have the complexity of sticking it after the vdso (it's distinctly nontrivial), but I see nothing wrong with giving it its own vma just before the vdso. The new _install_special_mapping function makes it considerably easier to do.

I'll give this a shot, thanks.

Quoted 3 lines · Nathan Lynch

Hmm, which tools? Shouldn't they be consulting AT_SYSINFO_EHDR in the auxiliary vector instead?

There were a bunch of random things written before getauxvec(3) was introduced that look for "[vdso]"

gdb is also a bit weird. I think it's changed a couple times, but IIRC it looks for a vma that has a start address that matches AT_SYSINFO_EDHR, and then it parses the ELF header and *section* headers (sigh) in that vma. So if I'm remembering correctly, it just won't notice the vdso at all of AT_SYSINFO_EHDR and the vma start aren't the same.

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Hi Nathan,

applying your patches I get the following compile error under Ubuntu 12.04:

  CC      arch/arm/kernel/vdso.o
  LDS     arch/arm/kernel/vdso/vdso.lds
  CC      arch/arm/kernel/vdso/vgettimeofday.o
  AS      arch/arm/kernel/vdso/datapage.o
  VDSOL   arch/arm/kernel/vdso/vdso.so.raw
  HOSTCC  arch/arm/kernel/vdso/vdsomunge

arch/arm/kernel/vdso/vdsomunge.c: In function ?main?:
arch/arm/kernel/vdso/vdsomunge.c:152:16: error: ?EF_ARM_ABI_FLOAT_HARD? undeclared (first use in this function) arch/arm/kernel/vdso/vdsomunge.c:152:16: note: each undeclared identifier is reported only once for each function it appears in arch/arm/kernel/vdso/vdsomunge.c:156:34: error: ?EF_ARM_ABI_FLOAT_SOFT? undeclared (first use in this function) make[2]: *** [arch/arm/kernel/vdso/vdsomunge] Error 1
make[1]: *** [arch/arm/kernel/vdso] Error 2
make: *** [arch/arm/kernel] Error 2

Looks like the libc dev is too old (EF_ARM_ABI_FLOAT_* not in /usr/include/elf.h). On a more recent Ubuntu it compiles fine. Not sure how to solve this, maybe disable vdso if the libc is too old?

Quoted 25 lines · Jan Glauber

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Hi Nathan,

applying your patches I get the following compile error under Ubuntu 12.04:

  CC      arch/arm/kernel/vdso.o
  LDS     arch/arm/kernel/vdso/vdso.lds
  CC      arch/arm/kernel/vdso/vgettimeofday.o
  AS      arch/arm/kernel/vdso/datapage.o
  VDSOL   arch/arm/kernel/vdso/vdso.so.raw
  HOSTCC  arch/arm/kernel/vdso/vdsomunge

arch/arm/kernel/vdso/vdsomunge.c: In function ?main?:
arch/arm/kernel/vdso/vdsomunge.c:152:16: error: ?EF_ARM_ABI_FLOAT_HARD? undeclared (first use in this function) arch/arm/kernel/vdso/vdsomunge.c:152:16: note: each undeclared identifier is reported only once for each function it appears in arch/arm/kernel/vdso/vdsomunge.c:156:34: error: ?EF_ARM_ABI_FLOAT_SOFT? undeclared (first use in this function) make[2]: *** [arch/arm/kernel/vdso/vdsomunge] Error 1
make[1]: *** [arch/arm/kernel/vdso] Error 2
make: *** [arch/arm/kernel] Error 2

Quoted 3 lines

Looks like the libc dev is too old (EF_ARM_ABI_FLOAT_* not in /usr/include/elf.h). On a more recent Ubuntu it compiles fine. Not sure how to solve this, maybe disable vdso if the libc is too old?

This is what really concerns me about the VDSO. It adds an additional dependence on the C library in order to build the kernel. What if you don't have a C library in your cross-build setup (I don't.)

Until that is solved, the VDSO support is not going in, as it will effectively render the kernel unbuildable for me.

--
FTTC broadband for 0.8mile line: now at 9.7Mbps down 460kbps up... slowly improving, and getting towards what was expected from it.

It appears that the EF_ARM_ABI_FLOAT_[SOFT|HARD] defines are just renames of

#define EF_ARM_SOFT_FLOAT 0x200
#define EF_ARM_VFP_FLOAT 0x400

so perhaps use those instead?

Quoted 4 lines

This is what really concerns me about the VDSO. It adds an additional dependence on the C library in order to build the kernel. What if you don't have a C library in your cross-build setup (I don't.)

This is a host tool, so this is about the host libc not the target libc.

Quoted 6 lines · Ard Biesheuvel

It appears that the EF_ARM_ABI_FLOAT_[SOFT|HARD] defines are just renames of

#define EF_ARM_SOFT_FLOAT 0x200
#define EF_ARM_VFP_FLOAT 0x400

so perhaps use those instead?

Or maybe provide our own definitions so we're not reliant on the host environment providing the correct definitions for these.

Quoted 7 lines

Quoted 5 lines

This is what really concerns me about the VDSO. It adds an additional dependence on the C library in order to build the kernel. What if you don't have a C library in your cross-build setup (I don't.)

This is a host tool, so this is about the host libc not the target libc.

It still sounds pretty fragile.

Since removing -lgcc from the link (one of the changes from v6 to v7), I believe the VDSO is not asking more of the toolchain or build environment than the rest of the kernel does. I have definitely built it without a target libc. Apart from the host program's use of too-recent ELF constants (now fixed), I would appreciate help identifying any remaining sources of fragility.

Quoted 6 lines · Nathan Lynch

Since removing -lgcc from the link (one of the changes from v6 to v7), I believe the VDSO is not asking more of the toolchain or build environment than the rest of the kernel does. I have definitely built it without a target libc. Apart from the host program's use of too-recent ELF constants (now fixed), I would appreciate help identifying any remaining sources of fragility.

There's still something wrong:

fs/binfmt_elf.c: In function 'create_elf_tables':
fs/binfmt_elf.c:231:35: error: 'AT_SYSINFO_EHDR' undeclared (first use in this function) fs/binfmt_elf.c:231:35: note: each undeclared identifier is reported only once for each function it appears in

The problem is the fscking generated includes. This is going to be a right pain - Arnd, is there anything which can be done about this?

Quoted 14 lines · Russell King - ARM Linux

Quoted 6 lines · Nathan Lynch

Since removing -lgcc from the link (one of the changes from v6 to v7), I believe the VDSO is not asking more of the toolchain or build environment than the rest of the kernel does. I have definitely built it without a target libc. Apart from the host program's use of too-recent ELF constants (now fixed), I would appreciate help identifying any remaining sources of fragility.

There's still something wrong:

fs/binfmt_elf.c: In function 'create_elf_tables':
fs/binfmt_elf.c:231:35: error: 'AT_SYSINFO_EHDR' undeclared (first use in this function) fs/binfmt_elf.c:231:35: note: each undeclared identifier is reported only once for each function it appears in

Quoted 2 lines

The problem is the fscking generated includes. This is going to be a right pain - Arnd, is there anything which can be done about this?

And yes, I see that you've mentioned this in the cover - but I think we /really/ need this solved before we can merge this series, otherwise we're likely to get lots of people asking about it.

Quoted 18 lines · Russell King - ARM Linux

Quoted 16 lines · Russell King - ARM Linux

Quoted 6 lines · Nathan Lynch

Since removing -lgcc from the link (one of the changes from v6 to v7), I believe the VDSO is not asking more of the toolchain or build environment than the rest of the kernel does. I have definitely built it without a target libc. Apart from the host program's use of too-recent ELF constants (now fixed), I would appreciate help identifying any remaining sources of fragility.

There's still something wrong:

fs/binfmt_elf.c: In function 'create_elf_tables':
fs/binfmt_elf.c:231:35: error: 'AT_SYSINFO_EHDR' undeclared (first use in this function) fs/binfmt_elf.c:231:35: note: each undeclared identifier is reported only once for each function it appears in

The problem is the fscking generated includes. This is going to be a right pain - Arnd, is there anything which can be done about this?

Quoted 3 lines

And yes, I see that you've mentioned this in the cover - but I think we /really/ need this solved before we can merge this series, otherwise we're likely to get lots of people asking about it.

Thinking about this some more - if this can't be resolved by kbuild, then I think we should add to arch/arm/Kconfig to detect the generated include file, print a helpful message telling the user how to fix their build tree, and error out.

What's happening is if you try and rebuild a kernel with these patches applied, you already have an arch/arm/include/generated/asm/auxvec.h which points at the empty asm-generic version. This remains after these patches are applied, so the new include of asm/auxvec.h in asm/elf.h picks up on the empty version, not the newly introduced version.

Quoted 6 lines · On Saturday 28 June 2014 10:42:19 Russell King - ARM Linux wrote:

What's happening is if you try and rebuild a kernel with these patches applied, you already have an arch/arm/include/generated/asm/auxvec.h which points at the empty asm-generic version. This remains after these patches are applied, so the new include of asm/auxvec.h in asm/elf.h picks up on the empty version, not the newly introduced version.

One more piece of the puzzle: the top-level Makefile always includes the headers from the source tree before generated headers, so in theory this should not be happening. However, for some reason this ends up as arch/arm/include/generated/asm/auxvec.h rather than arch/arm/include/generated/uapi/asm/auxvec.h, and the kernel-only headers get included before any of the uapi headers.

An easy workaround would probably be to add an arch/arm/include/asm/auxvec.h file that contains only

#include <uapi/asm/auxvec.h>

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Hi Nathan,

applying your patches I get the following compile error under Ubuntu 12.04:

  CC      arch/arm/kernel/vdso.o
  LDS     arch/arm/kernel/vdso/vdso.lds
  CC      arch/arm/kernel/vdso/vgettimeofday.o
  AS      arch/arm/kernel/vdso/datapage.o
  VDSOL   arch/arm/kernel/vdso/vdso.so.raw
  HOSTCC  arch/arm/kernel/vdso/vdsomunge

arch/arm/kernel/vdso/vdsomunge.c: In function ?main?:
arch/arm/kernel/vdso/vdsomunge.c:152:16: error: ?EF_ARM_ABI_FLOAT_HARD? undeclared (first use in this function) arch/arm/kernel/vdso/vdsomunge.c:152:16: note: each undeclared identifier is reported only once for each function it appears in arch/arm/kernel/vdso/vdsomunge.c:156:34: error: ?EF_ARM_ABI_FLOAT_SOFT? undeclared (first use in this function) make[2]: *** [arch/arm/kernel/vdso/vdsomunge] Error 1
make[1]: *** [arch/arm/kernel/vdso] Error 2
make: *** [arch/arm/kernel] Error 2

Looks like the libc dev is too old (EF_ARM_ABI_FLOAT_* not in /usr/include/elf.h). On a more recent Ubuntu it compiles fine. Not sure how to solve this, maybe disable vdso if the libc is too old?

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Clocks supported:
- CLOCK_REALTIME
- CLOCK_MONOTONIC
- CLOCK_REALTIME_COARSE
- CLOCK_MONOTONIC_COARSE

This also provides clock_getres (as arm64 does).

getcpu support is planned but not included at this time.

Note that while the high-precision realtime and monotonic clock support depends on the generic timers extension, support for clock_getres and coarse clocks is independent of the timer implementation and is provided unconditionally.

Run-time tested on OMAP5 and i.MX6, verifying that results obtained with the vdso are consistent with those obtained from the kernel. On OMAP5 I observe a 3- to 4-fold speedup for gettimeofday / CLOCK_REALTIME, with even better (if less interesting) speedups for the coarse clock ids and clock_getres.

I've been testing and benchmarking this with some custom test code which I have hosted here:

https://github.com/nlynch-mentor/vdsotest

If this series is applied to an already-built tree and CONFIG_VDSO is enabled, an incremental build may fail with:

fs/binfmt_elf.c: In function 'create_elf_tables':
fs/binfmt_elf.c:231:35: error: 'AT_SYSINFO_EHDR' undeclared (first use in this function)

This can be worked around by removing arch/arm/include/generated/asm/auxvec.h from the build tree. I lean toward attributing this to a Kbuild limitation/bug, but would appreciate extra scrutiny on the patch "ARM: miscellaneous vdso infrastructure, preparation" to see if there's anything to correct there.

Changes since v6:
- Update to 3.16-rc1.
- Remove -lgcc from link step - need to support GCC installations without libgcc.
- Force -O2 compilation to prevent GCC from emitting calls to libgcc math routines.
- Use custom post-processing to clear the EF_ARM_ABI_FLOAT_SOFT flag if set in the ELF header to produce a shared object which is architecturally allowed to be used by both soft- and hard-float code.
- Consolidate common arch timer code instead of duplicating it.
- Prevent the VDSO from attempting CP15 access on memory-only architected timer implementations by renaming the clocksource.

Changes since v5:
- Update to 3.15-rc1.
- Place vdso at a randomized offset above the stack along with the sigpage.
- Properly export asm/auxvec.h.
- Split patch into series for ease of review.

Changes since v4:
- Map data page at the beginning of the VMA to prevent orphan sections at the end of output invalidating the calculated offset.
- Move checkundef into cmd_vdsold to avoid spurious rebuilds.
- Change vdso_init message to pr_debug.
- Add -fno-stack-protector to cflags.

Changes since v3:
- Update to 3.14-rc6.
- Record vdso base in mm context before installing mapping (for the sake of perf_mmap_event).
- Use a more seqcount-like API for critical sections. Using seqcount API directly, however, would leak kernel pointers to userspace when lockdep is enabled.
- Trap instead of looping forever in division-by-zero stubs.

Changes since v2:
- Update to 3.14-rc4.
- Make vDSO configurable, depending on AEABI and MMU.
- Defer shifting of nanosecond component of timespec: fixes observed 1ns inconsistencies for CLOCK_REALTIME, CLOCK_MONOTONIC (see 45a7905fc48f for arm64 equivalent).
- Force reload of seq_count when spinning: without a memory clobber after the load of vdata->seq_count, GCC can generate code like this:

    2f8:   e59c9020        ldr     r9, [ip, #32]
    2fc:   e3190001        tst     r9, #1
    300:   1a000033        bne     3d4 <do_realtime+0x104>
    304:   f57ff05b        dmb     ish
    308:   e59c3034        ldr     r3, [ip, #52]   ; 0x34
    ...
    3d4:   eafffffe        b       3d4 <do_realtime+0x104>

- Build vdso.so with -lgcc: calls to __lshrdi3, __divsi3 sometimes emitted (especially with -Os). Override certain libgcc functions to prevent undefined symbols.
- Do not clear PG_reserved on vdso pages.
- Remove unnecessary get_page calls.
- Simplify ELF signature check during init.
- Use volatile for asm syscall fallbacks.
- Check whether vdso_pagelist is initialized in arm_install_vdso.
- Record clocksource mask in data page.
- Reduce code duplication in do_realtime, do_monotonic.
- Reduce calculations performed in critical sections.
- Simplify coarse clock handling.
- Move datapage load to its own assembly routine.
- Tune vdso_data layout and tweak field names.
- Check vdso shared object for undefined symbols during build.

Changes since v1:
- update to 3.14-rc1
- ensure cache coherency for data page
- Document the kernel-to-userspace protocol for vdso data page updates, and note that the timekeeping core prevents concurrent updates.
- update wall-to-monotonic fields unconditionally
- move vdso_start, vdso_end declarations to vdso.h
- correctly build and run when CONFIG_ARM_ARCH_TIMER=n
- rearrange linker script to avoid overlapping sections when CONFIG_DEBUGINFO=n
- remove use_syscall checks from coarse clock paths
- crib BUG_INSTR (0xe7f001f2) from asm/bug.h for text fill

Nathan Lynch (9):
clocksource: arm_arch_timer: change clocksource name if CP15
unavailable
clocksource: arm_arch_timer: enable counter access for 32-bit ARM
ARM: arch_timer: remove unused functions
arm64: arch_timer: remove unused functions
ARM: place sigpage at a random offset above stack
ARM: miscellaneous vdso infrastructure, preparation
ARM: add vdso user-space code
ARM: vdso initialization, mapping, and synchronization
ARM: add CONFIG_VDSO Kconfig and Makefile bits

 arch/arm/include/asm/Kbuild          |   1 -
 arch/arm/include/asm/arch_timer.h    |  25 ---
 arch/arm/include/asm/elf.h           |  11 ++
 arch/arm/include/asm/mmu.h           |   3 +
 arch/arm/include/asm/vdso.h          |  47 +++++
 arch/arm/include/asm/vdso_datapage.h |  60 +++++++
 arch/arm/include/uapi/asm/Kbuild     |   1 +
 arch/arm/include/uapi/asm/auxvec.h   |   7 +
 arch/arm/kernel/Makefile             |   1 +
 arch/arm/kernel/asm-offsets.c        |   5 +
 arch/arm/kernel/process.c            |  56 +++++-
 arch/arm/kernel/vdso.c               | 168 ++++++++++++++++++
 arch/arm/kernel/vdso/.gitignore      |   1 +
 arch/arm/kernel/vdso/Makefile        |  59 +++++++
 arch/arm/kernel/vdso/checkundef.sh   |   9 +
 arch/arm/kernel/vdso/datapage.S      |  15 ++
 arch/arm/kernel/vdso/vdso.S          |  35 ++++
 arch/arm/kernel/vdso/vdso.lds.S      |  88 ++++++++++
 arch/arm/kernel/vdso/vdsomunge.c     | 193 +++++++++++++++++++++
 arch/arm/kernel/vdso/vgettimeofday.c | 320 +++++++++++++++++++++++++++++++++++
 arch/arm/mm/Kconfig                  |  15 ++
 arch/arm64/include/asm/arch_timer.h  |  31 ----
 drivers/clocksource/arm_arch_timer.c |  48 +++++-
 23 files changed, 1134 insertions(+), 65 deletions(-)
 create mode 100644 arch/arm/include/asm/vdso.h
 create mode 100644 arch/arm/include/asm/vdso_datapage.h
 create mode 100644 arch/arm/include/uapi/asm/auxvec.h
 create mode 100644 arch/arm/kernel/vdso.c
 create mode 100644 arch/arm/kernel/vdso/.gitignore
 create mode 100644 arch/arm/kernel/vdso/Makefile
 create mode 100755 arch/arm/kernel/vdso/checkundef.sh
 create mode 100644 arch/arm/kernel/vdso/datapage.S
 create mode 100644 arch/arm/kernel/vdso/vdso.S
 create mode 100644 arch/arm/kernel/vdso/vdso.lds.S
 create mode 100644 arch/arm/kernel/vdso/vdsomunge.c
 create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

--
1.9.3

Quoted 3 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7.

Tested-by Ard Biesheuvel <[email protected]>

On Exynos-5250 (Cortex-A15):

clock-gettime-monotonic system calls per second: 1759517
clock-gettime-monotonic vdso calls per second: 6027915 (3.43x speedup)
clock-getres-monotonic system calls per second: 2144055
clock-getres-monotonic vdso calls per second: 82783103 (38.61x speedup)
clock-gettime-monotonic-coarse system calls per second: 1971433
clock-gettime-monotonic-coarse vdso calls per second: 10710734 (5.43x speedup)
clock-getres-monotonic-coarse system calls per second: 2182380
clock-getres-monotonic-coarse vdso calls per second: 76710594 (35.15x speedup)
clock-gettime-realtime system calls per second: 1722524
clock-gettime-realtime vdso calls per second: 6309212 (3.66x speedup)
clock-getres-realtime system calls per second: 2182357
clock-getres-realtime vdso calls per second: 83120713 (38.09x speedup)
clock-gettime-realtime-coarse system calls per second: 2085498
clock-gettime-realtime-coarse vdso calls per second: 11326069 (5.43x speedup)
clock-getres-realtime-coarse system calls per second: 2178266

clock-getres-realtime-coarse vdso calls per second: 76731175 (35.23x speedup)
Note: vDSO version of getcpu not found getcpu system calls per second: 2645300
getcpu vdso calls per second: 2612154 (0.99x speedup)
Note: vDSO version of getcpu not found
Note: vDSO version of getcpu not found gettimeofday system calls per second: 1707745
gettimeofday vdso calls per second: 6474501 (3.79x speedup)

Quoted 4 lines · Ard Biesheuvel

Quoted 3 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7.

...

Quoted 19 lines

Tested-by Ard Biesheuvel <[email protected]>

On Exynos-5250 (Cortex-A15):

clock-gettime-monotonic system calls per second: 1759517
clock-gettime-monotonic vdso calls per second: 6027915 (3.43x speedup)
clock-getres-monotonic system calls per second: 2144055
clock-getres-monotonic vdso calls per second: 82783103 (38.61x speedup)
clock-gettime-monotonic-coarse system calls per second: 1971433
clock-gettime-monotonic-coarse vdso calls per second: 10710734 (5.43x speedup)
clock-getres-monotonic-coarse system calls per second: 2182380
clock-getres-monotonic-coarse vdso calls per second: 76710594 (35.15x speedup)
clock-gettime-realtime system calls per second: 1722524
clock-gettime-realtime vdso calls per second: 6309212 (3.66x speedup)
clock-getres-realtime system calls per second: 2182357
clock-getres-realtime vdso calls per second: 83120713 (38.09x speedup)
clock-gettime-realtime-coarse system calls per second: 2085498
clock-gettime-realtime-coarse vdso calls per second: 11326069 (5.43x speedup)
clock-getres-realtime-coarse system calls per second: 2178266

Quoted 8 lines

clock-getres-realtime-coarse vdso calls per second: 76731175 (35.23x speedup)
Note: vDSO version of getcpu not found getcpu system calls per second: 2645300
getcpu vdso calls per second: 2612154 (0.99x speedup)
Note: vDSO version of getcpu not found
Note: vDSO version of getcpu not found gettimeofday system calls per second: 1707745
gettimeofday vdso calls per second: 6474501 (3.79x speedup)

Thank you very much for testing (especially on a platform I don't have).

These results match up with what I get on OMAP5 (also Cortex-A15).

Since it looks like I will need to make some significant changes for the next version, I think I can't in good conscience carry your Tested-by on to v8 (although I'm open to different opinions). But I still very much appreciate it.

We (at Linaro) had independently identified VDSO for ARM as a topic of interest after seeing gettimeofday() turn up in a couple of benchmarks (which btw seem to be the primary users of gettimeofday() at such a high rate), so of course, when someone else steps up and does the work, i am happy to at least spend some time testing it. Therefore, I would be happy to test your new version either before or after you (re)post it, whichever you prefer.

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Hi Nathan,

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

[...]

This series is starting to gain traction now, so it's probably time to start thinking about implementing this for CONFIG_COMPAT on arm64. We currently map a fake vectors page there, but this would be much more flexible, particularly as its written in C (and so we would magically get any new functions added to the 32-bit version).

So, there are a couple of questions that spring to mind:

- How difficult is it to share the vdso code with arm64 to build a compat vdso? (getting the Makefile to work will be hard, as we'll need an arm toolchain...).

- If we use your code to build an arm64 vdso, how much slower is it than the handcrafted asm we currently have?

Quoted 19 lines · Will Deacon

Hi Nathan,

Quoted 5 lines · Nathan Lynch

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

[...]

This series is starting to gain traction now, so it's probably time to start thinking about implementing this for CONFIG_COMPAT on arm64. We currently map a fake vectors page there, but this would be much more flexible, particularly as its written in C (and so we would magically get any new functions added to the 32-bit version).

So, there are a couple of questions that spring to mind:

Quoted 3 lines

- How difficult is it to share the vdso code with arm64 to build a compat vdso? (getting the Makefile to work will be hard, as we'll need an arm toolchain...).

Not sure. x86 might be the best example to work from here, although it has the advantage of everything living under arch/x86. 64-bit powerpc kernels provide a 32-bit VDSO for compat tasks but I think it doesn't really share code with the 64-bit VDSO.

Quoted 4 lines

- If we use your code to build an arm64 vdso, how much slower is it than the handcrafted asm we currently have?

Someone with access to 64-bit hardware (i.e. not me) would have to find this out, but I would not be surprised if the C code could be tightened up -- I have not spent much time on optimizing it.

Quoted 8 lines · Nathan Lynch

Quoted 2 lines · Will Deacon

- If we use your code to build an arm64 vdso, how much slower is it than the handcrafted asm we currently have?

Someone with access to 64-bit hardware (i.e. not me) would have to find this out, but I would not be surprised if the C code could be tightened up -- I have not spent much time on optimizing it.

I can't comment on ARM specifically, but the only assembly in the x86 vclock_gettime is an asm volatile (""), which causes gcc to emit more optimal code. I doubt I could shave off more than a cycle or two by rewriting it in assembly.