ARM: vDSO gettimeofday using generic timer architecture

Provide fast userspace implementations of gettimeofday and clock_gettime on systems that implement the generic timers extension defined in ARMv7. This follows the example of arm64 in conception but significantly differs in some aspects of the implementation (C vs assembly, mainly).

Clocks supported:
- CLOCK_REALTIME
- CLOCK_MONOTONIC
- CLOCK_REALTIME_COARSE
- CLOCK_MONOTONIC_COARSE

This also provides clock_getres (as arm64 does).

Note that while the high-precision realtime and monotonic clock support depends on the generic timers extension, support for clock_getres and coarse clocks is independent of the timer implementation and is provided unconditionally.

Run-time tested on OMAP5 and i.MX6 using a patched glibc[1], verifying that results from the vDSO are consistent with results from the kernel. Build-tested against arch/arm/configs/*_defconfig with GCC 4.4 and 4.8. Build-tested with multi_v7_defconfig with GCC 4.[4-8].

[1] RFC glibc patch here:
https://www.sourceware.org/ml/libc-alpha/2014-02/msg00680.html

---

Changes since v2:
- Update to 3.14-rc4.
- Make vDSO configurable, depending on AEABI and MMU.
- Defer shifting of nanosecond component of timespec: fixes observed 1ns inconsistencies for CLOCK_REALTIME, CLOCK_MONOTONIC (see 45a7905fc48f for arm64 equivalent).
- Force reload of seq_count when spinning: without a memory clobber after the load of vdata->seq_count, GCC can generate code like this:

    2f8:   e59c9020        ldr     r9, [ip, #32]
    2fc:   e3190001        tst     r9, #1
    300:   1a000033        bne     3d4 <do_realtime+0x104>
    304:   f57ff05b        dmb     ish
    308:   e59c3034        ldr     r3, [ip, #52]   ; 0x34
    ...
    3d4:   eafffffe        b       3d4 <do_realtime+0x104>

- Build vdso.so with -lgcc: calls to __lshrdi3, __divsi3 sometimes emitted (especially with -Os). Override certain libgcc functions to prevent undefined symbols.
- Do not clear PG_reserved on vdso pages.
- Remove unnecessary get_page calls.
- Simplify ELF signature check during init.
- Use volatile for asm syscall fallbacks.
- Check whether vdso_pagelist is initialized in arm_install_vdso.
- Record clocksource mask in data page.
- Reduce code duplication in do_realtime, do_monotonic.
- Reduce calculations performed in critical sections.
- Simplify coarse clock handling.
- Move datapage load to its own assembly routine.
- Tune vdso_data layout and tweak field names.
- Check vdso shared object for undefined symbols during build.

Changes since v1:
- update to 3.14-rc1
- ensure cache coherency for data page
- Document the kernel-to-userspace protocol for vdso data page updates, and note that the timekeeping core prevents concurrent updates.
- update wall-to-monotonic fields unconditionally
- move vdso_start, vdso_end declarations to vdso.h
- correctly build and run when CONFIG_ARM_ARCH_TIMER=n
- rearrange linker script to avoid overlapping sections when CONFIG_DEBUGINFO=n
- remove use_syscall checks from coarse clock paths
- crib BUG_INSTR (0xe7f001f2) from asm/bug.h for text fill

 arch/arm/include/asm/arch_timer.h    |   7 +-
 arch/arm/include/asm/auxvec.h        |   7 +
 arch/arm/include/asm/elf.h           |   8 +
 arch/arm/include/asm/mmu.h           |   3 +
 arch/arm/include/asm/vdso.h          |  43 +++++
 arch/arm/include/asm/vdso_datapage.h |  53 ++++++
 arch/arm/kernel/Makefile             |   1 +
 arch/arm/kernel/process.c            |  16 +-
 arch/arm/kernel/vdso.c               | 162 +++++++++++++++++
 arch/arm/kernel/vdso/.gitignore      |   1 +
 arch/arm/kernel/vdso/Makefile        |  52 ++++++
 arch/arm/kernel/vdso/checkundef.sh   |   9 +
 arch/arm/kernel/vdso/datapage.S      |  14 ++
 arch/arm/kernel/vdso/vdso.S          |  35 ++++
 arch/arm/kernel/vdso/vdso.lds.S      |  89 ++++++++++
 arch/arm/kernel/vdso/vgettimeofday.c | 330 +++++++++++++++++++++++++++++++++++
 arch/arm/mm/Kconfig                  |  15 ++
 17 files changed, 840 insertions(+), 5 deletions(-)
 create mode 100644 arch/arm/include/asm/auxvec.h
 create mode 100644 arch/arm/include/asm/vdso.h
 create mode 100644 arch/arm/include/asm/vdso_datapage.h
 create mode 100644 arch/arm/kernel/vdso.c
 create mode 100644 arch/arm/kernel/vdso/.gitignore
 create mode 100644 arch/arm/kernel/vdso/Makefile
 create mode 100755 arch/arm/kernel/vdso/checkundef.sh
 create mode 100644 arch/arm/kernel/vdso/datapage.S
 create mode 100644 arch/arm/kernel/vdso/vdso.S
 create mode 100644 arch/arm/kernel/vdso/vdso.lds.S
 create mode 100644 arch/arm/kernel/vdso/vgettimeofday.c

index 0704e0cf5571..047c800b57f0 100644
@@ -103,13 +103,16 @@ static inline void arch_counter_set_user_access(void)
 {
 	u32 cntkctl = arch_timer_get_cntkctl();
 
-	/* Disable user access to both physical/virtual counters/timers */
+	/* Disable user access to the timers and the physical counter */
 	/* Also disable virtual event stream */
 	cntkctl &= ~(ARCH_TIMER_USR_PT_ACCESS_EN
 			| ARCH_TIMER_USR_VT_ACCESS_EN
 			| ARCH_TIMER_VIRT_EVT_EN
-			| ARCH_TIMER_USR_VCT_ACCESS_EN
 			| ARCH_TIMER_USR_PCT_ACCESS_EN);
+
+	/* Enable user access to the virtual counter */
+	cntkctl |= ARCH_TIMER_USR_VCT_ACCESS_EN;
+
 	arch_timer_set_cntkctl(cntkctl);
 }
 

new file mode 100644
index 000000000000..f56936b97ec2
@@ -0,0 +1,7 @@
+#ifndef __ASM_AUXVEC_H
+#define __ASM_AUXVEC_H
+
+/* vDSO location */
+#define AT_SYSINFO_EHDR	33
+
+#endif

index f4b46d39b9cf..fc96b78a8102 100644
@@ -1,6 +1,7 @@
 #ifndef __ASMARM_ELF_H
 #define __ASMARM_ELF_H
 
+#include <asm/auxvec.h>
 #include <asm/hwcap.h>
 
 /*

@@ -129,6 +130,13 @@ extern unsigned long arch_randomize_brk(struct mm_struct *mm);
 #define arch_randomize_brk arch_randomize_brk
 
 #ifdef CONFIG_MMU
+#ifdef CONFIG_VDSO
+#define ARCH_DLINFO							\
+do {									\
+	NEW_AUX_ENT(AT_SYSINFO_EHDR,					\
+		    (elf_addr_t)current->mm->context.vdso);		\
+} while (0)
+#endif
 #define ARCH_HAS_SETUP_ADDITIONAL_PAGES 1
 struct linux_binprm;
 int arch_setup_additional_pages(struct linux_binprm *, int);

index 64fd15159b7d..a5b47421059d 100644
@@ -11,6 +11,9 @@ typedef struct {
 #endif
 	unsigned int	vmalloc_seq;
 	unsigned long	sigpage;
+#ifdef CONFIG_VDSO
+	unsigned long	vdso;
+#endif
 } mm_context_t;
 
 #ifdef CONFIG_CPU_HAS_ASID

index a30fc9be9e9e..a02d74c175f8 100644
@@ -83,6 +83,7 @@ obj-$(CONFIG_PERF_EVENTS)	+= perf_regs.o
 obj-$(CONFIG_HW_PERF_EVENTS)	+= perf_event.o perf_event_cpu.o
 AFLAGS_iwmmxt.o			:= -Wa,-mcpu=iwmmxt
 obj-$(CONFIG_ARM_CPU_TOPOLOGY)  += topology.o
+obj-$(CONFIG_VDSO)		+= vdso.o vdso/
 
 ifneq ($(CONFIG_ARCH_EBSA110),y)
   obj-y		+= io.o

index 92f7b15dd221..6ebd87d1c4cc 100644
@@ -41,6 +41,7 @@
 #include <asm/stacktrace.h>
 #include <asm/mach/time.h>
 #include <asm/tls.h>
+#include <asm/vdso.h>
 
 #ifdef CONFIG_CC_STACKPROTECTOR
 #include <linux/stackprotector.h>

@@ -472,9 +473,16 @@ int in_gate_area_no_mm(unsigned long addr)
 
 const char *arch_vma_name(struct vm_area_struct *vma)
 {
-	return is_gate_vma(vma) ? "[vectors]" :
-		(vma->vm_mm && vma->vm_start == vma->vm_mm->context.sigpage) ?
-		 "[sigpage]" : NULL;
+	if (is_gate_vma(vma))
+		return "[vectors]";
+
+	if (vma->vm_mm && vma->vm_start == vma->vm_mm->context.sigpage)
+		return "[sigpage]";
+
+	if (vma_is_vdso(vma))
+		return "[vdso]";
+
+	return NULL;
 }
 
 static struct page *signal_page;

@@ -505,6 +513,8 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
 	if (ret == 0)
 		mm->context.sigpage = addr;
 
+	arm_install_vdso(mm);
+
  up_fail:
 	up_write(&mm->mmap_sem);
 	return ret;

new file mode 100644
index 000000000000..f8b69d84238e
@@ -0,0 +1 @@
+vdso.lds

new file mode 100755
index 000000000000..185c30da202b
@@ -0,0 +1,9 @@
+#!/bin/sh
+nm="$1"
+file="$2"
+"$nm" -u "$file" | ( ret=0; while read discard symbol
+do
+    echo "$file: undefined symbol $symbol"
+    ret=1
+done ; exit $ret )
+exit $?

new file mode 100644
index 000000000000..91b19b815888
@@ -0,0 +1,14 @@
+#include <linux/linkage.h>
+
+	.align 2
+.L_vdso_data_ptr:
+	.long	_vdso_data - .
+
+ENTRY(__get_datapage)
+	.cfi_startproc
+	adr	r0, .L_vdso_data_ptr
+	ldr	r1, [r0]
+	add	r0, r0, r1
+	bx	lr
+	.cfi_endproc
+ENDPROC(__get_datapage)

new file mode 100644
index 000000000000..aed16ff84c5f
@@ -0,0 +1,35 @@
+/*
+ * Adapted from arm64 version.
+ *
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <[email protected]>
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+#include <linux/const.h>
+#include <asm/page.h>
+
+	__PAGE_ALIGNED_DATA
+
+	.globl vdso_start, vdso_end
+	.balign PAGE_SIZE
+vdso_start:
+	.incbin "arch/arm/kernel/vdso/vdso.so"
+	.balign PAGE_SIZE
+vdso_end:
+
+	.previous

index 1f8fed94c2a4..84898cf4b030 100644
@@ -825,6 +825,21 @@ config KUSER_HELPERS
 	  Say N here only if you are absolutely certain that you do not
 	  need these helpers; otherwise, the safe option is to say Y.
 
+config VDSO
+	bool "Enable vDSO for acceleration of some system calls"
+	depends on AEABI && MMU
+	default y if ARM_ARCH_TIMER
+	select GENERIC_TIME_VSYSCALL
+	help
+	  Place in the process address space an ELF shared object
+	  providing fast implementations of several system calls,
+	  including gettimeofday and clock_gettime.  Systems that
+	  implement the ARM architected timer will receive maximum
+	  benefit.
+
+	  You must have glibc 2.20 or later for programs to seamlessly
+	  take advantage of this.
+
 config DMA_CACHE_RWFO
 	bool "Enable read/write for ownership DMA cache maintenance"
 	depends on CPU_V6K && SMP

1.8.3.1

Hi Nathan,

Have you thought about using a seqlock instead? It looks like a drop-in replacement for your seqcnt_* stuff.

Will

Hi Will,

I have considered that, and x86_64 in fact uses the seqlock APIs for its vDSO. I've made the change locally and it passes my tests fine, but I have one concern -- seqcount_t is defined as:

typedef struct seqcount {
	unsigned sequence;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
	struct lockdep_map dep_map;

#endif
} seqcount_t;

If we use a seqcount_t in the vdso_data structure and CONFIG_DEBUG_LOCK_ALLOC=y, not only does this bloat the structure to no purpose (we can't use the lockdep-enabled read accessors in userspace), it also leaks kernel pointer values and such through the lockdep_map member. So I think the seqcount would have to be a plain integer, and we'd have to cast to seqcount_t to manipulate it, e.g.

seq = raw_read_seqcount_begin((seqcount_t *)&vdata->seq);

See the incremental patch (damaged by my mailer, sorry) below for how this would look.

b/arch/arm/include/asm/vdso_datapage.h
index 9011ec75a24b..83a752f969da 100644

@@ -27,7 +27,7 @@
  * 32 bytes.
  */
 struct vdso_data {
-	u32 seq_count;		/* sequence count - odd during updates */
+	unsigned long seq;      /* sequence counter */
 	u16 use_syscall;	/* whether to fall back to syscalls */
 	u16 cs_shift;		/* clocksource shift */
 	u32 xtime_coarse_sec;	/* coarse time */

index 3db74240a766..6ff2893feb45 100644
@@ -19,6 +19,7 @@
 #include <linux/err.h>
 #include <linux/kernel.h>
 #include <linux/mm.h>
+#include <linux/seqlock.h>
 #include <linux/slab.h>
 #include <linux/timekeeper_internal.h>
 #include <linux/vmalloc.h>

@@ -109,18 +110,8 @@ void arm_install_vdso(struct mm_struct *mm)
 /**
  * update_vsyscall - update the vdso data page
  *
- * Increment the sequence counter, making it odd, indicating to
- * userspace that an update is in progress.  Update the fields used
- * for coarse clocks and, if the architected system timer is in use,
- * the fields used for high precision clocks.  Increment the sequence
- * counter again, making it even, indicating to userspace that the
- * update is finished.
- *
- * Userspace is expected to sample seq_count before reading any other
- * fields from the data page.  If seq_count is odd, userspace is
- * expected to wait until it becomes even.  After copying data from
- * the page, userspace must sample seq_count again; if it has changed
- * from its previous value, userspace must retry the whole sequence.
+ * Update the fields used for coarse clocks and, if the architected
+ * system timer is in use, the fields used for high precision clocks.
  *
  * Calls to update_vsyscall are serialized by the timekeeping core.
  */

@@ -130,8 +121,11 @@ void update_vsyscall(struct timekeeper *tk)
 	struct timespec *wtm = &tk->wall_to_monotonic;
 	bool use_syscall = strcmp(tk->clock->name, "arch_sys_counter");
 
-	++vdso_data->seq_count;
-	smp_wmb();
+	BUILD_BUG_ON(offsetof(struct seqcount, sequence) != 0);
+	BUILD_BUG_ON(FIELD_SIZEOF(struct vdso_data, seq) !=
+		     FIELD_SIZEOF(struct seqcount, sequence));
+
+	raw_write_seqcount_begin((seqcount_t *)&vdso_data->seq);
 
 	xtime_coarse = __current_kernel_time();
 	vdso_data->use_syscall			= use_syscall;

@@ -149,8 +143,7 @@ void update_vsyscall(struct timekeeper *tk)
 		vdso_data->cs_mask		= tk->clock->mask;
 	}
 
-	smp_wmb();
-	++vdso_data->seq_count;
+	raw_write_seqcount_end((seqcount_t *)&vdso_data->seq);
 	flush_dcache_page(virt_to_page(vdso_data));
 }
 

b/arch/arm/kernel/vdso/vgettimeofday.c
index d5d12528eed9..e434769817d3 100644

@@ -10,6 +10,7 @@
 
 #include <linux/compiler.h>
 #include <linux/hrtimer.h>
+#include <linux/seqlock.h>
 #include <linux/time.h>
 #include <asm/arch_timer.h>
 #include <asm/barrier.h>

@@ -23,28 +24,6 @@
 
 extern struct vdso_data *__get_datapage(void);
 
-static u32 seqcnt_acquire(struct vdso_data *vdata)
-{
-	u32 seq;
-
-	do {
-		seq = vdata->seq_count;
-
-		/* Force gcc to reload from memory when spinning. */
-		asm volatile("" : : : "memory");
-	} while (seq & 1);
-
-	dmb(ish);
-	return seq;
-}
-
-static u32 seqcnt_read(struct vdso_data *vdata)
-{
-	dmb(ish);
-
-	return ACCESS_ONCE(vdata->seq_count);
-}
-
 static long clock_gettime_fallback(clockid_t _clkid, struct timespec *_ts)
 {
 	register struct timespec *ts asm("r1") = _ts;

@@ -63,15 +42,15 @@ static long clock_gettime_fallback(clockid_t _clkid,

struct timespec *_ts)

static int do_realtime_coarse(struct timespec *ts, struct vdso_data *vdata) {
- u32 seq;
+ unsigned long seq;

do {
- seq = seqcnt_acquire(vdata);
+ seq = raw_read_seqcount_begin((seqcount_t *)&vdata->seq);

ts->tv_sec = vdata->xtime_coarse_sec; ts->tv_nsec = vdata->xtime_coarse_nsec;

- } while (seq != seqcnt_read(vdata));
+ } while (read_seqcount_retry((seqcount_t *)&vdata->seq, seq));

return 0;
}
@@ -79,10 +58,10 @@ static int do_realtime_coarse(struct timespec *ts, struct vdso_data *vdata)
static int do_monotonic_coarse(struct timespec *ts, struct vdso_data
*vdata)
{
struct timespec tomono;
- u32 seq;
+ unsigned long seq;

do {
- seq = seqcnt_acquire(vdata);
+ seq = raw_read_seqcount_begin((seqcount_t *)&vdata->seq);

ts->tv_sec = vdata->xtime_coarse_sec; ts->tv_nsec = vdata->xtime_coarse_nsec;
@@ -90,7 +69,7 @@ static int do_monotonic_coarse(struct timespec *ts, struct vdso_data *vdata)
tomono.tv_sec = vdata->wtm_clock_sec; tomono.tv_nsec = vdata->wtm_clock_nsec;

- } while (seq != seqcnt_read(vdata));
+ } while (read_seqcount_retry((seqcount_t *)&vdata->seq, seq));

ts->tv_sec += tomono.tv_sec;
timespec_add_ns(ts, tomono.tv_nsec);
@@ -119,10 +98,10 @@ static u64 get_ns(struct vdso_data *vdata)
static int do_realtime(struct timespec *ts, struct vdso_data *vdata) {
u64 nsecs;
- u32 seq;
+ unsigned long seq;

do {
- seq = seqcnt_acquire(vdata);
+ seq = raw_read_seqcount_begin((seqcount_t *)&vdata->seq);

if (vdata->use_syscall)
return -1;
@@ -130,7 +109,7 @@ static int do_realtime(struct timespec *ts, struct vdso_data *vdata)
ts->tv_sec = vdata->xtime_clock_sec;
nsecs = get_ns(vdata);

- } while (seq != seqcnt_read(vdata));
+ } while (read_seqcount_retry((seqcount_t *)&vdata->seq, seq));

ts->tv_nsec = 0;
timespec_add_ns(ts, nsecs);
@@ -142,10 +121,10 @@ static int do_monotonic(struct timespec *ts, struct vdso_data *vdata)
{
struct timespec tomono;
u64 nsecs;
- u32 seq;
+ unsigned long seq;

do {
- seq = seqcnt_acquire(vdata);
+ seq = raw_read_seqcount_begin((seqcount_t *)&vdata->seq);

if (vdata->use_syscall)
return -1;
@@ -156,7 +135,7 @@ static int do_monotonic(struct timespec *ts, struct vdso_data *vdata)
tomono.tv_sec = vdata->wtm_clock_sec; tomono.tv_nsec = vdata->wtm_clock_nsec;

- } while (seq != seqcnt_read(vdata));
+ } while (read_seqcount_retry((seqcount_t *)&vdata->seq, seq));

ts->tv_sec += tomono.tv_sec;
ts->tv_nsec = 0;

Hi Nathan,
Thanks for sending in a V3.
I've given this a test with simple C program on an Arndale board (running A15 with 3.14-rc6 and LPAE) and it worked well. I also short circuited the VDSO to always fallback on syscalls and that worked well too.

So please add:

Also, I have some feedback below.

[snip]

[snip]

Thank you Steve!

Probably. It puzzles me that these __*div0 routines are pulled in when the file has only a division by constant, and I haven't found a way to suppress this. But if we're stuck with it, some kind of trap is likely better, I agree.

Falling back to syscall when conditions require it (e.g. a clock id the vDSO doesn't handle, or clocksource != generic timer) is essential to correct operation of the vDSO. So while I don't necessarily disagree with making CONFIG_VDSO default to y unconditionally, I don't follow your reasoning :-)

My reasoning for 'default y if ARM_ARCH_TIMER' is that the vDSO without the generic timer imposes a small overhead without much benefit. The _COARSE clock ids are handled regardless of clocksource, but I have the impression that users of those are rare.

Anyway, I don't have a strongly-held position on the default setting for CONFIG_VDSO. Just thought I would explain how I arrived at what I wrote.