x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
Subsystems (5)
v2 cover x86/vdso: Cleanups, simmplifications and CLOCK_TAI support
Matt attempted to add CLOCK_TAI support to the VDSO clock_gettime() implementation, which extended the clockid switch case and added yet another slightly different copy of the same code.
Especially the extended switch case is problematic as the compiler tends to generate a jump table which then requires to use retpolines. If jump tables are disabled it adds yet another conditional to the existing maze.
This series takes a different approach by consolidating the almost identical functions into one implementation for high resolution clocks and one for the coarse grained clock ids by storing the base data for each clock id in an array which is indexed by the clock id.
This completely eliminates the switch case and allows further simplifications of the code base, which at the end all together gain a few cycles performance or at least stay on par with todays code. The resulting performance depends heavily on the micro architecture and the compiler.
Changes vs. V1:
- Fix the VCLOCK_MAX sanity check
- Remove the magic clock masking and extend the storage array
Thanks,
tglx
8<-------------------
arch/x86/Kconfig | 1 arch/x86/entry/vdso/vclock_gettime.c | 199 ++++++++------------------------ arch/x86/entry/vsyscall/vsyscall_gtod.c | 55 ++++---- arch/x86/include/asm/vgtod.h | 42 +++--- arch/x86/kernel/time.c | 22 +++ include/linux/clocksource.h | 5 kernel/time/Kconfig | 4 kernel/time/clocksource.c | 2 8 files changed, 140 insertions(+), 190 deletions(-)
PATCH v2 1/11 clocksource: Provide clocksource_arch_init()
Architectures have extra archdata in the clocksource, e.g. for VDSO support. There are no sanity checks or general initializations for this available. Add support for that.
--- include/linux/clocksource.h | 5 +++++ kernel/time/Kconfig | 4 ++++ kernel/time/clocksource.c | 2 ++ 3 files changed, 11 insertions(+)
include/linux/clocksource.h +5 -0
@@ -241,6 +241,11 @@ static inline void __clocksource_update___clocksource_update_freq_scale (cs,1000 , khz); } +#ifdef CONFIG_ARCH_CLOCKSOURCE_INIT +extern void clocksource_arch_init (struct clocksource *cs); +#else +static inline void clocksource_arch_init (struct clocksource *cs) { } +#endif extern int timekeeping_notify (struct clocksource *clock);
kernel/time/Kconfig +4 -0
@@ -12,6 +12,10 @@ config CLOCKSOURCE_WATCHDOG config ARCH_CLOCKSOURCE_DATA bool +# Architecture has extra clocksource init called from registration +config ARCH_CLOCKSOURCE_INIT + bool + # Clocksources require validation of the clocksource against the last # cycle update - x86/TSC misfeature config CLOCKSOURCE_VALIDATE_LAST_CYCLE
kernel/time/clocksource.c +2 -0
@@ -937,6 +937,8 @@ int __clocksource_register_scale(struct {unsigned long flags; +clocksource_arch_init (cs); +/* Initialize mult/shift and max_idle_ns */ __clocksource_update_freq_scale (cs, scale, freq);
PATCH v2 1/11 clocksource: Provide clocksource_arch_init()
In reply to Thomas GleixnerArchitectures have extra archdata in the clocksource, e.g. for VDSO support. There are no sanity checks or general initializations for this available. Add support for that.
--- include/linux/clocksource.h | 5 +++++ kernel/time/Kconfig | 4 ++++ kernel/time/clocksource.c | 2 ++ 3 files changed, 11 insertions(+)
include/linux/clocksource.h +5 -0
@@ -241,6 +241,11 @@ static inline void __clocksource_update___clocksource_update_freq_scale (cs,1000 , khz); } +#ifdef CONFIG_ARCH_CLOCKSOURCE_INIT +extern void clocksource_arch_init (struct clocksource *cs); +#else +static inline void clocksource_arch_init (struct clocksource *cs) { } +#endif extern int timekeeping_notify (struct clocksource *clock);
kernel/time/Kconfig +4 -0
@@ -12,6 +12,10 @@ config CLOCKSOURCE_WATCHDOG config ARCH_CLOCKSOURCE_DATA bool +# Architecture has extra clocksource init called from registration +config ARCH_CLOCKSOURCE_INIT + bool + # Clocksources require validation of the clocksource against the last # cycle update - x86/TSC misfeature config CLOCKSOURCE_VALIDATE_LAST_CYCLE
kernel/time/clocksource.c +2 -0
@@ -937,6 +937,8 @@ int __clocksource_register_scale(struct {unsigned long flags; +clocksource_arch_init (cs); +/* Initialize mult/shift and max_idle_ns */ __clocksource_update_freq_scale (cs, scale, freq);
Re: [patch V2 01/11] clocksource: Provide clocksource_arch_init()
Quoted 5 lines · Thomas Gleixner
Architectures have extra archdata in the clocksource, e.g. for VDSO support. There are no sanity checks or general initializations for this available. Add support for that.
Signed-off-by Thomas Gleixner <[email protected]>
Sorry, Let me try reply-all this time. :)
thanks
-john
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] clocksource: Provide clocksource_arch_init()
In reply to Thomas Gleixnerclocksource: Provide clocksource_arch_init()
Commit-ID: d67f34c19a679436dd2963b588015e119279e7a8 Gitweb: https://git.kernel.org/tip/d67f34c19a679436dd2963b588015e119279e7a8 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:34 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:24 +0200
clocksource: Provide clocksource_arch_init()
Architectures have extra archdata in the clocksource, e.g. for VDSO support. There are no sanity checks or general initializations for this available. Add support for that.
--- include/linux/clocksource.h | 5 +++++ kernel/time/Kconfig | 4 ++++ kernel/time/clocksource.c | 2 ++ 3 files changed, 11 insertions(+)
include/linux/clocksource.h +5 -0
@@ -241,6 +241,11 @@ static inline void __clocksource_update_freq_khz(struct clocksource *cs, u32 khz__clocksource_update_freq_scale (cs,1000 , khz); } +#ifdef CONFIG_ARCH_CLOCKSOURCE_INIT +extern void clocksource_arch_init (struct clocksource *cs); +#else +static inline void clocksource_arch_init (struct clocksource *cs) { } +#endif extern int timekeeping_notify (struct clocksource *clock);
kernel/time/Kconfig +4 -0
@@ -12,6 +12,10 @@ config CLOCKSOURCE_WATCHDOG config ARCH_CLOCKSOURCE_DATA bool +# Architecture has extra clocksource init called from registration +config ARCH_CLOCKSOURCE_INIT + bool + # Clocksources require validation of the clocksource against the last # cycle update - x86/TSC misfeature config CLOCKSOURCE_VALIDATE_LAST_CYCLE
kernel/time/clocksource.c +2 -0
@@ -937,6 +937,8 @@ int __clocksource_register_scale(struct clocksource *cs, u32 scale, u32 freq) {unsigned long flags; +clocksource_arch_init (cs); +/* Initialize mult/shift and max_idle_ns */ __clocksource_update_freq_scale (cs, scale, freq);
PATCH v2 2/11 x86/time: Implement clocksource_arch_init()
In reply to Thomas GleixnerRuntime validate the VCLOCK_MODE in clocksource::archdata and disable VCLOCK if invalid, which disables the VDSO but keeps the system running.
---
V2: Fix the VCLOCK_MAX check
arch/x86/Kconfig | 1 + arch/x86/kernel/time.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
arch/x86/Kconfig +1 -0
@@ -48,6 +48,7 @@ config X86 select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ANON_INODES select ARCH_CLOCKSOURCE_DATA + select ARCH_CLOCKSOURCE_INIT select ARCH_DISCARD_MEMBLOCK select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL
arch/x86/kernel/time.c +16 -0
@@ -10,6 +10,7 @@* */ +#include <linux/clocksource.h> #include <linux/clockchips.h> #include <linux/interrupt.h> #include <linux/irq.h>
@@ -105,3 +106,18 @@ void __init time_init(void) { late_time_init = x86_late_time_init; } + +/* +* Sanity check the vdso related archdata content. +*/ +void clocksource_arch_init (struct clocksource *cs) +{ +if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +return ; + +if (cs->archdata.vclock_mode >VCLOCK_MAX ) { +pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , + cs->name, cs->archdata.vclock_mode); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } +}
PATCH v2 2/11 x86/time: Implement clocksource_arch_init()
In reply to Thomas GleixnerRuntime validate the VCLOCK_MODE in clocksource::archdata and disable VCLOCK if invalid, which disables the VDSO but keeps the system running.
---
V2: Fix the VCLOCK_MAX check
arch/x86/Kconfig | 1 + arch/x86/kernel/time.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
arch/x86/Kconfig +1 -0
@@ -48,6 +48,7 @@ config X86 select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ANON_INODES select ARCH_CLOCKSOURCE_DATA + select ARCH_CLOCKSOURCE_INIT select ARCH_DISCARD_MEMBLOCK select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL
arch/x86/kernel/time.c +16 -0
@@ -10,6 +10,7 @@* */ +#include <linux/clocksource.h> #include <linux/clockchips.h> #include <linux/interrupt.h> #include <linux/irq.h>
@@ -105,3 +106,18 @@ void __init time_init(void) { late_time_init = x86_late_time_init; } + +/* +* Sanity check the vdso related archdata content. +*/ +void clocksource_arch_init (struct clocksource *cs) +{ +if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +return ; + +if (cs->archdata.vclock_mode >VCLOCK_MAX ) { +pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , + cs->name, cs->archdata.vclock_mode); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } +}
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/time: Implement clocksource_arch_init()
x86/time: Implement clocksource_arch_init()
Commit-ID: 2a21ad571b62440a4ad66d70b4b5d09c00029af4 Gitweb: https://git.kernel.org/tip/2a21ad571b62440a4ad66d70b4b5d09c00029af4 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:35 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:24 +0200
x86/time: Implement clocksource_arch_init()
Runtime validate the VCLOCK_MODE in clocksource::archdata and disable VCLOCK if invalid, which disables the VDSO but keeps the system running.
--- arch/x86/Kconfig | 1 + arch/x86/kernel/time.c | 16 ++++++++++++++++ 2 files changed, 17 insertions(+)
arch/x86/Kconfig +1 -0
@@ -48,6 +48,7 @@ config X86 select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ANON_INODES select ARCH_CLOCKSOURCE_DATA + select ARCH_CLOCKSOURCE_INIT select ARCH_DISCARD_MEMBLOCK select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL
arch/x86/kernel/time.c +16 -0
@@ -10,6 +10,7 @@* */ +#include <linux/clocksource.h> #include <linux/clockchips.h> #include <linux/interrupt.h> #include <linux/irq.h>
@@ -105,3 +106,18 @@ void __init time_init(void) { late_time_init = x86_late_time_init; } + +/* +* Sanity check the vdso related archdata content. +*/ +void clocksource_arch_init (struct clocksource *cs) +{ +if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +return ; + +if (cs->archdata.vclock_mode >VCLOCK_MAX ) { +pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , + cs->name, cs->archdata.vclock_mode); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } +}
PATCH v2 3/11 x86/vdso: Enforce 64bit clocksource
In reply to Thomas GleixnerAll VDSO clock sources are TSC based and use CLOCKSOURCE_MASK(64). There is no point in masking with all FF. Get rid of it and enforce the mask in the sanity checker.
--- arch/x86/entry/vdso/vclock_gettime.c | 2 +- arch/x86/kernel/time.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vclock_gettime.c +1 -1
@@ -199,7 +199,7 @@ notrace static inline u64 vgetsns(int *m#endif else return 0; - v = (cycles - gtod->cycle_last) & gtod->mask; + v = cycles - gtod->cycle_last;return v * gtod->mult; }
arch/x86/kernel/time.c +6 -0
@@ -120,4 +120,10 @@ void clocksource_arch_init(struct clocks cs->name, cs->archdata.vclock_mode); cs->archdata.vclock_mode =VCLOCK_NONE ; } + +if (cs->mask !=CLOCKSOURCE_MASK (64 )) { +pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , + cs->name, cs->mask); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } }
PATCH v2 3/11 x86/vdso: Enforce 64bit clocksource
In reply to Thomas GleixnerAll VDSO clock sources are TSC based and use CLOCKSOURCE_MASK(64). There is no point in masking with all FF. Get rid of it and enforce the mask in the sanity checker.
--- arch/x86/entry/vdso/vclock_gettime.c | 2 +- arch/x86/kernel/time.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vclock_gettime.c +1 -1
@@ -199,7 +199,7 @@ notrace static inline u64 vgetsns(int *m#endif else return 0; - v = (cycles - gtod->cycle_last) & gtod->mask; + v = cycles - gtod->cycle_last;return v * gtod->mult; }
arch/x86/kernel/time.c +6 -0
@@ -120,4 +120,10 @@ void clocksource_arch_init(struct clocks cs->name, cs->archdata.vclock_mode); cs->archdata.vclock_mode =VCLOCK_NONE ; } + +if (cs->mask !=CLOCKSOURCE_MASK (64 )) { +pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , + cs->name, cs->mask); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } }
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Enforce 64bit clocksource
x86/vdso: Enforce 64bit clocksource
Commit-ID: a51e996d48ac9fa0a1260a3822a14f3d570d3be7 Gitweb: https://git.kernel.org/tip/a51e996d48ac9fa0a1260a3822a14f3d570d3be7 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:36 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:25 +0200
x86/vdso: Enforce 64bit clocksource
All VDSO clock sources are TSC based and use CLOCKSOURCE_MASK(64). There is no point in masking with all FF. Get rid of it and enforce the mask in the sanity checker.
--- arch/x86/entry/vdso/vclock_gettime.c | 2 +- arch/x86/kernel/time.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/entry/vdso/vclock_gettime.c +1 -1
@@ -201,7 +201,7 @@ notrace static inline u64 vgetsns(int *mode)#endif else return 0; - v = (cycles - gtod->cycle_last) & gtod->mask; + v = cycles - gtod->cycle_last;return v * gtod->mult; }
arch/x86/kernel/time.c +6 -0
@@ -120,4 +120,10 @@ void clocksource_arch_init(struct clocksource *cs) cs->name, cs->archdata.vclock_mode); cs->archdata.vclock_mode =VCLOCK_NONE ; } + +if (cs->mask !=CLOCKSOURCE_MASK (64 )) { +pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , + cs->name, cs->mask); + cs->archdata.vclock_mode =VCLOCK_NONE ; + } }
PATCH v2 4/11 x86/vdso: Use unsigned int consistently for vsyscall_gtod_data::seq
In reply to Thomas GleixnerThe sequence count in vgtod_data is unsigned int, but the call sites use unsigned long, which is a pointless exercise. Fix the call sites and replace 'unsigned' with unsinged 'int' while at it.
--- arch/x86/entry/vdso/vclock_gettime.c | 8 ++++---- arch/x86/include/asm/vgtod.h | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -4
@@ -206,7 +206,7 @@ notrace static inline u64 vgetsns(int *m/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -227,7 +227,7 @@ notrace static int __always_inline do_re notrace static int __always_inline do_monotonic(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -248,7 +248,7 @@ notrace static int __always_inline do_mo notrace static void do_realtime_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->wall_time_coarse_sec;
@@ -258,7 +258,7 @@ notrace static void do_realtime_coarse(s notrace static void do_monotonic_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->monotonic_time_coarse_sec;
arch/x86/include/asm/vgtod.h +5 -5
@@ -15,9 +15,9 @@ typedef unsigned long gtod_long_t;* so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned seq; +unsigned int seq; -int vclock_mode; +int vclock_mode;u64 cycle_last;u64 mask;u32 mult;
@@ -44,9 +44,9 @@ static inline bool vclock_was_used(int vreturn READ_ONCE (vclocks_used) & (1 << vclock); } -static inline unsigned gtod_read_begin (const struct vsyscall_gtod_data *s) +static inline unsigned int gtod_read_begin (const struct vsyscall_gtod_data *s) { -unsigned ret; +unsigned int ret; repeat: ret =READ_ONCE (s->seq);
@@ -59,7 +59,7 @@ static inline unsigned gtod_read_begin(c }static inline int gtod_read_retry (const struct vsyscall_gtod_data *s, -unsigned start) +unsigned int start) {smp_rmb ();return unlikely (s->seq != start);
PATCH v2 4/11 x86/vdso: Use unsigned int consistently for vsyscall_gtod_data::seq
In reply to Thomas GleixnerThe sequence count in vgtod_data is unsigned int, but the call sites use unsigned long, which is a pointless exercise. Fix the call sites and replace 'unsigned' with unsinged 'int' while at it.
--- arch/x86/entry/vdso/vclock_gettime.c | 8 ++++---- arch/x86/include/asm/vgtod.h | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -4
@@ -206,7 +206,7 @@ notrace static inline u64 vgetsns(int *m/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -227,7 +227,7 @@ notrace static int __always_inline do_re notrace static int __always_inline do_monotonic(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -248,7 +248,7 @@ notrace static int __always_inline do_mo notrace static void do_realtime_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->wall_time_coarse_sec;
@@ -258,7 +258,7 @@ notrace static void do_realtime_coarse(s notrace static void do_monotonic_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->monotonic_time_coarse_sec;
arch/x86/include/asm/vgtod.h +5 -5
@@ -15,9 +15,9 @@ typedef unsigned long gtod_long_t;* so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned seq; +unsigned int seq; -int vclock_mode; +int vclock_mode;u64 cycle_last;u64 mask;u32 mult;
@@ -44,9 +44,9 @@ static inline bool vclock_was_used(int vreturn READ_ONCE (vclocks_used) & (1 << vclock); } -static inline unsigned gtod_read_begin (const struct vsyscall_gtod_data *s) +static inline unsigned int gtod_read_begin (const struct vsyscall_gtod_data *s) { -unsigned ret; +unsigned int ret; repeat: ret =READ_ONCE (s->seq);
@@ -59,7 +59,7 @@ static inline unsigned gtod_read_begin(c }static inline int gtod_read_retry (const struct vsyscall_gtod_data *s, -unsigned start) +unsigned int start) {smp_rmb ();return unlikely (s->seq != start);
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Use unsigned int consistently for vsyscall_gtod_data:: Seq
x86/vdso: Use unsigned int consistently for vsyscall_gtod_data:: Seq
Commit-ID: 77e9c678c54f2d9214796c1c5bd0c7c7ccedd932 Gitweb: https://git.kernel.org/tip/77e9c678c54f2d9214796c1c5bd0c7c7ccedd932 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:37 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:25 +0200
x86/vdso: Use unsigned int consistently for vsyscall_gtod_data:: Seq
The sequence count in vgtod_data is unsigned int, but the call sites use unsigned long, which is a pointless exercise. Fix the call sites and replace 'unsigned' with unsinged 'int' while at it.
--- arch/x86/entry/vdso/vclock_gettime.c | 8 ++++---- arch/x86/include/asm/vgtod.h | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -4
@@ -208,7 +208,7 @@ notrace static inline u64 vgetsns(int *mode)/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -229,7 +229,7 @@ notrace static int __always_inline do_realtime(struct timespec *ts) notrace static int __always_inline do_monotonic(struct timespec *ts) { -unsigned long seq; +unsigned int seq;u64 ns;int mode;
@@ -250,7 +250,7 @@ notrace static int __always_inline do_monotonic(struct timespec *ts) notrace static void do_realtime_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->wall_time_coarse_sec;
@@ -260,7 +260,7 @@ notrace static void do_realtime_coarse(struct timespec *ts) notrace static void do_monotonic_coarse(struct timespec *ts) { -unsigned long seq; +unsigned int seq; do { seq = gtod_read_begin(gtod); ts->tv_sec = gtod->monotonic_time_coarse_sec;
arch/x86/include/asm/vgtod.h +5 -5
@@ -15,9 +15,9 @@ typedef unsigned long gtod_long_t;* so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned seq; +unsigned int seq; -int vclock_mode; +int vclock_mode;u64 cycle_last;u64 mask;u32 mult;
@@ -44,9 +44,9 @@ static inline bool vclock_was_used(int vclock)return READ_ONCE (vclocks_used) & (1 << vclock); } -static inline unsigned gtod_read_begin (const struct vsyscall_gtod_data *s) +static inline unsigned int gtod_read_begin (const struct vsyscall_gtod_data *s) { -unsigned ret; +unsigned int ret; repeat: ret =READ_ONCE (s->seq);
@@ -59,7 +59,7 @@ repeat: }static inline int gtod_read_retry (const struct vsyscall_gtod_data *s, -unsigned start) +unsigned int start) {smp_rmb ();return unlikely (s->seq != start);
PATCH v2 5/11 x86/vdso: Introduce and use vgtod_ts
In reply to Thomas GleixnerIt's desired to support more clocks in the VDSO, e.g. CLOCK_TAI. This results either in indirect calls due to the larger switch case, which then requires retpolines or when the compiler is forced to avoid jump tables it results in even more conditionals.
To avoid both variants which are bad for performance the high resolution functions and the coarse grained functions will be collapsed into one for each. That requires to store the clock specific base time in an array.
Introcude struct vgtod_ts for storage and convert the data store, the update function and the individual clock functions over to use it.
The new storage does not longer use gtod_long_t for seconds depending on 32 or 64 bit compile because this needs to be the full 64bit value even for 32bit when a Y2038 function is added. No point in keeping the distinction alive in the internal representation.
--- arch/x86/entry/vdso/vclock_gettime.c | 24 +++++++++------ arch/x86/entry/vsyscall/vsyscall_gtod.c | 51 ++++++++++++++++---------------- arch/x86/include/asm/vgtod.h | 36 ++++++++++++---------- 3 files changed, 61 insertions(+), 50 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +15 -9
@@ -206,6 +206,7 @@ notrace static inline u64 vgetsns(int *m/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ];unsigned int seq;u64 ns;int mode;
@@ -213,8 +214,8 @@ notrace static int __always_inline do_redo { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->wall_time_sec; - ns = gtod->wall_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -227,6 +228,7 @@ notrace static int __always_inline do_re notrace static int __always_inline do_monotonic(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ];unsigned int seq;u64 ns;int mode;
@@ -234,8 +236,8 @@ notrace static int __always_inline do_modo { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->monotonic_time_sec; - ns = gtod->monotonic_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -248,21 +250,25 @@ notrace static int __always_inline do_mo notrace static void do_realtime_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->wall_time_coarse_sec; - ts->tv_nsec = gtod->wall_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); } notrace static void do_monotonic_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->monotonic_time_coarse_sec; - ts->tv_nsec = gtod->monotonic_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); }
@@ -318,7 +324,7 @@ int gettimeofday(struct timeval *, struc notrace time_t __vdso_time(time_t *t) {/* This is atomic on x86 so we don't need any locks. */ -time_t result =READ_ONCE (gtod->wall_time_sec); +time_t result =READ_ONCE (gtod->basetime[CLOCK_REALTIME ].sec);if (t) *t = result;
arch/x86/entry/vsyscall/vsyscall_gtod.c +26 -25
66 lines not on this page · raw message
arch/x86/include/asm/vgtod.h +20 -16
@@ -5,33 +5,37 @@#include <linux/compiler.h> #include <linux/clocksource.h> +#include <uapi/linux/time.h> +#ifdef BUILD_VDSO32_64 typedef u64 gtod_long_t ;#else typedef unsigned long gtod_long_t ;#endif + +struct vgtod_ts { +u64 sec; +u64 nsec; +}; + +#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE)) +/* * vsyscall_gtod_data will be accessed by 32 and 64 bit code at the same time * so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned int seq; +unsigned int seq; + +int vclock_mode; +u64 cycle_last; +u64 mask; +u32 mult; +u32 shift; -int vclock_mode; -u64 cycle_last; -u64 mask; -u32 mult; -u32 shift; - -/* open coded 'struct timespec' */ -u64 wall_time_snsec; -gtod_long_t wall_time_sec; -gtod_long_t monotonic_time_sec; -u64 monotonic_time_snsec; -gtod_long_t wall_time_coarse_sec; -gtod_long_t wall_time_coarse_nsec; -gtod_long_t monotonic_time_coarse_sec; -gtod_long_t monotonic_time_coarse_nsec; +struct vgtod_ts basetime[VGTOD_BASES ];int tz_minuteswest;int tz_dsttime;
PATCH v2 5/11 x86/vdso: Introduce and use vgtod_ts
In reply to Thomas GleixnerIt's desired to support more clocks in the VDSO, e.g. CLOCK_TAI. This results either in indirect calls due to the larger switch case, which then requires retpolines or when the compiler is forced to avoid jump tables it results in even more conditionals.
To avoid both variants which are bad for performance the high resolution functions and the coarse grained functions will be collapsed into one for each. That requires to store the clock specific base time in an array.
Introcude struct vgtod_ts for storage and convert the data store, the update function and the individual clock functions over to use it.
The new storage does not longer use gtod_long_t for seconds depending on 32 or 64 bit compile because this needs to be the full 64bit value even for 32bit when a Y2038 function is added. No point in keeping the distinction alive in the internal representation.
--- arch/x86/entry/vdso/vclock_gettime.c | 24 +++++++++------ arch/x86/entry/vsyscall/vsyscall_gtod.c | 51 ++++++++++++++++---------------- arch/x86/include/asm/vgtod.h | 36 ++++++++++++---------- 3 files changed, 61 insertions(+), 50 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +15 -9
@@ -206,6 +206,7 @@ notrace static inline u64 vgetsns(int *m/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ];unsigned int seq;u64 ns;int mode;
@@ -213,8 +214,8 @@ notrace static int __always_inline do_redo { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->wall_time_sec; - ns = gtod->wall_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -227,6 +228,7 @@ notrace static int __always_inline do_re notrace static int __always_inline do_monotonic(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ];unsigned int seq;u64 ns;int mode;
@@ -234,8 +236,8 @@ notrace static int __always_inline do_modo { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->monotonic_time_sec; - ns = gtod->monotonic_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -248,21 +250,25 @@ notrace static int __always_inline do_mo notrace static void do_realtime_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->wall_time_coarse_sec; - ts->tv_nsec = gtod->wall_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); } notrace static void do_monotonic_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->monotonic_time_coarse_sec; - ts->tv_nsec = gtod->monotonic_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); }
@@ -318,7 +324,7 @@ int gettimeofday(struct timeval *, struc notrace time_t __vdso_time(time_t *t) {/* This is atomic on x86 so we don't need any locks. */ -time_t result =READ_ONCE (gtod->wall_time_sec); +time_t result =READ_ONCE (gtod->basetime[CLOCK_REALTIME ].sec);if (t) *t = result;
arch/x86/entry/vsyscall/vsyscall_gtod.c +26 -25
66 lines not on this page · raw message
arch/x86/include/asm/vgtod.h +20 -16
@@ -5,33 +5,37 @@#include <linux/compiler.h> #include <linux/clocksource.h> +#include <uapi/linux/time.h> +#ifdef BUILD_VDSO32_64 typedef u64 gtod_long_t ;#else typedef unsigned long gtod_long_t ;#endif + +struct vgtod_ts { +u64 sec; +u64 nsec; +}; + +#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE)) +/* * vsyscall_gtod_data will be accessed by 32 and 64 bit code at the same time * so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned int seq; +unsigned int seq; + +int vclock_mode; +u64 cycle_last; +u64 mask; +u32 mult; +u32 shift; -int vclock_mode; -u64 cycle_last; -u64 mask; -u32 mult; -u32 shift; - -/* open coded 'struct timespec' */ -u64 wall_time_snsec; -gtod_long_t wall_time_sec; -gtod_long_t monotonic_time_sec; -u64 monotonic_time_snsec; -gtod_long_t wall_time_coarse_sec; -gtod_long_t wall_time_coarse_nsec; -gtod_long_t monotonic_time_coarse_sec; -gtod_long_t monotonic_time_coarse_nsec; +struct vgtod_ts basetime[VGTOD_BASES ];int tz_minuteswest;int tz_dsttime;
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Introduce and use vgtod_ts
x86/vdso: Introduce and use vgtod_ts
Commit-ID: 49116f2081eeaf75316956705d46602b7eb735cc Gitweb: https://git.kernel.org/tip/49116f2081eeaf75316956705d46602b7eb735cc Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:38 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:25 +0200
x86/vdso: Introduce and use vgtod_ts
It's desired to support more clocks in the VDSO, e.g. CLOCK_TAI. This results either in indirect calls due to the larger switch case, which then requires retpolines or when the compiler is forced to avoid jump tables it results in even more conditionals.
To avoid both variants which are bad for performance the high resolution functions and the coarse grained functions will be collapsed into one for each. That requires to store the clock specific base time in an array.
Introcude struct vgtod_ts for storage and convert the data store, the update function and the individual clock functions over to use it.
The new storage does not longer use gtod_long_t for seconds depending on 32 or 64 bit compile because this needs to be the full 64bit value even for 32bit when a Y2038 function is added. No point in keeping the distinction alive in the internal representation.
--- arch/x86/entry/vdso/vclock_gettime.c | 24 ++++++++++------- arch/x86/entry/vsyscall/vsyscall_gtod.c | 47 +++++++++++++++++---------------- arch/x86/include/asm/vgtod.h | 38 ++++++++++++++------------ 3 files changed, 60 insertions(+), 49 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +15 -9
@@ -208,6 +208,7 @@ notrace static inline u64 vgetsns(int *mode)/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ notrace static int __always_inline do_realtime(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ];unsigned int seq;u64 ns;int mode;
@@ -215,8 +216,8 @@ notrace static int __always_inline do_realtime(struct timespec *ts)do { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->wall_time_sec; - ns = gtod->wall_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -229,6 +230,7 @@ notrace static int __always_inline do_realtime(struct timespec *ts) notrace static int __always_inline do_monotonic(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ];unsigned int seq;u64 ns;int mode;
@@ -236,8 +238,8 @@ notrace static int __always_inline do_monotonic(struct timespec *ts)do { seq =gtod_read_begin (gtod); mode = gtod->vclock_mode; - ts->tv_sec = gtod->monotonic_time_sec; - ns = gtod->monotonic_time_snsec; + ts->tv_sec = base->sec; + ns = base->nsec; ns +=vgetsns (&mode); ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
@@ -250,21 +252,25 @@ notrace static int __always_inline do_monotonic(struct timespec *ts) notrace static void do_realtime_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->wall_time_coarse_sec; - ts->tv_nsec = gtod->wall_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); } notrace static void do_monotonic_coarse(struct timespec *ts) { +struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ];unsigned int seq; +do { seq =gtod_read_begin (gtod); - ts->tv_sec = gtod->monotonic_time_coarse_sec; - ts->tv_nsec = gtod->monotonic_time_coarse_nsec; + ts->tv_sec = base->sec; + ts->tv_nsec = base->nsec; }while (unlikely (gtod_read_retry (gtod, seq))); }
@@ -320,7 +326,7 @@ int gettimeofday(struct timeval *, struct timezone *) notrace time_t __vdso_time(time_t *t) {/* This is atomic on x86 so we don't need any locks. */ -time_t result =READ_ONCE (gtod->wall_time_sec); +time_t result =READ_ONCE (gtod->basetime[CLOCK_REALTIME ].sec);if (t) *t = result;
arch/x86/entry/vsyscall/vsyscall_gtod.c +24 -23
64 lines not on this page · raw message
arch/x86/include/asm/vgtod.h +21 -17
@@ -5,33 +5,37 @@#include <linux/compiler.h> #include <linux/clocksource.h> +#include <uapi/linux/time.h> +#ifdef BUILD_VDSO32_64 typedef u64 gtod_long_t ;#else typedef unsigned long gtod_long_t ;#endif + +struct vgtod_ts { +u64 sec; +u64 nsec; +}; + +#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE)) +/* * vsyscall_gtod_data will be accessed by 32 and 64 bit code at the same time * so be carefull by modifying this structure. */ struct vsyscall_gtod_data { -unsigned int seq; - -int vclock_mode; -u64 cycle_last; -u64 mask; -u32 mult; -u32 shift; - -/* open coded 'struct timespec' */ -u64 wall_time_snsec; -gtod_long_t wall_time_sec; -gtod_long_t monotonic_time_sec; -u64 monotonic_time_snsec; -gtod_long_t wall_time_coarse_sec; -gtod_long_t wall_time_coarse_nsec; -gtod_long_t monotonic_time_coarse_sec; -gtod_long_t monotonic_time_coarse_nsec; +unsigned int seq; + +int vclock_mode; +u64 cycle_last; +u64 mask; +u32 mult; +u32 shift; + +struct vgtod_ts basetime[VGTOD_BASES ];int tz_minuteswest;int tz_dsttime;
PATCH v2 6/11 x86/vdso: Collapse high resolution functions
In reply to Thomas Gleixnerdo_realtime() and do_monotonic() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -28
@@ -203,35 +203,12 @@ notrace static inline u64 vgetsns(int *mreturn v * gtod->mult; } -/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ -notrace static int __always_inline do_realtime(struct timespec *ts) +notrace static int do_hres(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -u64 ns;int mode; - -do { - seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; - ts->tv_sec = base->sec; - ns = base->nsec; - ns +=vgetsns (&mode); - ns >>= gtod->shift; - }while (unlikely (gtod_read_retry (gtod, seq))); - - ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); - ts->tv_nsec = ns; - -return mode; -} - -notrace static int __always_inline do_monotonic(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ]; -unsigned int seq;u64 ns; -int mode; do { seq = gtod_read_begin(gtod);
@@ -276,11 +253,11 @@ notrace int __vdso_clock_gettime(clockid {switch (clock) {case CLOCK_REALTIME : -if (do_realtime (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_MONOTONIC : -if (do_monotonic (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_REALTIME_COARSE :
@@ -303,7 +280,9 @@ int clock_gettime(clockid_t, struct time notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) {if (likely (tv !=NULL )) { -if (unlikely (do_realtime ((struct timespec *)tv) ==VCLOCK_NONE )) +struct timespec *ts = (struct timespec *) tv; + +if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ))return vdso_fallback_gtod (tv, tz); tv->tv_usec /=1000 ; }
PATCH v2 6/11 x86/vdso: Collapse high resolution functions
In reply to Thomas Gleixnerdo_realtime() and do_monotonic() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -28
@@ -203,35 +203,12 @@ notrace static inline u64 vgetsns(int *mreturn v * gtod->mult; } -/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ -notrace static int __always_inline do_realtime(struct timespec *ts) +notrace static int do_hres(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -u64 ns;int mode; - -do { - seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; - ts->tv_sec = base->sec; - ns = base->nsec; - ns +=vgetsns (&mode); - ns >>= gtod->shift; - }while (unlikely (gtod_read_retry (gtod, seq))); - - ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); - ts->tv_nsec = ns; - -return mode; -} - -notrace static int __always_inline do_monotonic(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ]; -unsigned int seq;u64 ns; -int mode; do { seq = gtod_read_begin(gtod);
@@ -276,11 +253,11 @@ notrace int __vdso_clock_gettime(clockid {switch (clock) {case CLOCK_REALTIME : -if (do_realtime (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_MONOTONIC : -if (do_monotonic (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_REALTIME_COARSE :
@@ -303,7 +280,9 @@ int clock_gettime(clockid_t, struct time notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) {if (likely (tv !=NULL )) { -if (unlikely (do_realtime ((struct timespec *)tv) ==VCLOCK_NONE )) +struct timespec *ts = (struct timespec *) tv; + +if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ))return vdso_fallback_gtod (tv, tz); tv->tv_usec /=1000 ; }
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Collapse high resolution functions
x86/vdso: Collapse high resolution functions
Commit-ID: e9a62f76f93c6f28b6c7e908eb12e5f1313870a2 Gitweb: https://git.kernel.org/tip/e9a62f76f93c6f28b6c7e908eb12e5f1313870a2 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:39 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:25 +0200
x86/vdso: Collapse high resolution functions
do_realtime() and do_monotonic() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 35 +++++++---------------------------- 1 file changed, 7 insertions(+), 28 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -28
@@ -205,35 +205,12 @@ notrace static inline u64 vgetsns(int *mode)return v * gtod->mult; } -/* Code size doesn't matter (vdso is 4k anyway) and this is faster. */ -notrace static int __always_inline do_realtime(struct timespec *ts) +notrace static int do_hres(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -u64 ns;int mode; - -do { - seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; - ts->tv_sec = base->sec; - ns = base->nsec; - ns +=vgetsns (&mode); - ns >>= gtod->shift; - }while (unlikely (gtod_read_retry (gtod, seq))); - - ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); - ts->tv_nsec = ns; - -return mode; -} - -notrace static int __always_inline do_monotonic(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC ]; -unsigned int seq;u64 ns; -int mode; do { seq = gtod_read_begin(gtod);
@@ -278,11 +255,11 @@ notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) {switch (clock) {case CLOCK_REALTIME : -if (do_realtime (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_MONOTONIC : -if (do_monotonic (ts) ==VCLOCK_NONE ) +if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) goto fallback;break ;case CLOCK_REALTIME_COARSE :
@@ -305,7 +282,9 @@ int clock_gettime(clockid_t, struct timespec *) notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz) {if (likely (tv !=NULL )) { -if (unlikely (do_realtime ((struct timespec *)tv) ==VCLOCK_NONE )) +struct timespec *ts = (struct timespec *) tv; + +if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ))return vdso_fallback_gtod (tv, tz); tv->tv_usec /=1000 ; }
PATCH v2 7/11 x86/vdso: Collapse coarse functions
In reply to Thomas Gleixnerdo_realtime_coarse() and do_monotonic_coarse() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -16
@@ -225,21 +225,9 @@ notrace static int do_hres(clockid_t clkreturn mode; } -notrace static void do_realtime_coarse(struct timespec *ts) +notrace static void do_coarse(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ]; -unsigned int seq; - -do { - seq =gtod_read_begin (gtod); - ts->tv_sec = base->sec; - ts->tv_nsec = base->nsec; - }while (unlikely (gtod_read_retry (gtod, seq))); -} - -notrace static void do_monotonic_coarse(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; do {
@@ -261,10 +249,10 @@ notrace int __vdso_clock_gettime(clockid goto fallback;break ;case CLOCK_REALTIME_COARSE : -do_realtime_coarse (ts); +do_coarse (CLOCK_REALTIME_COARSE , ts);break ;case CLOCK_MONOTONIC_COARSE : -do_monotonic_coarse (ts); +do_coarse (CLOCK_MONOTONIC_COARSE , ts);break ;default : goto fallback;
PATCH v2 7/11 x86/vdso: Collapse coarse functions
In reply to Thomas Gleixnerdo_realtime_coarse() and do_monotonic_coarse() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -16
@@ -225,21 +225,9 @@ notrace static int do_hres(clockid_t clkreturn mode; } -notrace static void do_realtime_coarse(struct timespec *ts) +notrace static void do_coarse(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ]; -unsigned int seq; - -do { - seq =gtod_read_begin (gtod); - ts->tv_sec = base->sec; - ts->tv_nsec = base->nsec; - }while (unlikely (gtod_read_retry (gtod, seq))); -} - -notrace static void do_monotonic_coarse(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; do {
@@ -261,10 +249,10 @@ notrace int __vdso_clock_gettime(clockid goto fallback;break ;case CLOCK_REALTIME_COARSE : -do_realtime_coarse (ts); +do_coarse (CLOCK_REALTIME_COARSE , ts);break ;case CLOCK_MONOTONIC_COARSE : -do_monotonic_coarse (ts); +do_coarse (CLOCK_MONOTONIC_COARSE , ts);break ;default : goto fallback;
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Collapse coarse functions
x86/vdso: Collapse coarse functions
Commit-ID: 6deec5bdef4518bd6524a47be9d621ff650d3ba4 Gitweb: https://git.kernel.org/tip/6deec5bdef4518bd6524a47be9d621ff650d3ba4 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:40 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:26 +0200
x86/vdso: Collapse coarse functions
do_realtime_coarse() and do_monotonic_coarse() are now the same except for the storage array index. Hand the index in as an argument and collapse the functions.
--- arch/x86/entry/vdso/vclock_gettime.c | 20 ++++---------------- 1 file changed, 4 insertions(+), 16 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +4 -16
@@ -227,21 +227,9 @@ notrace static int do_hres(clockid_t clk, struct timespec *ts)return mode; } -notrace static void do_realtime_coarse(struct timespec *ts) +notrace static void do_coarse(clockid_t clk, struct timespec *ts) { -struct vgtod_ts *base = >od->basetime[CLOCK_REALTIME_COARSE ]; -unsigned int seq; - -do { - seq =gtod_read_begin (gtod); - ts->tv_sec = base->sec; - ts->tv_nsec = base->nsec; - }while (unlikely (gtod_read_retry (gtod, seq))); -} - -notrace static void do_monotonic_coarse(struct timespec *ts) -{ -struct vgtod_ts *base = >od->basetime[CLOCK_MONOTONIC_COARSE ]; +struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; do {
@@ -263,10 +251,10 @@ notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) goto fallback;break ;case CLOCK_REALTIME_COARSE : -do_realtime_coarse (ts); +do_coarse (CLOCK_REALTIME_COARSE , ts);break ;case CLOCK_MONOTONIC_COARSE : -do_monotonic_coarse (ts); +do_coarse (CLOCK_MONOTONIC_COARSE , ts);break ;default : goto fallback;
PATCH v2 8/11 x86/vdso: Replace the clockid switch case
In reply to Thomas GleixnerNow that the time getter functions use the clockid as index into the storage array for the base time access, the switch case can be replaced.
- Check for clockid >= MAX_CLOCKS and for negative clockid (CPU/FD) first and call the fallback function right away.
- After establishing that clockid is < MAX_CLOCKS, convert the clockid to a bitmask
- Check for the supported high resolution and coarse functions by anding the bitmask of supported clocks and check whether a bit is set.
This completely avoids jump tables, reduces the number of conditionals and makes the VDSO extensible for other clock ids.
--- arch/x86/entry/vdso/vclock_gettime.c | 38 ++++++++++++++++------------------- 1 file changed, 18 insertions(+), 20 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +18 -20
@@ -239,29 +239,27 @@ notrace static void do_coarse(clockid_t notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) { -switch (clock) { -case CLOCK_REALTIME : -if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_MONOTONIC : -if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_REALTIME_COARSE : -do_coarse (CLOCK_REALTIME_COARSE , ts); -break ; -case CLOCK_MONOTONIC_COARSE : -do_coarse (CLOCK_MONOTONIC_COARSE , ts); -break ; -default : - goto fallback; - } +unsigned int msk; + +/* Sort out negative (CPU/FD) and invalid clocks */ +if (unlikely ((unsigned int ) clock >=MAX_CLOCKS )) +return vdso_fallback_gettime (clock, ts); -return 0 ; -fallback: +/* +* Convert the clockid to a bitmask and use it to check which +* clocks are handled in the VDSO directly. +*/ + msk =1U << clock; +if (likely (msk &VGTOD_HRES )) { +if (do_hres (clock, ts) !=VCLOCK_NONE ) +return 0 ; + }else if (msk &VGTOD_COARSE ) { +do_coarse (clock, ts); +return 0 ; + }return vdso_fallback_gettime (clock, ts); } +int clock_gettime (clockid_t ,struct timespec *) __attribute__((weak,alias ("__vdso_clock_gettime" )));
PATCH v2 8/11 x86/vdso: Replace the clockid switch case
In reply to Thomas GleixnerNow that the time getter functions use the clockid as index into the storage array for the base time access, the switch case can be replaced.
- Check for clockid >= MAX_CLOCKS and for negative clockid (CPU/FD) first and call the fallback function right away.
- After establishing that clockid is < MAX_CLOCKS, convert the clockid to a bitmask
- Check for the supported high resolution and coarse functions by anding the bitmask of supported clocks and check whether a bit is set.
This completely avoids jump tables, reduces the number of conditionals and makes the VDSO extensible for other clock ids.
--- arch/x86/entry/vdso/vclock_gettime.c | 38 ++++++++++++++++------------------- 1 file changed, 18 insertions(+), 20 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +18 -20
@@ -239,29 +239,27 @@ notrace static void do_coarse(clockid_t notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) { -switch (clock) { -case CLOCK_REALTIME : -if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_MONOTONIC : -if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_REALTIME_COARSE : -do_coarse (CLOCK_REALTIME_COARSE , ts); -break ; -case CLOCK_MONOTONIC_COARSE : -do_coarse (CLOCK_MONOTONIC_COARSE , ts); -break ; -default : - goto fallback; - } +unsigned int msk; + +/* Sort out negative (CPU/FD) and invalid clocks */ +if (unlikely ((unsigned int ) clock >=MAX_CLOCKS )) +return vdso_fallback_gettime (clock, ts); -return 0 ; -fallback: +/* +* Convert the clockid to a bitmask and use it to check which +* clocks are handled in the VDSO directly. +*/ + msk =1U << clock; +if (likely (msk &VGTOD_HRES )) { +if (do_hres (clock, ts) !=VCLOCK_NONE ) +return 0 ; + }else if (msk &VGTOD_COARSE ) { +do_coarse (clock, ts); +return 0 ; + }return vdso_fallback_gettime (clock, ts); } +int clock_gettime (clockid_t ,struct timespec *) __attribute__((weak,alias ("__vdso_clock_gettime" )));
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Replace the clockid switch case
x86/vdso: Replace the clockid switch case
Commit-ID: f3e839384164cf86faedd185b8f6024f73050f5e Gitweb: https://git.kernel.org/tip/f3e839384164cf86faedd185b8f6024f73050f5e Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:41 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:26 +0200
x86/vdso: Replace the clockid switch case
Now that the time getter functions use the clockid as index into the storage array for the base time access, the switch case can be replaced.
- Check for clockid >= MAX_CLOCKS and for negative clockid (CPU/FD) first and call the fallback function right away.
- After establishing that clockid is < MAX_CLOCKS, convert the clockid to a bitmask
- Check for the supported high resolution and coarse functions by anding the bitmask of supported clocks and check whether a bit is set.
This completely avoids jump tables, reduces the number of conditionals and makes the VDSO extensible for other clock ids.
--- arch/x86/entry/vdso/vclock_gettime.c | 38 +++++++++++++++++------------------- 1 file changed, 18 insertions(+), 20 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +18 -20
@@ -241,29 +241,27 @@ notrace static void do_coarse(clockid_t clk, struct timespec *ts) notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts) { -switch (clock) { -case CLOCK_REALTIME : -if (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_MONOTONIC : -if (do_hres (CLOCK_MONOTONIC , ts) ==VCLOCK_NONE ) - goto fallback; -break ; -case CLOCK_REALTIME_COARSE : -do_coarse (CLOCK_REALTIME_COARSE , ts); -break ; -case CLOCK_MONOTONIC_COARSE : -do_coarse (CLOCK_MONOTONIC_COARSE , ts); -break ; -default : - goto fallback; - } +unsigned int msk; -return 0 ; -fallback: +/* Sort out negative (CPU/FD) and invalid clocks */ +if (unlikely ((unsigned int ) clock >=MAX_CLOCKS )) +return vdso_fallback_gettime (clock, ts); + +/* +* Convert the clockid to a bitmask and use it to check which +* clocks are handled in the VDSO directly. +*/ + msk =1U << clock; +if (likely (msk &VGTOD_HRES )) { +if (do_hres (clock, ts) !=VCLOCK_NONE ) +return 0 ; + }else if (msk &VGTOD_COARSE ) { +do_coarse (clock, ts); +return 0 ; + }return vdso_fallback_gettime (clock, ts); } +int clock_gettime (clockid_t ,struct timespec *) __attribute__((weak,alias ("__vdso_clock_gettime" )));
PATCH v2 9/11 x86/vdso: Simplify the invalid vclock case
In reply to Thomas GleixnerThe code flow for the vclocks is convoluted as it requires the vclocks which can be invalidated separately from the vsyscall_gtod_data sequence to store the fact in a separate variable. That's inefficient.
Restructure the code so the vclock readout returns cycles and the conversion to nanoseconds is handled at the call site.
If the clock gets invalidated or vclock is already VCLOCK_NONE, return U64_MAX as the cycle value, which is invalid for all clocks and leave the sequence loop immediately in that case by calling the fallback function directly.
This allows to remove the gettimeofday fallback as it now uses the clock_gettime() fallback and does the nanoseconds to microseconds conversion in the same way as it does when the vclock is functional. It does not make a difference whether the division by 1000 happens in the kernel fallback or in userspace.
Generates way better code and gains a few cycles back.
--- arch/x86/entry/vdso/vclock_gettime.c | 81 +++++++++-------------------------- 1 file changed, 21 insertions(+), 60 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +21 -60
@@ -48,16 +48,6 @@ notrace static long vdso_fallback_gettimreturn ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm("syscall" :"=a" (ret) : -"0" (__NR_gettimeofday),"D" (tv),"S" (tz) :"memory" ); -return ret; -} - -#else notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
@@ -75,21 +65,6 @@ notrace static long vdso_fallback_gettim return ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm( -"mov %%ebx, %%edx \n" -"mov %2, %%ebx \n" -"call __kernel_vsyscall \n" -"mov %%edx, %%ebx \n" - :"=a" (ret) - :"0" (__NR_gettimeofday),"g" (tv),"c" (tz) - :"memory" ,"edx" ); -return ret; -} -#endif #ifdef CONFIG_PARAVIRT_CLOCK
@@ -98,7 +73,7 @@ static notrace const struct pvclock_vsysreturn (const struct pvclock_vsyscall_time_info *)&pvclock_page; } -static notrace u64 vread_pvclock(int *mode) +static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti;u64 ret;
@@ -130,10 +105,8 @@ static notrace u64 vread_pvclock(int *modo { version =pvclock_read_begin (pvti); -if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) { - *mode =VCLOCK_NONE ; -return 0 ; - } +if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) +return U64_MAX ; ret =__pvclock_read_cycles (pvti,rdtsc_ordered ()); }while (pvclock_read_retry (pvti, version));
@@ -148,17 +121,12 @@ static notrace u64 vread_pvclock(int *mo }#endif #ifdef CONFIG_HYPERV_TSCPAGE -static notrace u64 vread_hvclock(int *mode) +static notrace u64 vread_hvclock(void) {const struct ms_hyperv_tsc_page *tsc_pg = (const struct ms_hyperv_tsc_page *)&hvclock_page; -u64 current_tick =hv_read_tsc_page (tsc_pg); - -if (current_tick !=U64_MAX ) -return current_tick; - *mode =VCLOCK_NONE ; -return 0 ; +return hv_read_tsc_page (tsc_pg); }#endif
@@ -182,47 +150,42 @@ notrace static u64 vread_tsc(void)return last; } -notrace static inline u64 vgetsns(int *mode) +notrace static inline u64 vgetcyc(int mode) { -u64 v; -cycles_t cycles; - -if (gtod->vclock_mode ==VCLOCK_TSC ) - cycles =vread_tsc (); +if (mode ==VCLOCK_TSC ) +return vread_tsc ();#ifdef CONFIG_PARAVIRT_CLOCK - else if (gtod->vclock_mode == VCLOCK_PVCLOCK) - cycles =vread_pvclock (mode); + else if (mode == VCLOCK_PVCLOCK) + return vread_pvclock();#endif #ifdef CONFIG_HYPERV_TSCPAGE - else if (gtod->vclock_mode == VCLOCK_HVCLOCK) - cycles =vread_hvclock (mode); + else if (mode == VCLOCK_HVCLOCK) + return vread_hvclock();#endif - else - return 0; - v = cycles - gtod->cycle_last; -return v * gtod->mult; +return U64_MAX ; } notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -int mode; -u64 ns; +u64 cycles, ns;do { seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; ts->tv_sec = base->sec; ns = base->nsec; - ns +=vgetsns (&mode); + cycles =vgetcyc (gtod->vclock_mode); +if (unlikely ((s64 )cycles <0 )) +return vdso_fallback_gettime (clk, ts); + ns += (cycles - gtod->cycle_last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq))); ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); ts->tv_nsec = ns; -return mode; +return 0 ; } notrace static void do_coarse(clockid_t clk, struct timespec *ts)
@@ -251,8 +214,7 @@ notrace int __vdso_clock_gettime(clockid*/ msk =1U << clock;if (likely (msk &VGTOD_HRES )) { -if (do_hres (clock, ts) !=VCLOCK_NONE ) -return 0 ; +return do_hres (clock, ts); }else if (msk &VGTOD_COARSE ) {do_coarse (clock, ts);return 0 ;
@@ -268,8 +230,7 @@ notrace int __vdso_gettimeofday(struct tif (likely (tv !=NULL )) {struct timespec *ts = (struct timespec *) tv; -if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE )) -return vdso_fallback_gtod (tv, tz); +do_hres (CLOCK_REALTIME , ts); tv->tv_usec /=1000 ; }if (unlikely (tz !=NULL )) {
PATCH v2 9/11 x86/vdso: Simplify the invalid vclock case
In reply to Thomas GleixnerThe code flow for the vclocks is convoluted as it requires the vclocks which can be invalidated separately from the vsyscall_gtod_data sequence to store the fact in a separate variable. That's inefficient.
Restructure the code so the vclock readout returns cycles and the conversion to nanoseconds is handled at the call site.
If the clock gets invalidated or vclock is already VCLOCK_NONE, return U64_MAX as the cycle value, which is invalid for all clocks and leave the sequence loop immediately in that case by calling the fallback function directly.
This allows to remove the gettimeofday fallback as it now uses the clock_gettime() fallback and does the nanoseconds to microseconds conversion in the same way as it does when the vclock is functional. It does not make a difference whether the division by 1000 happens in the kernel fallback or in userspace.
Generates way better code and gains a few cycles back.
--- arch/x86/entry/vdso/vclock_gettime.c | 81 +++++++++-------------------------- 1 file changed, 21 insertions(+), 60 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +21 -60
@@ -48,16 +48,6 @@ notrace static long vdso_fallback_gettimreturn ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm("syscall" :"=a" (ret) : -"0" (__NR_gettimeofday),"D" (tv),"S" (tz) :"memory" ); -return ret; -} - -#else notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
@@ -75,21 +65,6 @@ notrace static long vdso_fallback_gettim return ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm( -"mov %%ebx, %%edx \n" -"mov %2, %%ebx \n" -"call __kernel_vsyscall \n" -"mov %%edx, %%ebx \n" - :"=a" (ret) - :"0" (__NR_gettimeofday),"g" (tv),"c" (tz) - :"memory" ,"edx" ); -return ret; -} -#endif #ifdef CONFIG_PARAVIRT_CLOCK
@@ -98,7 +73,7 @@ static notrace const struct pvclock_vsysreturn (const struct pvclock_vsyscall_time_info *)&pvclock_page; } -static notrace u64 vread_pvclock(int *mode) +static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti;u64 ret;
@@ -130,10 +105,8 @@ static notrace u64 vread_pvclock(int *modo { version =pvclock_read_begin (pvti); -if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) { - *mode =VCLOCK_NONE ; -return 0 ; - } +if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) +return U64_MAX ; ret =__pvclock_read_cycles (pvti,rdtsc_ordered ()); }while (pvclock_read_retry (pvti, version));
@@ -148,17 +121,12 @@ static notrace u64 vread_pvclock(int *mo }#endif #ifdef CONFIG_HYPERV_TSCPAGE -static notrace u64 vread_hvclock(int *mode) +static notrace u64 vread_hvclock(void) {const struct ms_hyperv_tsc_page *tsc_pg = (const struct ms_hyperv_tsc_page *)&hvclock_page; -u64 current_tick =hv_read_tsc_page (tsc_pg); - -if (current_tick !=U64_MAX ) -return current_tick; - *mode =VCLOCK_NONE ; -return 0 ; +return hv_read_tsc_page (tsc_pg); }#endif
@@ -182,47 +150,42 @@ notrace static u64 vread_tsc(void)return last; } -notrace static inline u64 vgetsns(int *mode) +notrace static inline u64 vgetcyc(int mode) { -u64 v; -cycles_t cycles; - -if (gtod->vclock_mode ==VCLOCK_TSC ) - cycles =vread_tsc (); +if (mode ==VCLOCK_TSC ) +return vread_tsc ();#ifdef CONFIG_PARAVIRT_CLOCK - else if (gtod->vclock_mode == VCLOCK_PVCLOCK) - cycles =vread_pvclock (mode); + else if (mode == VCLOCK_PVCLOCK) + return vread_pvclock();#endif #ifdef CONFIG_HYPERV_TSCPAGE - else if (gtod->vclock_mode == VCLOCK_HVCLOCK) - cycles =vread_hvclock (mode); + else if (mode == VCLOCK_HVCLOCK) + return vread_hvclock();#endif - else - return 0; - v = cycles - gtod->cycle_last; -return v * gtod->mult; +return U64_MAX ; } notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -int mode; -u64 ns; +u64 cycles, ns;do { seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; ts->tv_sec = base->sec; ns = base->nsec; - ns +=vgetsns (&mode); + cycles =vgetcyc (gtod->vclock_mode); +if (unlikely ((s64 )cycles <0 )) +return vdso_fallback_gettime (clk, ts); + ns += (cycles - gtod->cycle_last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq))); ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); ts->tv_nsec = ns; -return mode; +return 0 ; } notrace static void do_coarse(clockid_t clk, struct timespec *ts)
@@ -251,8 +214,7 @@ notrace int __vdso_clock_gettime(clockid*/ msk =1U << clock;if (likely (msk &VGTOD_HRES )) { -if (do_hres (clock, ts) !=VCLOCK_NONE ) -return 0 ; +return do_hres (clock, ts); }else if (msk &VGTOD_COARSE ) {do_coarse (clock, ts);return 0 ;
@@ -268,8 +230,7 @@ notrace int __vdso_gettimeofday(struct tif (likely (tv !=NULL )) {struct timespec *ts = (struct timespec *) tv; -if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE )) -return vdso_fallback_gtod (tv, tz); +do_hres (CLOCK_REALTIME , ts); tv->tv_usec /=1000 ; }if (unlikely (tz !=NULL )) {
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Simplify the invalid vclock case
x86/vdso: Simplify the invalid vclock case
Commit-ID: 4f72adc5068294268387a81a6bf91d9bb07ecc5c Gitweb: https://git.kernel.org/tip/4f72adc5068294268387a81a6bf91d9bb07ecc5c Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:42 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:26 +0200
x86/vdso: Simplify the invalid vclock case
The code flow for the vclocks is convoluted as it requires the vclocks which can be invalidated separately from the vsyscall_gtod_data sequence to store the fact in a separate variable. That's inefficient.
Restructure the code so the vclock readout returns cycles and the conversion to nanoseconds is handled at the call site.
If the clock gets invalidated or vclock is already VCLOCK_NONE, return U64_MAX as the cycle value, which is invalid for all clocks and leave the sequence loop immediately in that case by calling the fallback function directly.
This allows to remove the gettimeofday fallback as it now uses the clock_gettime() fallback and does the nanoseconds to microseconds conversion in the same way as it does when the vclock is functional. It does not make a difference whether the division by 1000 happens in the kernel fallback or in userspace.
Generates way better code and gains a few cycles back.
--- arch/x86/entry/vdso/vclock_gettime.c | 82 +++++++++--------------------------- 1 file changed, 21 insertions(+), 61 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +21 -61
@@ -49,17 +49,6 @@ notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)return ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm ("syscall" :"=a" (ret),"=m" (*tv),"=m" (*tz) : -"0" (__NR_gettimeofday),"D" (tv),"S" (tz) : -"memory" ,"rcx" ,"r11" ); -return ret; -} - -#else notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
@@ -77,21 +66,6 @@ notrace static long vdso_fallback_gettime(long clock, struct timespec *ts) return ret; } -notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz) -{ -long ret; - - asm ( -"mov %%ebx, %%edx \n" -"mov %[tv], %%ebx \n" -"call __kernel_vsyscall \n" -"mov %%edx, %%ebx \n" - :"=a" (ret),"=m" (*tv),"=m" (*tz) - :"0" (__NR_gettimeofday), [tv]"g" (tv),"c" (tz) - :"memory" ,"edx" ); -return ret; -} -#endif #ifdef CONFIG_PARAVIRT_CLOCK
@@ -100,7 +74,7 @@ static notrace const struct pvclock_vsyscall_time_info *get_pvti0(void)return (const struct pvclock_vsyscall_time_info *)&pvclock_page; } -static notrace u64 vread_pvclock(int *mode) +static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti;u64 ret;
@@ -132,10 +106,8 @@ static notrace u64 vread_pvclock(int *mode)do { version =pvclock_read_begin (pvti); -if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) { - *mode =VCLOCK_NONE ; -return 0 ; - } +if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) +return U64_MAX ; ret =__pvclock_read_cycles (pvti,rdtsc_ordered ()); }while (pvclock_read_retry (pvti, version));
@@ -150,17 +122,12 @@ static notrace u64 vread_pvclock(int *mode) }#endif #ifdef CONFIG_HYPERV_TSCPAGE -static notrace u64 vread_hvclock(int *mode) +static notrace u64 vread_hvclock(void) {const struct ms_hyperv_tsc_page *tsc_pg = (const struct ms_hyperv_tsc_page *)&hvclock_page; -u64 current_tick =hv_read_tsc_page (tsc_pg); - -if (current_tick !=U64_MAX ) -return current_tick; - *mode =VCLOCK_NONE ; -return 0 ; +return hv_read_tsc_page (tsc_pg); }#endif
@@ -184,47 +151,42 @@ notrace static u64 vread_tsc(void)return last; } -notrace static inline u64 vgetsns(int *mode) +notrace static inline u64 vgetcyc(int mode) { -u64 v; -cycles_t cycles; - -if (gtod->vclock_mode ==VCLOCK_TSC ) - cycles =vread_tsc (); +if (mode ==VCLOCK_TSC ) +return vread_tsc ();#ifdef CONFIG_PARAVIRT_CLOCK - else if (gtod->vclock_mode == VCLOCK_PVCLOCK) - cycles =vread_pvclock (mode); + else if (mode == VCLOCK_PVCLOCK) + return vread_pvclock();#endif #ifdef CONFIG_HYPERV_TSCPAGE - else if (gtod->vclock_mode == VCLOCK_HVCLOCK) - cycles =vread_hvclock (mode); + else if (mode == VCLOCK_HVCLOCK) + return vread_hvclock();#endif - else - return 0; - v = cycles - gtod->cycle_last; -return v * gtod->mult; +return U64_MAX ; } notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk];unsigned int seq; -int mode; -u64 ns; +u64 cycles, ns;do { seq =gtod_read_begin (gtod); - mode = gtod->vclock_mode; ts->tv_sec = base->sec; ns = base->nsec; - ns +=vgetsns (&mode); + cycles =vgetcyc (gtod->vclock_mode); +if (unlikely ((s64 )cycles <0 )) +return vdso_fallback_gettime (clk, ts); + ns += (cycles - gtod->cycle_last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq))); ts->tv_sec +=__iter_div_u64_rem (ns,NSEC_PER_SEC , &ns); ts->tv_nsec = ns; -return mode; +return 0 ; } notrace static void do_coarse(clockid_t clk, struct timespec *ts)
@@ -253,8 +215,7 @@ notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts)*/ msk =1U << clock;if (likely (msk &VGTOD_HRES )) { -if (do_hres (clock, ts) !=VCLOCK_NONE ) -return 0 ; +return do_hres (clock, ts); }else if (msk &VGTOD_COARSE ) {do_coarse (clock, ts);return 0 ;
@@ -270,8 +231,7 @@ notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz)if (likely (tv !=NULL )) {struct timespec *ts = (struct timespec *) tv; -if (unlikely (do_hres (CLOCK_REALTIME , ts) ==VCLOCK_NONE )) -return vdso_fallback_gtod (tv, tz); +do_hres (CLOCK_REALTIME , ts); tv->tv_usec /=1000 ; }if (unlikely (tz !=NULL )) {
PATCH v2 10/11 x86/vdso: Move cycle_last handling into the caller
In reply to Thomas GleixnerDereferencing gtod->cycle_last all over the place and foing the cycles < last comparison in the vclock read functions generates horrible code. Doing it at the call site is much better and gains a few cycles both for TSC and pvclock.
Caveat: This adds the comparison to the hyperv vclock as well, but I have no way to test that.
--- arch/x86/entry/vdso/vclock_gettime.c | 39 ++++++----------------------------- 1 file changed, 7 insertions(+), 32 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -32
@@ -76,9 +76,8 @@ static notrace const struct pvclock_vsys static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti; -u64 ret; -u64 last;u32 version; +u64 ret;/* * Note: The kernel and hypervisor must guarantee that cpu ID
@@ -111,13 +110,7 @@ static notrace u64 vread_pvclock(void) ret = __pvclock_read_cycles(pvti, rdtsc_ordered()); }while (pvclock_read_retry (pvti, version)); -/* refer to vread_tsc() comment for rationale */ - last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -return last; +return ret; }#endif #ifdef CONFIG_HYPERV_TSCPAGE
@@ -130,30 +123,10 @@ static notrace u64 vread_hvclock(void) }#endif -notrace static u64 vread_tsc(void) -{ -u64 ret = (u64 )rdtsc_ordered (); -u64 last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -/* -* GCC likes to generate cmov here, but this branch is extremely -* predictable (it's just a function of time and the likely is -* very likely) and there's a data dependence, so force GCC -* to generate a branch instead. I don't barrier() because -* we don't actually need a barrier, and if this function -* ever gets inlined it will generate worse code. -*/ - asmvolatile ("" ); -return last; -} - notrace static inline u64 vgetcyc(int mode) {if (mode ==VCLOCK_TSC ) -return vread_tsc (); +return (u64 )rdtsc_ordered ();#ifdef CONFIG_PARAVIRT_CLOCK else if (mode == VCLOCK_PVCLOCK) return vread_pvclock();
@@ -168,17 +141,19 @@ notrace static inline u64 vgetcyc(int mo notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk]; +u64 cycles, last, ns;unsigned int seq; -u64 cycles, ns;do { seq =gtod_read_begin (gtod); ts->tv_sec = base->sec; ns = base->nsec; + last = gtod->cycle_last; cycles =vgetcyc (gtod->vclock_mode);if (unlikely ((s64 )cycles <0 ))return vdso_fallback_gettime (clk, ts); - ns += (cycles - gtod->cycle_last) * gtod->mult; +if (cycles > last) + ns += (cycles - last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
PATCH v2 10/11 x86/vdso: Move cycle_last handling into the caller
In reply to Thomas GleixnerDereferencing gtod->cycle_last all over the place and foing the cycles < last comparison in the vclock read functions generates horrible code. Doing it at the call site is much better and gains a few cycles both for TSC and pvclock.
Caveat: This adds the comparison to the hyperv vclock as well, but I have no way to test that.
--- arch/x86/entry/vdso/vclock_gettime.c | 39 ++++++----------------------------- 1 file changed, 7 insertions(+), 32 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -32
@@ -76,9 +76,8 @@ static notrace const struct pvclock_vsys static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti; -u64 ret; -u64 last;u32 version; +u64 ret;/* * Note: The kernel and hypervisor must guarantee that cpu ID
@@ -111,13 +110,7 @@ static notrace u64 vread_pvclock(void) ret = __pvclock_read_cycles(pvti, rdtsc_ordered()); }while (pvclock_read_retry (pvti, version)); -/* refer to vread_tsc() comment for rationale */ - last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -return last; +return ret; }#endif #ifdef CONFIG_HYPERV_TSCPAGE
@@ -130,30 +123,10 @@ static notrace u64 vread_hvclock(void) }#endif -notrace static u64 vread_tsc(void) -{ -u64 ret = (u64 )rdtsc_ordered (); -u64 last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -/* -* GCC likes to generate cmov here, but this branch is extremely -* predictable (it's just a function of time and the likely is -* very likely) and there's a data dependence, so force GCC -* to generate a branch instead. I don't barrier() because -* we don't actually need a barrier, and if this function -* ever gets inlined it will generate worse code. -*/ - asmvolatile ("" ); -return last; -} - notrace static inline u64 vgetcyc(int mode) {if (mode ==VCLOCK_TSC ) -return vread_tsc (); +return (u64 )rdtsc_ordered ();#ifdef CONFIG_PARAVIRT_CLOCK else if (mode == VCLOCK_PVCLOCK) return vread_pvclock();
@@ -168,17 +141,19 @@ notrace static inline u64 vgetcyc(int mo notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk]; +u64 cycles, last, ns;unsigned int seq; -u64 cycles, ns;do { seq =gtod_read_begin (gtod); ts->tv_sec = base->sec; ns = base->nsec; + last = gtod->cycle_last; cycles =vgetcyc (gtod->vclock_mode);if (unlikely ((s64 )cycles <0 ))return vdso_fallback_gettime (clk, ts); - ns += (cycles - gtod->cycle_last) * gtod->mult; +if (cycles > last) + ns += (cycles - last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x86/vdso: Move cycle_last handling into the caller
x86/vdso: Move cycle_last handling into the caller
Commit-ID: 3e89bf35ebf59c12e8c1476f6681fae0ebdcb2a7 Gitweb: https://git.kernel.org/tip/3e89bf35ebf59c12e8c1476f6681fae0ebdcb2a7 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:43 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:27 +0200
x86/vdso: Move cycle_last handling into the caller
Dereferencing gtod->cycle_last all over the place and foing the cycles < last comparison in the vclock read functions generates horrible code. Doing it at the call site is much better and gains a few cycles both for TSC and pvclock.
Caveat: This adds the comparison to the hyperv vclock as well, but I have no way to test that.
--- arch/x86/entry/vdso/vclock_gettime.c | 39 +++++++----------------------------- 1 file changed, 7 insertions(+), 32 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +7 -32
@@ -77,9 +77,8 @@ static notrace const struct pvclock_vsyscall_time_info *get_pvti0(void) static notrace u64 vread_pvclock(void) {const struct pvclock_vcpu_time_info *pvti = &get_pvti0 ()->pvti; -u64 ret; -u64 last;u32 version; +u64 ret;/* * Note: The kernel and hypervisor must guarantee that cpu ID
@@ -112,13 +111,7 @@ static notrace u64 vread_pvclock(void) ret = __pvclock_read_cycles(pvti, rdtsc_ordered()); }while (pvclock_read_retry (pvti, version)); -/* refer to vread_tsc() comment for rationale */ - last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -return last; +return ret; }#endif #ifdef CONFIG_HYPERV_TSCPAGE
@@ -131,30 +124,10 @@ static notrace u64 vread_hvclock(void) }#endif -notrace static u64 vread_tsc(void) -{ -u64 ret = (u64 )rdtsc_ordered (); -u64 last = gtod->cycle_last; - -if (likely (ret >= last)) -return ret; - -/* -* GCC likes to generate cmov here, but this branch is extremely -* predictable (it's just a function of time and the likely is -* very likely) and there's a data dependence, so force GCC -* to generate a branch instead. I don't barrier() because -* we don't actually need a barrier, and if this function -* ever gets inlined it will generate worse code. -*/ - asmvolatile ("" ); -return last; -} - notrace static inline u64 vgetcyc(int mode) {if (mode ==VCLOCK_TSC ) -return vread_tsc (); +return (u64 )rdtsc_ordered ();#ifdef CONFIG_PARAVIRT_CLOCK else if (mode == VCLOCK_PVCLOCK) return vread_pvclock();
@@ -169,17 +142,19 @@ notrace static inline u64 vgetcyc(int mode) notrace static int do_hres(clockid_t clk, struct timespec *ts) {struct vgtod_ts *base = >od->basetime[clk]; +u64 cycles, last, ns;unsigned int seq; -u64 cycles, ns;do { seq =gtod_read_begin (gtod); ts->tv_sec = base->sec; ns = base->nsec; + last = gtod->cycle_last; cycles =vgetcyc (gtod->vclock_mode);if (unlikely ((s64 )cycles <0 ))return vdso_fallback_gettime (clk, ts); - ns += (cycles - gtod->cycle_last) * gtod->mult; +if (cycles > last) + ns += (cycles - last) * gtod->mult; ns >>= gtod->shift; }while (unlikely (gtod_read_retry (gtod, seq)));
PATCH v2 11/11 x66/vdso: Add CLOCK_TAI support
In reply to Thomas GleixnerWith the storage array in place it's now trivial to support CLOCK_TAI in the vdso. Extend the base time storage array and add the update code.
---
V2: Remove the masking trick
arch/x86/entry/vsyscall/vsyscall_gtod.c | 4 ++++ arch/x86/include/asm/vgtod.h | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/entry/vsyscall/vsyscall_gtod.c +4 -0
@@ -51,6 +51,10 @@ void update_vsyscall(struct timekeeper * base->sec = tk->xtime_sec; base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_TAI ]; + base->sec = tk->xtime_sec + (s64 )tk->tai_offset; + base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_MONOTONIC ]; base->sec = tk->xtime_sec + tk->wall_to_monotonic.tv_sec; nsec = tk->tkr_mono.xtime_nsec;
arch/x86/include/asm/vgtod.h +2 -2
@@ -18,8 +18,8 @@ struct vgtod_ts {u64 nsec; }; -#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) -#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_BASES (CLOCK_TAI + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC) | BIT(CLOCK_TAI))#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE))/*
PATCH v2 11/11 x66/vdso: Add CLOCK_TAI support
In reply to Thomas GleixnerWith the storage array in place it's now trivial to support CLOCK_TAI in the vdso. Extend the base time storage array and add the update code.
---
V2: Remove the masking trick
arch/x86/entry/vsyscall/vsyscall_gtod.c | 4 ++++ arch/x86/include/asm/vgtod.h | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/entry/vsyscall/vsyscall_gtod.c +4 -0
@@ -51,6 +51,10 @@ void update_vsyscall(struct timekeeper * base->sec = tk->xtime_sec; base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_TAI ]; + base->sec = tk->xtime_sec + (s64 )tk->tai_offset; + base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_MONOTONIC ]; base->sec = tk->xtime_sec + tk->wall_to_monotonic.tv_sec; nsec = tk->tkr_mono.xtime_nsec;
arch/x86/include/asm/vgtod.h +2 -2
@@ -18,8 +18,8 @@ struct vgtod_ts {u64 nsec; }; -#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) -#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_BASES (CLOCK_TAI + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC) | BIT(CLOCK_TAI))#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE))/*
Re: [patch V2 11/11] x66/vdso: Add CLOCK_TAI support
Quoted 4 lines · Thomas Gleixner
With the storage array in place it's now trivial to support
CLOCK_TAIin the vdso. Extend the base time storage array and add the update code.Signed-off-by Thomas Gleixner<[email protected]>
That's much better. Thanks
tip-bot for Thomas Gleixner
merge bot[tip:x86/vdso] x66/vdso: Add CLOCK_TAI support
In reply to Thomas Gleixnerx66/vdso: Add CLOCK_TAI support
Commit-ID: 315f28fa3a7bfbbb6233d800dffa327e87f96129 Gitweb: https://git.kernel.org/tip/315f28fa3a7bfbbb6233d800dffa327e87f96129 Author: Thomas Gleixner <[email protected]> AuthorDate: Mon, 17 Sep 2018 14:45:44 +0200 Committer: Thomas Gleixner <[email protected]> CommitDate: Thu, 4 Oct 2018 23:00:27 +0200
x66/vdso: Add CLOCK_TAI support
With the storage array in place it's now trivial to support CLOCK_TAI in the vdso. Extend the base time storage array and add the update code.
--- arch/x86/entry/vsyscall/vsyscall_gtod.c | 4 ++++ arch/x86/include/asm/vgtod.h | 4 ++-- 2 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/entry/vsyscall/vsyscall_gtod.c +4 -0
@@ -51,6 +51,10 @@ void update_vsyscall(struct timekeeper *tk) base->sec = tk->xtime_sec; base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_TAI ]; + base->sec = tk->xtime_sec + (s64 )tk->tai_offset; + base->nsec = tk->tkr_mono.xtime_nsec; + base = &vdata->basetime[CLOCK_MONOTONIC ]; base->sec = tk->xtime_sec + tk->wall_to_monotonic.tv_sec; nsec = tk->tkr_mono.xtime_nsec;
arch/x86/include/asm/vgtod.h +2 -2
@@ -18,8 +18,8 @@ struct vgtod_ts {u64 nsec; }; -#define VGTOD_BASES (CLOCK_MONOTONIC_COARSE + 1) -#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC)) +#define VGTOD_BASES (CLOCK_TAI + 1) +#define VGTOD_HRES (BIT(CLOCK_REALTIME) | BIT(CLOCK_MONOTONIC) | BIT(CLOCK_TAI))#define VGTOD_COARSE (BIT(CLOCK_REALTIME_COARSE) | BIT(CLOCK_MONOTONIC_COARSE))/*