VDSO consolidation
Subsystems (17)
v2 cover VDSO consolidation
This is the second version of the VDSO consolidation series. The first version can be found here:
https://lore.kernel.org/lkml/r/[email protected]
The changes since V1:
- Tiny optimization of x86/TSC (new)
- Address review comments from V1 (bisectability, spelling ...)
- Include the preparatory patches from Christophe which allow powerpc to be switched over.
This conflicts slightly with the ARM64 time namespace patch series, but that's trivial to fix up.
Thanks,
tglx
----
arch/arm/Kconfig | 1 arch/arm/include/asm/clocksource.h | 5 - arch/arm/include/asm/vdso/gettimeofday.h | 6 + arch/arm/include/asm/vdso/vsyscall.h | 35 ------- arch/arm64/Kconfig | 1 arch/arm64/include/asm/clocksource.h | 5 - arch/arm64/include/asm/vdso/vsyscall.h | 9 - arch/mips/Kconfig | 1 arch/mips/include/asm/clocksource.h | 18 --- arch/mips/include/asm/vdso/vsyscall.h | 9 - arch/mips/kernel/csrc-r4k.c | 2 arch/x86/Kconfig | 1 arch/x86/entry/vdso/vma.c | 8 + arch/x86/include/asm/clocksource.h | 23 +++- arch/x86/include/asm/mshyperv.h | 4 arch/x86/include/asm/vdso/gettimeofday.h | 6 - arch/x86/include/asm/vdso/vsyscall.h | 15 --- arch/x86/include/asm/vgtod.h | 6 - arch/x86/kernel/kvmclock.c | 9 + arch/x86/kernel/pvclock.c | 2 arch/x86/kernel/time.c | 12 -- arch/x86/kernel/tsc.c | 32 ++++-- arch/x86/kvm/trace.h | 4 arch/x86/kvm/x86.c | 22 ++-- arch/x86/xen/time.c | 36 ++++--- b/arch/mips/include/asm/vdso/gettimeofday.h | 29 ++---- drivers/clocksource/arm_arch_timer.c | 8 - drivers/clocksource/hyperv_timer.c | 7 + drivers/clocksource/mips-gic-timer.c | 8 - include/asm-generic/vdso/vsyscall.h | 14 -- include/linux/clocksource.h | 102 ++++++++++++--------- include/vdso/datapage.h | 2 kernel/time/clocksource.c | 9 + kernel/time/namespace.c | 7 - kernel/time/vsyscall.c | 12 +- lib/vdso/gettimeofday.c | 133 +++++++++++++++++++++------- 36 files changed, 317 insertions(+), 286 deletions(-)
PATCH v2 1/17 x86/vdso: Mark the TSC clocksource path likely
Jumping out of line for the TSC clcoksource read is creating awful code. TSC is likely to be the clocksource at least on bare metal and the PV interfaces are sufficiently more work that the jump over the TSC read is just in the noise.
--- arch/x86/include/asm/vdso/gettimeofday.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/vdso/gettimeofday.h +1 -1
@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (clock_mode ==VCLOCK_TSC ) +if (likely (clock_mode ==VCLOCK_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc
Re: [patch V2 01/17] x86/vdso: Mark the TSC clocksource path likely
Quoted 5 lines · Thomas Gleixner
Jumping out of line for the TSC clcoksource read is creating awful code. TSC is likely to be the clocksource at least on bare metal and the PV interfaces are sufficiently more work that the jump over the TSC read is just in the noise.
Quoted 17 lines
Signed-off-by Thomas Gleixner <[email protected]>--- arch/x86/include/asm/vdso/gettimeofday.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/include/asm/vdso/gettimeofday.h +1
-1@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (clock_mode ==VCLOCK_TSC ) +if (likely (clock_mode ==VCLOCK_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] x86/vdso: Mark the TSC clocksource path likely
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 50e818715821b89c7abac90a97721f106e893d83 Gitweb: https://git.kernel.org/tip/50e818715821b89c7abac90a97721f106e893d83 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:48 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:19 +01:00
x86/vdso: Mark the TSC clocksource path likely
Jumping out of line for the TSC clcoksource read is creating awful code. TSC is likely to be the clocksource at least on bare metal and the PV interfaces are sufficiently more work that the jump over the TSC read is just in the noise.
--- arch/x86/include/asm/vdso/gettimeofday.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/vdso/gettimeofday.h +1 -1
@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (clock_mode ==VCLOCK_TSC ) +if (likely (clock_mode ==VCLOCK_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc
PATCH v2 2/17 ARM: vdso: Remove unused function
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
The function is nowhere used. Aside of that this check should only cover the high resolution parts of the VDSO which require a VDSO capable clocksource and not the complete functionality as the name suggests. Will be replaced with something more useful.
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- 1 file changed, 7 deletions(-)
arch/arm/include/asm/vdso/vsyscall.h +0 -7
@@ -50,13 +50,6 @@ int __arm_get_clock_mode(struct timekeep#define __arch_get_clock_mode __arm_get_clock_modestatic __always_inline -int __arm_use_vsyscall(struct vdso_data *vdata) -{ -return vdata[CS_HRES_COARSE ].clock_mode; -} -#define __arch_use_vsyscall __arm_use_vsyscall - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
Re: [patch V2 02/17] ARM: vdso: Remove unused function
Quoted 9 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
The function is nowhere used. Aside of that this check should only cover the high resolution parts of the VDSO which require a VDSO capable clocksource and not the complete functionality as the name suggests. Will be replaced with something more useful.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 21 lines
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- 1 file changed, 7 deletions(-)arch/arm/include/asm/vdso/vsyscall.h +0
-7@@ -50,13 +50,6 @@ int __arm_get_clock_mode(struct timekeep#define __arch_get_clock_mode __arm_get_clock_modestatic __always_inline -int __arm_use_vsyscall(struct vdso_data *vdata) -{ -return vdata[CS_HRES_COARSE ].clock_mode; -} -#define __arch_use_vsyscall __arm_use_vsyscall - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] ARM: vdso: Remove unused function
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 78560d41064ad3d377e3d1a1ee87526301f4e946 Gitweb: https://git.kernel.org/tip/78560d41064ad3d377e3d1a1ee87526301f4e946 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:49 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:20 +01:00
ARM: vdso: Remove unused function
The function is nowhere used. Aside of that this check should only cover the high resolution parts of the VDSO which require a VDSO capable clocksource and not the complete functionality as the name suggests. Will be replaced with something more useful.
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- 1 file changed, 7 deletions(-)
arch/arm/include/asm/vdso/vsyscall.h +0 -7
@@ -50,13 +50,6 @@ int __arm_get_clock_mode(struct timekeeper *tk)#define __arch_get_clock_mode __arm_get_clock_modestatic __always_inline -int __arm_use_vsyscall(struct vdso_data *vdata) -{ -return vdata[CS_HRES_COARSE ].clock_mode; -} -#define __arch_use_vsyscall __arm_use_vsyscall - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
PATCH v2 3/17 lib/vdso: Allow the high resolution parts to be compiled out
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
If the architecture knows at compile time that there is no VDSO capable clocksource supported it makes sense to optimize the guts of the high resolution parts of the VDSO out at build time. Add a helper function to check whether the VDSO should be high resolution capable and provide a stub which can be overridden by an architecture.
--- lib/vdso/gettimeofday.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
lib/vdso/gettimeofday.c +11 -0
@@ -38,6 +38,13 @@ u64 vdso_calc_delta(u64 cycles, u64 last }#endif +#ifndef __arch_vdso_hres_capable +static inline bool __arch_vdso_hres_capable (void ) +{ +return true; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)
@@ -101,6 +108,10 @@ static __always_inline int do_hres(constu64 cycles, last, sec, ns;u32 seq; +/* Allows to compile the high resolution parts out */ +if (!__arch_vdso_hres_capable ()) +return -1 ; + do { /* * Open coded to handle VCLOCK_TIMENS. Time namespace
Re: [patch V2 03/17] lib/vdso: Allow the high resolution parts to be compiled out
Quoted 10 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
If the architecture knows at compile time that there is no VDSO capable clocksource supported it makes sense to optimize the guts of the high resolution parts of the VDSO out at build time. Add a helper function to check whether the VDSO should be high resolution capable and provide a stub which can be overridden by an architecture.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 32 lines
--- lib/vdso/gettimeofday.c | 11 +++++++++++ 1 file changed, 11 insertions(+)lib/vdso/gettimeofday.c +11
-0@@ -38,6 +38,13 @@ u64 vdso_calc_delta(u64 cycles, u64 last }#endif +#ifndef __arch_vdso_hres_capable +static inline bool __arch_vdso_hres_capable (void ) +{ +return true; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)@@ -101,6 +108,10 @@ static __always_inline int do_hres(constu64 cycles, last, sec, ns;u32 seq; +/* Allows to compile the high resolution parts out */ +if (!__arch_vdso_hres_capable ()) +return -1 ; + do { /* * Open coded to handle VCLOCK_TIMENS. Time namespace
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] lib/vdso: Allow the high resolution parts to be compiled out
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 1dff4156d1f63b525c54aea7f097a657cbbbf837 Gitweb: https://git.kernel.org/tip/1dff4156d1f63b525c54aea7f097a657cbbbf837 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:50 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:20 +01:00
lib/vdso: Allow the high resolution parts to be compiled out
If the architecture knows at compile time that there is no VDSO capable clocksource supported it makes sense to optimize the guts of the high resolution parts of the VDSO out at build time. Add a helper function to check whether the VDSO should be high resolution capable and provide a stub which can be overridden by an architecture.
--- lib/vdso/gettimeofday.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
lib/vdso/gettimeofday.c +11 -0
@@ -38,6 +38,13 @@ u64 vdso_calc_delta(u64 cycles, u64 last, u64 mask, u32 mult) }#endif +#ifndef __arch_vdso_hres_capable +static inline bool __arch_vdso_hres_capable (void ) +{ +return true; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)
@@ -101,6 +108,10 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk,u64 cycles, last, sec, ns;u32 seq; +/* Allows to compile the high resolution parts out */ +if (!__arch_vdso_hres_capable ()) +return -1 ; + do { /* * Open coded to handle VCLOCK_TIMENS. Time namespace
PATCH v2 4/17 ARM: vdso: Compile high resolution parts conditionally
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
If the architected timer is disabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
--- arch/arm/include/asm/vdso/gettimeofday.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm/include/asm/vdso/gettimeofday.h +6 -0
@@ -106,6 +106,12 @@ static __always_inline int clock_getres3return ret; } +static inline bool arm_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_ARM_ARCH_TIMER ); +} +#define __arch_vdso_hres_capable arm_vdso_hres_capable + static __always_inline u64 __arch_get_hw_counter(int clock_mode) {#ifdef CONFIG_ARM_ARCH_TIMER
Re: [patch V2 04/17] ARM: vdso: Compile high resolution parts conditionally
Quoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
If the architected timer is disabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 20 lines
--- arch/arm/include/asm/vdso/gettimeofday.h | 6 ++++++ 1 file changed, 6 insertions(+)arch/arm/include/asm/vdso/gettimeofday.h +6
-0@@ -106,6 +106,12 @@ static __always_inline int clock_getres3return ret; } +static inline bool arm_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_ARM_ARCH_TIMER ); +} +#define __arch_vdso_hres_capable arm_vdso_hres_capable + static __always_inline u64 __arch_get_hw_counter(int clock_mode) {#ifdef CONFIG_ARM_ARCH_TIMER
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] ARM: vdso: Compile high resolution parts conditionally
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 3280badbe1b289622ce12b94d064ddb624cbaef1 Gitweb: https://git.kernel.org/tip/3280badbe1b289622ce12b94d064ddb624cbaef1 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:51 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:21 +01:00
ARM: vdso: Compile high resolution parts conditionally
If the architected timer is disabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
--- arch/arm/include/asm/vdso/gettimeofday.h | 6 ++++++ 1 file changed, 6 insertions(+)
arch/arm/include/asm/vdso/gettimeofday.h +6 -0
@@ -106,6 +106,12 @@ static __always_inline int clock_getres32_fallback(return ret; } +static inline bool arm_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_ARM_ARCH_TIMER ); +} +#define __arch_vdso_hres_capable arm_vdso_hres_capable + static __always_inline u64 __arch_get_hw_counter(int clock_mode) {#ifdef CONFIG_ARM_ARCH_TIMER
PATCH v2 5/17 MIPS: vdso: Compile high resolution parts conditionally
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
If neither the R4K nor the GIC timer is enabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
--- arch/mips/include/asm/vdso/gettimeofday.h | 7 +++++++ 1 file changed, 7 insertions(+)
arch/mips/include/asm/vdso/gettimeofday.h +7 -0
@@ -199,6 +199,13 @@ static __always_inline u64 __arch_get_hwreturn cycle_now; } +static inline bool mips_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_CSRC_R4K ) || +IS_ENABLED (CONFIG_CLKSRC_MIPS_GIC ); +} +#define __arch_vdso_hres_capable mips_vdso_hres_capable + static __always_inline const struct vdso_data *__arch_get_vdso_data(void) {return get_vdso_data ();
Re: [patch V2 05/17] MIPS: vdso: Compile high resolution parts conditionally
Quoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
If neither the R4K nor the GIC timer is enabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
Signed-off-by Thomas Gleixner <[email protected]>
Quoted 21 lines
--- arch/mips/include/asm/vdso/gettimeofday.h | 7 +++++++ 1 file changed, 7 insertions(+)arch/mips/include/asm/vdso/gettimeofday.h +7
-0@@ -199,6 +199,13 @@ static __always_inline u64 __arch_get_hwreturn cycle_now; } +static inline bool mips_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_CSRC_R4K ) || +IS_ENABLED (CONFIG_CLKSRC_MIPS_GIC ); +} +#define __arch_vdso_hres_capable mips_vdso_hres_capable + static __always_inline const struct vdso_data *__arch_get_vdso_data(void) {return get_vdso_data ();
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] MIPS: vdso: Compile high resolution parts conditionally
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 25a2a6567829119f5e3e11eb0ce3d8ae985b6019 Gitweb: https://git.kernel.org/tip/25a2a6567829119f5e3e11eb0ce3d8ae985b6019 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:52 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:21 +01:00
MIPS: vdso: Compile high resolution parts conditionally
If neither the R4K nor the GIC timer is enabled in the kernel configuration then let the core VDSO code drop the high resolution parts at compile time.
--- arch/mips/include/asm/vdso/gettimeofday.h | 7 +++++++ 1 file changed, 7 insertions(+)
arch/mips/include/asm/vdso/gettimeofday.h +7 -0
@@ -199,6 +199,13 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode)return cycle_now; } +static inline bool mips_vdso_hres_capable (void ) +{ +return IS_ENABLED (CONFIG_CSRC_R4K ) || +IS_ENABLED (CONFIG_CLKSRC_MIPS_GIC ); +} +#define __arch_vdso_hres_capable mips_vdso_hres_capable + static __always_inline const struct vdso_data *__arch_get_vdso_data(void) {return get_vdso_data ();
PATCH v2 6/17 clocksource: Cleanup struct clocksource and documentation
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Reformat the struct definition, add missing member documentation. No functional change.
--- include/linux/clocksource.h | 87 +++++++++++++++++++++++--------------------- 1 file changed, 47 insertions(+), 40 deletions(-)
include/linux/clocksource.h +47 -40
@@ -32,9 +32,19 @@ struct module;* Provides mostly state-free accessors to the underlying hardware. * This is the structure used for system time. * -* @name: ptr to clocksource name -* @list: list head for registration -* @rating: rating value for selection (higher is better) +* @read: Returns a cycle value, passes clocksource as argument +* @mask: Bitmask for two's complement +* subtraction of non 64 bit counters +* @mult: Cycle to nanosecond multiplier +* @shift: Cycle to nanosecond divisor (power of two) +* @max_idle_ns: Maximum idle time permitted by the clocksource (nsecs) +* @maxadj: Maximum adjustment value to mult (~11%) +* @archdata: Optional arch-specific data +* @max_cycles: Maximum safe cycle value which won't overflow on +* multiplication +* @name: Pointer to clocksource name +* @list: List head for registration (internal) +* @rating: Rating value for selection (higher is better) * To avoid rating inflation the following * list should give you a guide as to how * to assign your clocksource a rating
@@ -49,27 +59,23 @@ struct module;* 400-499: Perfect * The ideal clocksource. A must-use where * available. -* @read: returns a cycle value, passes clocksource as argument -* @enable: optional function to enable the clocksource -* @disable: optional function to disable the clocksource -* @mask: bitmask for two's complement -* subtraction of non 64 bit counters -* @mult: cycle to nanosecond multiplier -* @shift: cycle to nanosecond divisor (power of two) -* @max_idle_ns: max idle time permitted by the clocksource (nsecs) -* @maxadj: maximum adjustment value to mult (~11%) -* @max_cycles: maximum safe cycle value which won't overflow on multiplication -* @flags: flags describing special properties -* @archdata: arch-specific data -* @suspend: suspend function for the clocksource, if necessary -* @resume: resume function for the clocksource, if necessary +* @flags: Flags describing special properties +* @enable: Optional function to enable the clocksource +* @disable: Optional function to disable the clocksource +* @suspend: Optional suspend function for the clocksource +* @resume: Optional resume function for the clocksource * @mark_unstable: Optional function to inform the clocksource driver that * the watchdog marked the clocksource unstable -* @owner: module reference, must be set by clocksource in modules +* @tick_stable: Optional function called periodically from the watchdog +* code to provide stable syncrhonization points +* @wd_list: List head to enqueue into the watchdog list (internal) +* @cs_last: Last clocksource value for clocksource watchdog +* @wd_last: Last watchdog value corresponding to @cs_last +* @owner: Module reference, must be set by clocksource in modules * * Note: This struct is not used in hotpathes of the timekeeping code * because the timekeeper caches the hot path fields in its own data -* structure, so no line cache alignment is required, +* structure, so no cache line alignment is required, * * The pointer to the clocksource itself is handed to the read * callback. If you need extra information there you can wrap struct
@@ -78,35 +84,36 @@ struct module;* structure. */ struct clocksource { -u64 (*read)(struct clocksource *cs); -u64 mask; -u32 mult; -u32 shift; -u64 max_idle_ns; -u32 maxadj; +u64 (*read)(struct clocksource *cs); +u64 mask; +u32 mult; +u32 shift; +u64 max_idle_ns; +u32 maxadj;#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA struct arch_clocksource_data archdata;#endif -u64 max_cycles; -const char *name; -struct list_head list; -int rating; -int (*enable)(struct clocksource *cs); -void (*disable)(struct clocksource *cs); -unsigned long flags; -void (*suspend)(struct clocksource *cs); -void (*resume)(struct clocksource *cs); -void (*mark_unstable)(struct clocksource *cs); -void (*tick_stable)(struct clocksource *cs); +u64 max_cycles; +const char *name; +struct list_head list; +int rating; +unsigned long flags; + +int (*enable)(struct clocksource *cs); +void (*disable)(struct clocksource *cs); +void (*suspend)(struct clocksource *cs); +void (*resume)(struct clocksource *cs); +void (*mark_unstable)(struct clocksource *cs); +void (*tick_stable)(struct clocksource *cs);/* private: */ #ifdef CONFIG_CLOCKSOURCE_WATCHDOG /* Watchdog related data, used by the framework */ -struct list_head wd_list; -u64 cs_last; -u64 wd_last; +struct list_head wd_list; +u64 cs_last; +u64 wd_last;#endif -struct module *owner; +struct module *owner; };/*
Re: [patch V2 06/17] clocksource: Cleanup struct clocksource and documentation
Quoted 5 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Reformat the struct definition, add missing member documentation. No functional change.
Quoted 2 lines
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] clocksource: Cleanup struct clocksource and documentation
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 3bd142a46b561a12408e8db78cc6d62eb1c6b84e Gitweb: https://git.kernel.org/tip/3bd142a46b561a12408e8db78cc6d62eb1c6b84e Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:53 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:22 +01:00
clocksource: Cleanup struct clocksource and documentation
Reformat the struct definition, add missing member documentation. No functional change.
--- include/linux/clocksource.h | 87 +++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 40 deletions(-)
include/linux/clocksource.h +47 -40
@@ -32,9 +32,19 @@ struct module;* Provides mostly state-free accessors to the underlying hardware. * This is the structure used for system time. * -* @name: ptr to clocksource name -* @list: list head for registration -* @rating: rating value for selection (higher is better) +* @read: Returns a cycle value, passes clocksource as argument +* @mask: Bitmask for two's complement +* subtraction of non 64 bit counters +* @mult: Cycle to nanosecond multiplier +* @shift: Cycle to nanosecond divisor (power of two) +* @max_idle_ns: Maximum idle time permitted by the clocksource (nsecs) +* @maxadj: Maximum adjustment value to mult (~11%) +* @archdata: Optional arch-specific data +* @max_cycles: Maximum safe cycle value which won't overflow on +* multiplication +* @name: Pointer to clocksource name +* @list: List head for registration (internal) +* @rating: Rating value for selection (higher is better) * To avoid rating inflation the following * list should give you a guide as to how * to assign your clocksource a rating
@@ -49,27 +59,23 @@ struct module;* 400-499: Perfect * The ideal clocksource. A must-use where * available. -* @read: returns a cycle value, passes clocksource as argument -* @enable: optional function to enable the clocksource -* @disable: optional function to disable the clocksource -* @mask: bitmask for two's complement -* subtraction of non 64 bit counters -* @mult: cycle to nanosecond multiplier -* @shift: cycle to nanosecond divisor (power of two) -* @max_idle_ns: max idle time permitted by the clocksource (nsecs) -* @maxadj: maximum adjustment value to mult (~11%) -* @max_cycles: maximum safe cycle value which won't overflow on multiplication -* @flags: flags describing special properties -* @archdata: arch-specific data -* @suspend: suspend function for the clocksource, if necessary -* @resume: resume function for the clocksource, if necessary +* @flags: Flags describing special properties +* @enable: Optional function to enable the clocksource +* @disable: Optional function to disable the clocksource +* @suspend: Optional suspend function for the clocksource +* @resume: Optional resume function for the clocksource * @mark_unstable: Optional function to inform the clocksource driver that * the watchdog marked the clocksource unstable -* @owner: module reference, must be set by clocksource in modules +* @tick_stable: Optional function called periodically from the watchdog +* code to provide stable syncrhonization points +* @wd_list: List head to enqueue into the watchdog list (internal) +* @cs_last: Last clocksource value for clocksource watchdog +* @wd_last: Last watchdog value corresponding to @cs_last +* @owner: Module reference, must be set by clocksource in modules * * Note: This struct is not used in hotpathes of the timekeeping code * because the timekeeper caches the hot path fields in its own data -* structure, so no line cache alignment is required, +* structure, so no cache line alignment is required, * * The pointer to the clocksource itself is handed to the read * callback. If you need extra information there you can wrap struct
@@ -78,35 +84,36 @@ struct module;* structure. */ struct clocksource { -u64 (*read)(struct clocksource *cs); -u64 mask; -u32 mult; -u32 shift; -u64 max_idle_ns; -u32 maxadj; +u64 (*read)(struct clocksource *cs); +u64 mask; +u32 mult; +u32 shift; +u64 max_idle_ns; +u32 maxadj;#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA struct arch_clocksource_data archdata;#endif -u64 max_cycles; -const char *name; -struct list_head list; -int rating; -int (*enable)(struct clocksource *cs); -void (*disable)(struct clocksource *cs); -unsigned long flags; -void (*suspend)(struct clocksource *cs); -void (*resume)(struct clocksource *cs); -void (*mark_unstable)(struct clocksource *cs); -void (*tick_stable)(struct clocksource *cs); +u64 max_cycles; +const char *name; +struct list_head list; +int rating; +unsigned long flags; + +int (*enable)(struct clocksource *cs); +void (*disable)(struct clocksource *cs); +void (*suspend)(struct clocksource *cs); +void (*resume)(struct clocksource *cs); +void (*mark_unstable)(struct clocksource *cs); +void (*tick_stable)(struct clocksource *cs);/* private: */ #ifdef CONFIG_CLOCKSOURCE_WATCHDOG /* Watchdog related data, used by the framework */ -struct list_head wd_list; -u64 cs_last; -u64 wd_last; +struct list_head wd_list; +u64 cs_last; +u64 wd_last;#endif -struct module *owner; +struct module *owner; };/*
PATCH v2 7/17 x86/vdso: Move VDSO clocksource state tracking to callback
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
X86 abuses the function which retrieves the architecture specific clock mode storage to mark the clocksource as used in the VDSO. That's silly because this is invoked on every tick when the VDSO data is updated.
Move this functionality to the clocksource::enable() callback so it gets invoked once when the clocksource is installed. This allows to make the clock mode storage generic.
--- arch/x86/entry/vdso/vma.c | 4 ++++ arch/x86/include/asm/clocksource.h | 12 ++++++++++++ arch/x86/include/asm/mshyperv.h | 2 ++ arch/x86/include/asm/vdso/vsyscall.h | 10 +--------- arch/x86/include/asm/vgtod.h | 6 ------ arch/x86/kernel/kvmclock.c | 7 +++++++ arch/x86/kernel/tsc.c | 32 ++++++++++++++++++++------------ arch/x86/xen/time.c | 17 ++++++++++++----- drivers/clocksource/hyperv_timer.c | 7 +++++++ 9 files changed, 65 insertions(+), 32 deletions(-)
arch/x86/entry/vdso/vma.c +4 -0
@@ -38,6 +38,8 @@ struct vdso_data *arch_get_vdso_data(voi }#undef EMIT_VVAR +unsigned int vclocks_used__read_mostly ; +#if defined(CONFIG_X86_64 )unsigned int __read_mostly vdso64_enabled =1 ;#endif
@@ -445,6 +447,8 @@ static __init int vdso_setup(char *s)static int __init init_vdso (void ) { +BUILD_BUG_ON (VCLOCK_MAX >=32 ); +init_vdso_image (&vdso_image_64);#ifdef CONFIG_X86_X32_ABI
arch/x86/include/asm/clocksource.h +12 -0
@@ -14,4 +14,16 @@ struct arch_clocksource_data { int vclock_mode; }; +extern unsigned int vclocks_used; + +static inline bool vclock_was_used (int vclock) +{ +return READ_ONCE (vclocks_used) & (1U << vclock); +} + +static inline void vclocks_set_used (unsigned int which) +{ +WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << which)); +} +#endif /* _ASM_X86_CLOCKSOURCE_H */
arch/x86/include/asm/mshyperv.h +2 -0
@@ -47,6 +47,8 @@ typedef int (*hyperv_fill_flush_list_fun wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) +#define hv_enable_vdso_clocksource () \ + vclocks_set_used(VCLOCK_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );
arch/x86/include/asm/vdso/vsyscall.h +1 -9
@@ -10,8 +10,6 @@#include <asm/vgtod.h> #include <asm/vvar.h> -int vclocks_used__read_mostly ; - DEFINE_VVAR(struct vdso_data, _vdso_data);/* * Update the vDSO data page to keep in sync with kernel timekeeping.
@@ -26,13 +24,7 @@ struct vdso_data *__x86_get_k_vdso_data(static __always_inline int __x86_get_clock_mode(struct timekeeper *tk) { -int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; - -/* Mark the new vclock used. */ -BUILD_BUG_ON (VCLOCK_MAX >=32 ); -WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << vclock_mode)); - -return vclock_mode; +return tk->tkr_mono.clock->archdata.vclock_mode; }#define __arch_get_clock_mode __x86_get_clock_mode
arch/x86/include/asm/vgtod.h +0 -6
@@ -15,10 +15,4 @@ typedef u64 gtod_long_t;typedef unsigned long gtod_long_t ;#endif -extern int vclocks_used; -static inline bool vclock_was_used (int vclock) -{ -return READ_ONCE (vclocks_used) & (1 << vclock); -} -#endif /* _ASM_X86_VGTOD_H */
arch/x86/kernel/kvmclock.c +7 -0
@@ -159,12 +159,19 @@ bool kvm_check_and_clear_guest_paused(voreturn ret; } +static int kvm_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +struct clocksource kvm_clock = { .name ="kvm-clock" , .read = kvm_clock_get_cycles, .rating =400 , .mask =CLOCKSOURCE_MASK (64 ), .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = kvm_cs_enable, };EXPORT_SYMBOL_GPL (kvm_clock);
arch/x86/kernel/tsc.c +20 -12
@@ -1108,17 +1108,24 @@ static void tsc_cs_tick_stable(struct clsched_clock_tick_stable (); } +static int tsc_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_TSC ); +return 0 ; +} +/* * .mask MUST be CLOCKSOURCE_MASK(64). See comment above read_tsc() */ static struct clocksource clocksource_tsc_early = { - .name ="tsc-early" , - .rating =299 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc-early" , + .rating =299 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,
@@ -1131,14 +1138,15 @@ static struct clocksource clocksource_ts* been found good. */ static struct clocksource clocksource_tsc = { - .name ="tsc" , - .rating =300 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc" , + .rating =300 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,
arch/x86/xen/time.c +12 -5
@@ -145,12 +145,19 @@ static struct notifier_block xen_pvclock .notifier_call = xen_pvclock_gtod_notify, }; +static int xen_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +static struct clocksource xen_clocksource__read_mostly = { - .name ="xen" , - .rating =400 , - .read = xen_clocksource_get_cycles, - .mask = ~0 , - .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .name ="xen" , + .rating =400 , + .read = xen_clocksource_get_cycles, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = xen_cs_enable, };/*
drivers/clocksource/hyperv_timer.c +7 -0
@@ -369,6 +369,12 @@ static void resume_hv_clock_tsc(struct chv_set_reference_tsc (tsc_msr); } +static int hv_cs_enable (struct clocksource *cs) +{ +hv_enable_vdso_clocksource (); +return 0 ; +} +static struct clocksource hyperv_cs_tsc = { .name ="hyperv_clocksource_tsc_page" , .rating =250 ,
@@ -377,6 +383,7 @@ static struct clocksource hyperv_cs_tsc .flags = CLOCK_SOURCE_IS_CONTINUOUS, .suspend= suspend_hv_clock_tsc, .resume = resume_hv_clock_tsc, + .enable = hv_cs_enable, }; static u64 notrace read_hv_clock_msr(void)
Re: [patch V2 07/17] x86/vdso: Move VDSO clocksource state tracking to callback
Quoted 9 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
X86 abuses the function which retrieves the architecture specific clock mode storage to mark the clocksource as used in the VDSO. That's silly because this is invoked on every tick when the VDSO data is updated.
Quoted 8 lines
Move this functionality to the
clocksource::enable()callback so it gets invoked once when the clocksource is installed. This allows to make the clock mode storage generic.Signed-off-by Thomas Gleixner <[email protected]>
Quoted 229 lines
--- arch/x86/entry/vdso/vma.c | 4 ++++ arch/x86/include/asm/clocksource.h | 12 ++++++++++++ arch/x86/include/asm/mshyperv.h | 2 ++ arch/x86/include/asm/vdso/vsyscall.h | 10 +--------- arch/x86/include/asm/vgtod.h | 6 ------ arch/x86/kernel/kvmclock.c | 7 +++++++ arch/x86/kernel/tsc.c | 32 ++++++++++++++++++++------------ arch/x86/xen/time.c | 17 ++++++++++++----- drivers/clocksource/hyperv_timer.c | 7 +++++++ 9 files changed, 65 insertions(+), 32 deletions(-)arch/x86/entry/vdso/vma.c +4
-0@@ -38,6 +38,8 @@ struct vdso_data *arch_get_vdso_data(voi }#undef EMIT_VVAR +unsigned int vclocks_used__read_mostly ; +#if defined(CONFIG_X86_64 )unsigned int __read_mostly vdso64_enabled =1 ;#endif @@ -445,6 +447,8 @@ static __init int vdso_setup(char *s)static int __init init_vdso (void ) { +BUILD_BUG_ON (VCLOCK_MAX >=32 ); +init_vdso_image (&vdso_image_64);#ifdef CONFIG_X86_X32_ABI arch/x86/include/asm/clocksource.h +12
-0@@ -14,4 +14,16 @@ struct arch_clocksource_data { int vclock_mode; }; +extern unsigned int vclocks_used; + +static inline bool vclock_was_used (int vclock) +{ +return READ_ONCE (vclocks_used) & (1U << vclock); +} + +static inline void vclocks_set_used (unsigned int which) +{ +WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << which)); +} +#endif /* _ASM_X86_CLOCKSOURCE_H */ arch/x86/include/asm/mshyperv.h +2
-0@@ -47,6 +47,8 @@ typedef int (*hyperv_fill_flush_list_fun wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) +#define hv_enable_vdso_clocksource () \ + vclocks_set_used(VCLOCK_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );arch/x86/include/asm/vdso/vsyscall.h +1
-9@@ -10,8 +10,6 @@#include <asm/vgtod.h> #include <asm/vvar.h> -int vclocks_used__read_mostly ; - DEFINE_VVAR(struct vdso_data, _vdso_data);/* * Update the vDSO data page to keep in sync with kernel timekeeping. @@ -26,13 +24,7 @@ struct vdso_data *__x86_get_k_vdso_data(static __always_inline int __x86_get_clock_mode(struct timekeeper *tk) { -int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; - -/* Mark the new vclock used. */ -BUILD_BUG_ON (VCLOCK_MAX >=32 ); -WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << vclock_mode)); - -return vclock_mode; +return tk->tkr_mono.clock->archdata.vclock_mode; }#define __arch_get_clock_mode __x86_get_clock_modearch/x86/include/asm/vgtod.h +0
-6@@ -15,10 +15,4 @@ typedef u64 gtod_long_t;typedef unsigned long gtod_long_t ;#endif -extern int vclocks_used; -static inline bool vclock_was_used (int vclock) -{ -return READ_ONCE (vclocks_used) & (1 << vclock); -} -#endif /* _ASM_X86_VGTOD_H */ arch/x86/kernel/kvmclock.c +7
-0@@ -159,12 +159,19 @@ bool kvm_check_and_clear_guest_paused(voreturn ret; } +static int kvm_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +struct clocksource kvm_clock = { .name ="kvm-clock" , .read = kvm_clock_get_cycles, .rating =400 , .mask =CLOCKSOURCE_MASK (64 ), .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = kvm_cs_enable, };EXPORT_SYMBOL_GPL (kvm_clock);arch/x86/kernel/tsc.c +20
-12@@ -1108,17 +1108,24 @@ static void tsc_cs_tick_stable(struct clsched_clock_tick_stable (); } +static int tsc_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_TSC ); +return 0 ; +} +/* * .mask MUST be CLOCKSOURCE_MASK(64). See comment above read_tsc() */ static struct clocksource clocksource_tsc_early = { - .name ="tsc-early" , - .rating =299 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc-early" , + .rating =299 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,@@ -1131,14 +1138,15 @@ static struct clocksource clocksource_ts* been found good. */ static struct clocksource clocksource_tsc = { - .name ="tsc" , - .rating =300 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc" , + .rating =300 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,arch/x86/xen/time.c +12
-5@@ -145,12 +145,19 @@ static struct notifier_block xen_pvclock .notifier_call = xen_pvclock_gtod_notify, }; +static int xen_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +static struct clocksource xen_clocksource__read_mostly = { - .name ="xen" , - .rating =400 , - .read = xen_clocksource_get_cycles, - .mask = ~0 , - .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .name ="xen" , + .rating =400 , + .read = xen_clocksource_get_cycles, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = xen_cs_enable, };/* drivers/clocksource/hyperv_timer.c +7
-0@@ -369,6 +369,12 @@ static void resume_hv_clock_tsc(struct chv_set_reference_tsc (tsc_msr); } +static int hv_cs_enable (struct clocksource *cs) +{ +hv_enable_vdso_clocksource (); +return 0 ; +} +static struct clocksource hyperv_cs_tsc = { .name ="hyperv_clocksource_tsc_page" , .rating =250 ,@@ -377,6 +383,7 @@ static struct clocksource hyperv_cs_tsc .flags = CLOCK_SOURCE_IS_CONTINUOUS, .suspend= suspend_hv_clock_tsc, .resume = resume_hv_clock_tsc, + .enable = hv_cs_enable, }; static u64 notrace read_hv_clock_msr(void)
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] x86/vdso: Move VDSO clocksource state tracking to callback
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: eec399dd862762b9594df3659f15839a4e12f17a Gitweb: https://git.kernel.org/tip/eec399dd862762b9594df3659f15839a4e12f17a Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:54 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:22 +01:00
x86/vdso: Move VDSO clocksource state tracking to callback
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
X86 abuses the function which retrieves the architecture specific clock mode storage to mark the clocksource as used in the VDSO. That's silly because this is invoked on every tick when the VDSO data is updated.
Move this functionality to the clocksource::enable() callback so it gets invoked once when the clocksource is installed. This allows to make the clock mode storage generic.
--- arch/x86/entry/vdso/vma.c | 4 +++- arch/x86/include/asm/clocksource.h | 12 ++++++++++- arch/x86/include/asm/mshyperv.h | 2 ++- arch/x86/include/asm/vdso/vsyscall.h | 10 +-------- arch/x86/include/asm/vgtod.h | 6 +----- arch/x86/kernel/kvmclock.c | 7 ++++++- arch/x86/kernel/tsc.c | 32 ++++++++++++++++----------- arch/x86/xen/time.c | 17 +++++++++----- drivers/clocksource/hyperv_timer.c | 7 ++++++- 9 files changed, 65 insertions(+), 32 deletions(-)
arch/x86/entry/vdso/vma.c +4 -0
@@ -38,6 +38,8 @@ struct vdso_data *arch_get_vdso_data(void *vvar_page) }#undef EMIT_VVAR +unsigned int vclocks_used__read_mostly ; +#if defined(CONFIG_X86_64 )unsigned int __read_mostly vdso64_enabled =1 ;#endif
@@ -445,6 +447,8 @@ __setup("vdso=", vdso_setup);static int __init init_vdso (void ) { +BUILD_BUG_ON (VCLOCK_MAX >=32 ); +init_vdso_image (&vdso_image_64);#ifdef CONFIG_X86_X32_ABI
arch/x86/include/asm/clocksource.h +12 -0
@@ -14,4 +14,16 @@ struct arch_clocksource_data { int vclock_mode; }; +extern unsigned int vclocks_used; + +static inline bool vclock_was_used (int vclock) +{ +return READ_ONCE (vclocks_used) & (1U << vclock); +} + +static inline void vclocks_set_used (unsigned int which) +{ +WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << which)); +} +#endif /* _ASM_X86_CLOCKSOURCE_H */
arch/x86/include/asm/mshyperv.h +2 -0
@@ -47,6 +47,8 @@ typedef int (*hyperv_fill_flush_list_func)( wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) +#define hv_enable_vdso_clocksource () \ + vclocks_set_used(VCLOCK_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );
arch/x86/include/asm/vdso/vsyscall.h +1 -9
@@ -10,8 +10,6 @@#include <asm/vgtod.h> #include <asm/vvar.h> -int vclocks_used__read_mostly ; - DEFINE_VVAR(struct vdso_data, _vdso_data);/* * Update the vDSO data page to keep in sync with kernel timekeeping.
@@ -26,13 +24,7 @@ struct vdso_data *__x86_get_k_vdso_data(void)static __always_inline int __x86_get_clock_mode(struct timekeeper *tk) { -int vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; - -/* Mark the new vclock used. */ -BUILD_BUG_ON (VCLOCK_MAX >=32 ); -WRITE_ONCE (vclocks_used,READ_ONCE (vclocks_used) | (1 << vclock_mode)); - -return vclock_mode; +return tk->tkr_mono.clock->archdata.vclock_mode; }#define __arch_get_clock_mode __x86_get_clock_mode
arch/x86/include/asm/vgtod.h +0 -6
@@ -15,10 +15,4 @@ typedef u64 gtod_long_t;typedef unsigned long gtod_long_t ;#endif -extern int vclocks_used; -static inline bool vclock_was_used (int vclock) -{ -return READ_ONCE (vclocks_used) & (1 << vclock); -} -#endif /* _ASM_X86_VGTOD_H */
arch/x86/kernel/kvmclock.c +7 -0
@@ -159,12 +159,19 @@ bool kvm_check_and_clear_guest_paused(void)return ret; } +static int kvm_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +struct clocksource kvm_clock = { .name ="kvm-clock" , .read = kvm_clock_get_cycles, .rating =400 , .mask =CLOCKSOURCE_MASK (64 ), .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = kvm_cs_enable, };EXPORT_SYMBOL_GPL (kvm_clock);
arch/x86/kernel/tsc.c +20 -12
@@ -1108,17 +1108,24 @@ static void tsc_cs_tick_stable(struct clocksource *cs)sched_clock_tick_stable (); } +static int tsc_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_TSC ); +return 0 ; +} +/* * .mask MUST be CLOCKSOURCE_MASK(64). See comment above read_tsc() */ static struct clocksource clocksource_tsc_early = { - .name ="tsc-early" , - .rating =299 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc-early" , + .rating =299 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,
@@ -1131,14 +1138,15 @@ static struct clocksource clocksource_tsc_early = {* been found good. */ static struct clocksource clocksource_tsc = { - .name ="tsc" , - .rating =300 , - .read = read_tsc, - .mask =CLOCKSOURCE_MASK (64 ), - .flags =CLOCK_SOURCE_IS_CONTINUOUS | + .name ="tsc" , + .rating =300 , + .read = read_tsc, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES |CLOCK_SOURCE_MUST_VERIFY , - .archdata = { .vclock_mode =VCLOCK_TSC }, + .archdata = { .vclock_mode =VCLOCK_TSC }, + .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable, .tick_stable = tsc_cs_tick_stable,
arch/x86/xen/time.c +12 -5
@@ -145,12 +145,19 @@ static struct notifier_block xen_pvclock_gtod_notifier = { .notifier_call = xen_pvclock_gtod_notify, }; +static int xen_cs_enable (struct clocksource *cs) +{ +vclocks_set_used (VCLOCK_PVCLOCK ); +return 0 ; +} +static struct clocksource xen_clocksource__read_mostly = { - .name ="xen" , - .rating =400 , - .read = xen_clocksource_get_cycles, - .mask = ~0 , - .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .name ="xen" , + .rating =400 , + .read = xen_clocksource_get_cycles, + .mask =CLOCKSOURCE_MASK (64 ), + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .enable = xen_cs_enable, };/*
drivers/clocksource/hyperv_timer.c +7 -0
@@ -369,6 +369,12 @@ static void resume_hv_clock_tsc(struct clocksource *arg)hv_set_reference_tsc (tsc_msr); } +static int hv_cs_enable (struct clocksource *cs) +{ +hv_enable_vdso_clocksource (); +return 0 ; +} +static struct clocksource hyperv_cs_tsc = { .name ="hyperv_clocksource_tsc_page" , .rating =250 ,
@@ -377,6 +383,7 @@ static struct clocksource hyperv_cs_tsc = { .flags = CLOCK_SOURCE_IS_CONTINUOUS, .suspend= suspend_hv_clock_tsc, .resume = resume_hv_clock_tsc, + .enable = hv_cs_enable, }; static u64 notrace read_hv_clock_msr(void)
PATCH v2 8/17 clocksource: Add common vdso clock mode storage
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
Provide generic storage for it. The new Kconfig symbol is intermediate and will be removed once all architectures are converted over.
---
V2: Fix the unused variable warning and s/mode/clock_mode/ - Christophe
--- include/linux/clocksource.h | 12 +++++++++++- kernel/time/clocksource.c | 9 +++++++++ kernel/time/vsyscall.c | 10 ++++++++-- lib/vdso/Kconfig | 3 +++ lib/vdso/gettimeofday.c | 13 +++++++++++-- 5 files changed, 42 insertions(+), 5 deletions(-)
include/linux/clocksource.h +11 -1
@@ -23,10 +23,19 @@struct clocksource ;struct module ; -#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA +#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ + defined(CONFIG_GENERIC_VDSO_CLOCK_MODE )#include <asm/clocksource.h> #endif +enum vdso_clock_mode { +VDSO_CLOCKMODE_NONE , +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +VDSO_ARCH_CLOCKMODES , +#endif +VDSO_CLOCKMODE_MAX , +}; +/** * struct clocksource - hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware.
@@ -97,6 +106,7 @@ struct clocksource {const char *name;struct list_head list;int rating; +enum vdso_clock_mode vdso_clock_mode;unsigned long flags;int (*enable)(struct clocksource *cs);
kernel/time/clocksource.c +9 -0
@@ -928,6 +928,15 @@ int __clocksource_register_scale(structclocksource_arch_init (cs); +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +if (cs->vdso_clock_mode <0 || + cs->vdso_clock_mode >=VDSO_CLOCKMODE_MAX ) { +pr_warn ("clocksource %s registered with invalid VDSO mode %d. Disabling VDSO support.\n" , + cs->name, cs->vdso_clock_mode); + cs->vdso_clock_mode =VDSO_CLOCKMODE_NONE ; + } +#endif +/* Initialize mult/shift and max_idle_ns */ __clocksource_update_freq_scale (cs, scale, freq);
kernel/time/vsyscall.c +8 -2
@@ -71,13 +71,19 @@ void update_vsyscall(struct timekeeper * {struct vdso_data *vdata =__arch_get_k_vdso_data ();struct vdso_timestamp *vdso_ts; +s32 clock_mode;u64 nsec;/* copy vsyscall data */ vdso_write_begin (vdata); - vdata[CS_HRES_COARSE ].clock_mode =__arch_get_clock_mode (tk); - vdata[CS_RAW ].clock_mode =__arch_get_clock_mode (tk); +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE + clock_mode = tk->tkr_mono.clock->vdso_clock_mode; +#else + clock_mode =__arch_get_clock_mode (tk); +#endif + vdata[CS_HRES_COARSE ].clock_mode = clock_mode; + vdata[CS_RAW ].clock_mode = clock_mode;/* CLOCK_REALTIME also required for time() */ vdso_ts = &vdata[CS_HRES_COARSE ].basetime[CLOCK_REALTIME ];
lib/vdso/Kconfig +3 -0
@@ -30,4 +30,7 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO +config GENERIC_VDSO_CLOCK_MODE + bool + endif
lib/vdso/gettimeofday.c +11 -2
@@ -7,6 +7,7 @@#include <linux/time.h> #include <linux/kernel.h> #include <linux/hrtimer_defs.h> +#include <linux/clocksource.h> #include <vdso/datapage.h> #include <vdso/helpers.h>
@@ -64,10 +65,14 @@ static int do_hres_timens(const struct v do { seq = vdso_read_begin(vd); + if (IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && + vd->clock_mode == VDSO_CLOCKMODE_NONE) + return -1; cycles = __arch_get_hw_counter(vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; - if (unlikely((s64)cycles < 0)) + if (!IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && + unlikely((s64)cycles < 0)) return -1; ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult);
@@ -132,10 +137,14 @@ static __always_inline int do_hres(const }smp_rmb (); +if (IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && + vd->clock_mode ==VDSO_CLOCKMODE_NONE ) +return -1 ; cycles =__arch_get_hw_counter (vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; -if (unlikely ((s64 )cycles <0 )) +if (!IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && +unlikely ((s64 )cycles <0 ))return -1 ; ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult);
Re: [patch V2 08/17] clocksource: Add common vdso clock mode storage
Hi Thomas,
Quoted 9 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
Provide generic storage for it. The new
Kconfigsymbol is intermediate and will be removed once all architectures are converted over.Signed-off-by Thomas Gleixner <[email protected]>
Quoted 20 lines
---
V2: Fix the unused variable warning and s/mode/clock_mode/ - Christophe--- include/linux/clocksource.h | 12 +++++++++++- kernel/time/clocksource.c | 9 +++++++++ kernel/time/vsyscall.c | 10 ++++++++-- lib/vdso/Kconfig | 3 +++ lib/vdso/gettimeofday.c | 13 +++++++++++-- 5 files changed, 42 insertions(+), 5 deletions(-)include/linux/clocksource.h +2
-1@@ -23,10 +23,19 @@struct clocksource ;struct module ; -#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA +#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ + defined(CONFIG_GENERIC_VDSO_CLOCK_MODE )#include <asm/clocksource.h>
include/linux/clocksource.h line 38
Quoted 10 lines
include/linux/clocksource.h +8
-0#endif +enum vdso_clock_mode { +VDSO_CLOCKMODE_NONE , +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +VDSO_ARCH_CLOCKMODES , +#endif +VDSO_CLOCKMODE_MAX , +}; +
As part of my work on common headers I will move this enumeration in common/clocksource.h to be accessible by the vdso library. It will appear in the next version of my patches once I rebase on yours.
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] clocksource: Add common vdso clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 5d51bee725cc1497352d6b0b604e42a90c680540 Gitweb: https://git.kernel.org/tip/5d51bee725cc1497352d6b0b604e42a90c680540 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:55 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:23 +01:00
clocksource: Add common vdso clock mode storage
All architectures which use the generic VDSO code have their own storage for the VDSO clock mode. That's pointless and just requires duplicate code.
Provide generic storage for it. The new Kconfig symbol is intermediate and will be removed once all architectures are converted over.
--- include/linux/clocksource.h | 12 +++++++++++- kernel/time/clocksource.c | 9 +++++++++ kernel/time/vsyscall.c | 10 ++++++++-- lib/vdso/Kconfig | 3 +++ lib/vdso/gettimeofday.c | 13 +++++++++++-- 5 files changed, 42 insertions(+), 5 deletions(-)
include/linux/clocksource.h +11 -1
@@ -23,10 +23,19 @@struct clocksource ;struct module ; -#ifdef CONFIG_ARCH_CLOCKSOURCE_DATA +#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ + defined(CONFIG_GENERIC_VDSO_CLOCK_MODE )#include <asm/clocksource.h> #endif +enum vdso_clock_mode { +VDSO_CLOCKMODE_NONE , +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +VDSO_ARCH_CLOCKMODES , +#endif +VDSO_CLOCKMODE_MAX , +}; +/** * struct clocksource - hardware abstraction for a free running counter * Provides mostly state-free accessors to the underlying hardware.
@@ -97,6 +106,7 @@ struct clocksource {const char *name;struct list_head list;int rating; +enum vdso_clock_mode vdso_clock_mode;unsigned long flags;int (*enable)(struct clocksource *cs);
kernel/time/clocksource.c +9 -0
@@ -928,6 +928,15 @@ int __clocksource_register_scale(struct clocksource *cs, u32 scale, u32 freq)clocksource_arch_init (cs); +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +if (cs->vdso_clock_mode <0 || + cs->vdso_clock_mode >=VDSO_CLOCKMODE_MAX ) { +pr_warn ("clocksource %s registered with invalid VDSO mode %d. Disabling VDSO support.\n" , + cs->name, cs->vdso_clock_mode); + cs->vdso_clock_mode =VDSO_CLOCKMODE_NONE ; + } +#endif +/* Initialize mult/shift and max_idle_ns */ __clocksource_update_freq_scale (cs, scale, freq);
kernel/time/vsyscall.c +8 -2
@@ -71,13 +71,19 @@ void update_vsyscall(struct timekeeper *tk) {struct vdso_data *vdata =__arch_get_k_vdso_data ();struct vdso_timestamp *vdso_ts; +s32 clock_mode;u64 nsec;/* copy vsyscall data */ vdso_write_begin (vdata); - vdata[CS_HRES_COARSE ].clock_mode =__arch_get_clock_mode (tk); - vdata[CS_RAW ].clock_mode =__arch_get_clock_mode (tk); +#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE + clock_mode = tk->tkr_mono.clock->vdso_clock_mode; +#else + clock_mode =__arch_get_clock_mode (tk); +#endif + vdata[CS_HRES_COARSE ].clock_mode = clock_mode; + vdata[CS_RAW ].clock_mode = clock_mode;/* CLOCK_REALTIME also required for time() */ vdso_ts = &vdata[CS_HRES_COARSE ].basetime[CLOCK_REALTIME ];
lib/vdso/Kconfig +3 -0
@@ -30,4 +30,7 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO +config GENERIC_VDSO_CLOCK_MODE + bool + endif
lib/vdso/gettimeofday.c +11 -2
@@ -7,6 +7,7 @@#include <linux/time.h> #include <linux/kernel.h> #include <linux/hrtimer_defs.h> +#include <linux/clocksource.h> #include <vdso/datapage.h> #include <vdso/helpers.h>
@@ -64,10 +65,14 @@ static int do_hres_timens(const struct vdso_data *vdns, clockid_t clk, do { seq = vdso_read_begin(vd); + if (IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && + vd->clock_mode == VDSO_CLOCKMODE_NONE) + return -1; cycles = __arch_get_hw_counter(vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; - if (unlikely((s64)cycles < 0)) + if (!IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && + unlikely((s64)cycles < 0)) return -1; ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult);
@@ -132,10 +137,14 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, }smp_rmb (); +if (IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && + vd->clock_mode ==VDSO_CLOCKMODE_NONE ) +return -1 ; cycles =__arch_get_hw_counter (vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; -if (unlikely ((s64 )cycles <0 )) +if (!IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && +unlikely ((s64 )cycles <0 ))return -1 ; ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult);
PATCH v2 9/17 x86/vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Switch to the generic VDSO clock mode storage.
--- arch/x86/Kconfig | 2 +- arch/x86/entry/vdso/vma.c | 6 +++--- arch/x86/include/asm/clocksource.h | 13 ++++--------- arch/x86/include/asm/mshyperv.h | 4 ++-- arch/x86/include/asm/vdso/gettimeofday.h | 6 +++--- arch/x86/include/asm/vdso/vsyscall.h | 7 ------- arch/x86/kernel/kvmclock.c | 4 ++-- arch/x86/kernel/pvclock.c | 2 +- arch/x86/kernel/time.c | 12 +++--------- arch/x86/kernel/tsc.c | 6 +++--- arch/x86/kvm/trace.h | 4 ++-- arch/x86/kvm/x86.c | 22 +++++++++++----------- arch/x86/xen/time.c | 21 +++++++++++---------- 13 files changed, 46 insertions(+), 63 deletions(-)
arch/x86/Kconfig +1 -1
@@ -57,7 +57,6 @@ config X86 select ACPI_LEGACY_TABLES_LOOKUP if ACPI select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ARCH_32BIT_OFF_T if X86_32 - select ARCH_CLOCKSOURCE_DATA select ARCH_CLOCKSOURCE_INIT select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL
@@ -126,6 +125,7 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
arch/x86/entry/vdso/vma.c +3 -3
@@ -221,7 +221,7 @@ static vm_fault_t vvar_fault(const struc }else if (sym_offset == image->sym_pvclock_page) {struct pvclock_vsyscall_time_info *pvti =pvclock_get_pvti_cpu0_va (); -if (pvti &&vclock_was_used (VCLOCK_PVCLOCK )) { +if (pvti &&vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )) {return vmf_insert_pfn_prot (vma, vmf->address,__pa (pvti) >>PAGE_SHIFT ,pgprot_decrypted (vma->vm_page_prot));
@@ -229,7 +229,7 @@ static vm_fault_t vvar_fault(const struc }else if (sym_offset == image->sym_hvclock_page) {struct ms_hyperv_tsc_page *tsc_pg =hv_get_tsc_page (); -if (tsc_pg &&vclock_was_used (VCLOCK_HVCLOCK )) +if (tsc_pg &&vclock_was_used (VDSO_CLOCKMODE_HVCLOCK ))return vmf_insert_pfn (vma, vmf->address,virt_to_phys (tsc_pg) >>PAGE_SHIFT ); }else if (sym_offset == image->sym_timens_page) {
@@ -447,7 +447,7 @@ static __init int vdso_setup(char *s)static int __init init_vdso (void ) { -BUILD_BUG_ON (VCLOCK_MAX >=32 ); +BUILD_BUG_ON (VDSO_CLOCKMODE_MAX >=32 );init_vdso_image (&vdso_image_64);
arch/x86/include/asm/clocksource.h +4 -9
@@ -4,15 +4,10 @@#ifndef _ASM_X86_CLOCKSOURCE_H#define _ASM_X86_CLOCKSOURCE_H -#define VCLOCK_NONE 0/* No vDSO clock available. */ -#define VCLOCK_TSC 1/* vDSO should use vread_tsc. */ -#define VCLOCK_PVCLOCK 2/* vDSO should use vread_pvclock. */ -#define VCLOCK_HVCLOCK 3/* vDSO should use vread_hvclock. */ -#define VCLOCK_MAX 3 - -struct arch_clocksource_data { -int vclock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_TSC, \ + VDSO_CLOCKMODE_PVCLOCK, \ + VDSO_CLOCKMODE_HVCLOCKextern unsigned int vclocks_used;
arch/x86/include/asm/mshyperv.h +2 -2
@@ -46,9 +46,9 @@ typedef int (*hyperv_fill_flush_list_fun#define hv_set_reference_tsc (val) \ wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ - ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) + ((val).vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK)#define hv_enable_vdso_clocksource () \ - vclocks_set_used(VCLOCK_HVCLOCK); + vclocks_set_used(VDSO_CLOCKMODE_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );
arch/x86/include/asm/vdso/gettimeofday.h +3 -3
@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (likely (clock_mode ==VCLOCK_TSC )) +if (likely (clock_mode ==VDSO_CLOCKMODE_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc
@@ -252,13 +252,13 @@ static inline u64 __arch_get_hw_counter(* question isn't enabled, which will segfault. Hence the barriers. */ #ifdef CONFIG_PARAVIRT_CLOCK -if (clock_mode ==VCLOCK_PVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_PVCLOCK ) {barrier ();return vread_pvclock (); }#endif #ifdef CONFIG_HYPERV_TIMER -if (clock_mode ==VCLOCK_HVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_HVCLOCK ) {barrier ();return vread_hvclock (); }
arch/x86/include/asm/vdso/vsyscall.h +0 -7
@@ -21,13 +21,6 @@ struct vdso_data *__x86_get_k_vdso_data( }#define __arch_get_k_vdso_data __x86_get_k_vdso_data -static __always_inline -int __x86_get_clock_mode(struct timekeeper *tk) -{ -return tk->tkr_mono.clock->archdata.vclock_mode; -} -#define __arch_get_clock_mode __x86_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h>
arch/x86/kernel/kvmclock.c +2 -2
@@ -161,7 +161,7 @@ bool kvm_check_and_clear_guest_paused(vostatic int kvm_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }
@@ -279,7 +279,7 @@ static int __init kvm_setup_vsyscall_timif (!(flags &PVCLOCK_TSC_STABLE_BIT ))return 0 ; - kvm_clock.archdata.vclock_mode =VCLOCK_PVCLOCK ; + kvm_clock.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ;#endif kvmclock_init_mem ();
arch/x86/kernel/pvclock.c +1 -1
@@ -145,7 +145,7 @@ void pvclock_read_wallclock(struct pvclovoid pvclock_set_pvti_cpu0_va (struct pvclock_vsyscall_time_info *pvti) { -WARN_ON (vclock_was_used (VCLOCK_PVCLOCK )); +WARN_ON (vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )); pvti_cpu0_va = pvti; }
arch/x86/kernel/time.c +3 -9
@@ -114,18 +114,12 @@ void __init time_init(void)*/ void clocksource_arch_init (struct clocksource *cs) { -if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +if (cs->vdso_clock_mode ==VDSO_CLOCKMODE_NONE )return ; -if (cs->archdata.vclock_mode >VCLOCK_MAX ) { -pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , - cs->name, cs->archdata.vclock_mode); - cs->archdata.vclock_mode =VCLOCK_NONE ; - } -if (cs->mask !=CLOCKSOURCE_MASK (64 )) { -pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , +pr_warn ("clocksource %s registered with invalid mask %016llx for VDSO. Disabling VDSO support.\n" , cs->name, cs->mask); - cs->archdata.vclock_mode =VCLOCK_NONE ; + cs->vdso_clock_mode =VDSO_CLOCKMODE_NONE ; } }
arch/x86/kernel/tsc.c +3 -3
@@ -1110,7 +1110,7 @@ static void tsc_cs_tick_stable(struct clstatic int tsc_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_TSC ); +vclocks_set_used (VDSO_CLOCKMODE_TSC );return 0 ; }
@@ -1124,7 +1124,7 @@ static struct clocksource clocksource_ts .mask = CLOCKSOURCE_MASK(64), .flags =CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,
@@ -1145,7 +1145,7 @@ static struct clocksource clocksource_ts .flags = CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,
arch/x86/kvm/trace.h +2 -2
@@ -815,8 +815,8 @@ TRACE_EVENT(kvm_write_tsc_offset,#ifdef CONFIG_X86_64 #define host_clocks \ - {VCLOCK_NONE, "none"}, \ - {VCLOCK_TSC, "tsc"} \ + {VDSO_CLOCKMODE_NONE, "none"}, \ + {VDSO_CLOCKMODE_TSC, "tsc"} \ TRACE_EVENT(kvm_update_master_clock, TP_PROTO(bool use_master_clock, unsigned int host_clock, bool offset_matched),
arch/x86/kvm/x86.c +11 -11
@@ -1635,7 +1635,7 @@ static void update_pvclock_gtod(struct twrite_seqcount_begin (&vdata->seq);/* copy pvclock gtod data */ - vdata->clock.vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; + vdata->clock.vclock_mode = tk->tkr_mono.clock->vdso_clock_mode; vdata->clock.cycle_last = tk->tkr_mono.cycle_last; vdata->clock.mask = tk->tkr_mono.mask; vdata->clock.mult = tk->tkr_mono.mult;
@@ -1643,7 +1643,7 @@ static void update_pvclock_gtod(struct t vdata->clock.base_cycles = tk->tkr_mono.xtime_nsec; vdata->clock.offset = tk->tkr_mono.base; - vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->archdata.vclock_mode; + vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->vdso_clock_mode; vdata->raw_clock.cycle_last = tk->tkr_raw.cycle_last; vdata->raw_clock.mask = tk->tkr_raw.mask; vdata->raw_clock.mult = tk->tkr_raw.mult;
@@ -1844,7 +1844,7 @@ static u64 compute_guest_tsc(struct kvm_static inline int gtod_is_based_on_tsc (int mode) { -return mode ==VCLOCK_TSC || mode ==VCLOCK_HVCLOCK ; +return mode ==VDSO_CLOCKMODE_TSC || mode ==VDSO_CLOCKMODE_HVCLOCK ; }static void kvm_track_tsc_matching (struct kvm_vcpu *vcpu)
@@ -1937,7 +1937,7 @@ static inline bool kvm_check_tsc_unstabl* TSC is marked unstable when we're running on Hyper-V, * 'TSC page' clocksource is good. */ -if (pvclock_gtod_data.clock.vclock_mode ==VCLOCK_HVCLOCK ) +if (pvclock_gtod_data.clock.vclock_mode ==VDSO_CLOCKMODE_HVCLOCK )return false;#endif return check_tsc_unstable ();
@@ -2092,30 +2092,30 @@ static inline u64 vgettsc(struct pvclocku64 tsc_pg_val;switch (clock->vclock_mode) { -case VCLOCK_HVCLOCK : +case VDSO_CLOCKMODE_HVCLOCK : tsc_pg_val =hv_read_tsc_page_tsc (hv_get_tsc_page (), tsc_timestamp);if (tsc_pg_val !=U64_MAX ) {/* TSC page valid */ - *mode =VCLOCK_HVCLOCK ; + *mode =VDSO_CLOCKMODE_HVCLOCK ; v = (tsc_pg_val - clock->cycle_last) & clock->mask; }else {/* TSC page invalid */ - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; }break ; -case VCLOCK_TSC : - *mode =VCLOCK_TSC ; +case VDSO_CLOCKMODE_TSC : + *mode =VDSO_CLOCKMODE_TSC ; *tsc_timestamp =read_tsc (); v = (*tsc_timestamp - clock->cycle_last) & clock->mask;break ;default : - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; } -if (*mode ==VCLOCK_NONE ) +if (*mode ==VDSO_CLOCKMODE_NONE ) *tsc_timestamp = v =0 ;return v * clock->mult;
arch/x86/xen/time.c +11 -10
@@ -147,7 +147,7 @@ static struct notifier_block xen_pvclockstatic int xen_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }
@@ -419,12 +419,13 @@ void xen_restore_time_memory_area(void) ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);/* -* We don't disable VCLOCK_PVCLOCK entirely if it fails to register the -* secondary time info with Xen or if we migrated to a host without the -* necessary flags. On both of these cases what happens is either -* process seeing a zeroed out pvti or seeing no PVCLOCK_TSC_STABLE_BIT -* bit set. Userspace checks the latter and if 0, it discards the data -* in pvti and fallbacks to a system call for a reliable timestamp. +* We don't disable VDSO_CLOCKMODE_PVCLOCK entirely if it fails to +* register the secondary time info with Xen or if we migrated to a +* host without the necessary flags. On both of these cases what +* happens is either process seeing a zeroed out pvti or seeing no +* PVCLOCK_TSC_STABLE_BIT bit set. Userspace checks the latter and +* if 0, it discards the data in pvti and fallbacks to a system +* call for a reliable timestamp. */ if (ret !=0 ) pr_notice("Cannot restore secondary vcpu_time_info (err %d)",
@@ -450,7 +451,7 @@ static void xen_setup_vsyscall_time_info ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);if (ret) { -pr_notice ("xen: VCLOCK_PVCLOCK not supported (err %d)\n" , ret); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (err %d)\n" , ret);free_page ((unsigned long )ti);return ; }
@@ -467,14 +468,14 @@ static void xen_setup_vsyscall_time_infoif (!ret)free_page ((unsigned long )ti); -pr_notice ("xen: VCLOCK_PVCLOCK not supported (tsc unstable)\n" ); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (tsc unstable)\n" );return ; } xen_clock = ti;pvclock_set_pvti_cpu0_va (xen_clock); - xen_clocksource.archdata.vclock_mode =VCLOCK_PVCLOCK ; + xen_clocksource.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ; }static void __init xen_time_init (void )
Re: [patch V2 09/17] x86/vdso: Use generic VDSO clock mode storage
Quoted 6 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Switch to the generic VDSO clock mode storage.
Signed-off-by Thomas Gleixner <[email protected]>
Thanks,
Paolo
Quoted 373 lines
--- arch/x86/Kconfig | 2 +- arch/x86/entry/vdso/vma.c | 6 +++--- arch/x86/include/asm/clocksource.h | 13 ++++--------- arch/x86/include/asm/mshyperv.h | 4 ++-- arch/x86/include/asm/vdso/gettimeofday.h | 6 +++--- arch/x86/include/asm/vdso/vsyscall.h | 7 ------- arch/x86/kernel/kvmclock.c | 4 ++-- arch/x86/kernel/pvclock.c | 2 +- arch/x86/kernel/time.c | 12 +++--------- arch/x86/kernel/tsc.c | 6 +++--- arch/x86/kvm/trace.h | 4 ++-- arch/x86/kvm/x86.c | 22 +++++++++++----------- arch/x86/xen/time.c | 21 +++++++++++---------- 13 files changed, 46 insertions(+), 63 deletions(-)arch/x86/Kconfig +1
-1@@ -57,7 +57,6 @@ config X86 select ACPI_LEGACY_TABLES_LOOKUP if ACPI select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ARCH_32BIT_OFF_T if X86_32 - select ARCH_CLOCKSOURCE_DATA select ARCH_CLOCKSOURCE_INIT select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL@@ -126,6 +125,7 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64arch/x86/entry/vdso/vma.c +3
-3@@ -221,7 +221,7 @@ static vm_fault_t vvar_fault(const struc }else if (sym_offset == image->sym_pvclock_page) {struct pvclock_vsyscall_time_info *pvti =pvclock_get_pvti_cpu0_va (); -if (pvti &&vclock_was_used (VCLOCK_PVCLOCK )) { +if (pvti &&vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )) {return vmf_insert_pfn_prot (vma, vmf->address,__pa (pvti) >>PAGE_SHIFT ,pgprot_decrypted (vma->vm_page_prot));@@ -229,7 +229,7 @@ static vm_fault_t vvar_fault(const struc }else if (sym_offset == image->sym_hvclock_page) {struct ms_hyperv_tsc_page *tsc_pg =hv_get_tsc_page (); -if (tsc_pg &&vclock_was_used (VCLOCK_HVCLOCK )) +if (tsc_pg &&vclock_was_used (VDSO_CLOCKMODE_HVCLOCK ))return vmf_insert_pfn (vma, vmf->address,virt_to_phys (tsc_pg) >>PAGE_SHIFT ); }else if (sym_offset == image->sym_timens_page) {@@ -447,7 +447,7 @@ static __init int vdso_setup(char *s)static int __init init_vdso (void ) { -BUILD_BUG_ON (VCLOCK_MAX >=32 ); +BUILD_BUG_ON (VDSO_CLOCKMODE_MAX >=32 );init_vdso_image (&vdso_image_64);arch/x86/include/asm/clocksource.h +4
-9@@ -4,15 +4,10 @@#ifndef _ASM_X86_CLOCKSOURCE_H#define _ASM_X86_CLOCKSOURCE_H -#define VCLOCK_NONE 0/* No vDSO clock available. */ -#define VCLOCK_TSC 1/* vDSO should use vread_tsc. */ -#define VCLOCK_PVCLOCK 2/* vDSO should use vread_pvclock. */ -#define VCLOCK_HVCLOCK 3/* vDSO should use vread_hvclock. */ -#define VCLOCK_MAX 3 - -struct arch_clocksource_data { -int vclock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_TSC, \ + VDSO_CLOCKMODE_PVCLOCK, \ + VDSO_CLOCKMODE_HVCLOCKextern unsigned int vclocks_used;arch/x86/include/asm/mshyperv.h +2
-2@@ -46,9 +46,9 @@ typedef int (*hyperv_fill_flush_list_fun#define hv_set_reference_tsc (val) \ wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ - ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) + ((val).vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK)#define hv_enable_vdso_clocksource () \ - vclocks_set_used(VCLOCK_HVCLOCK); + vclocks_set_used(VDSO_CLOCKMODE_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );arch/x86/include/asm/vdso/gettimeofday.h +3
-3@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (likely (clock_mode ==VCLOCK_TSC )) +if (likely (clock_mode ==VDSO_CLOCKMODE_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc @@ -252,13 +252,13 @@ static inline u64 __arch_get_hw_counter(* question isn't enabled, which will segfault. Hence the barriers. */ #ifdef CONFIG_PARAVIRT_CLOCK -if (clock_mode ==VCLOCK_PVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_PVCLOCK ) {barrier ();return vread_pvclock (); }#endif #ifdef CONFIG_HYPERV_TIMER -if (clock_mode ==VCLOCK_HVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_HVCLOCK ) {barrier ();return vread_hvclock (); }arch/x86/include/asm/vdso/vsyscall.h +0
-7@@ -21,13 +21,6 @@ struct vdso_data *__x86_get_k_vdso_data( }#define __arch_get_k_vdso_data __x86_get_k_vdso_data -static __always_inline -int __x86_get_clock_mode(struct timekeeper *tk) -{ -return tk->tkr_mono.clock->archdata.vclock_mode; -} -#define __arch_get_clock_mode __x86_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h> arch/x86/kernel/kvmclock.c +2
-2@@ -161,7 +161,7 @@ bool kvm_check_and_clear_guest_paused(vostatic int kvm_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }@@ -279,7 +279,7 @@ static int __init kvm_setup_vsyscall_timif (!(flags &PVCLOCK_TSC_STABLE_BIT ))return 0 ; - kvm_clock.archdata.vclock_mode =VCLOCK_PVCLOCK ; + kvm_clock.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ;#endif kvmclock_init_mem ();arch/x86/kernel/pvclock.c +1
-1@@ -145,7 +145,7 @@ void pvclock_read_wallclock(struct pvclovoid pvclock_set_pvti_cpu0_va (struct pvclock_vsyscall_time_info *pvti) { -WARN_ON (vclock_was_used (VCLOCK_PVCLOCK )); +WARN_ON (vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )); pvti_cpu0_va = pvti; }arch/x86/kernel/time.c +3
-9@@ -114,18 +114,12 @@ void __init time_init(void)*/ void clocksource_arch_init (struct clocksource *cs) { -if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +if (cs->vdso_clock_mode ==VDSO_CLOCKMODE_NONE )return ; -if (cs->archdata.vclock_mode >VCLOCK_MAX ) { -pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , - cs->name, cs->archdata.vclock_mode); - cs->archdata.vclock_mode =VCLOCK_NONE ; - } -if (cs->mask !=CLOCKSOURCE_MASK (64 )) { -pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , +pr_warn ("clocksource %s registered with invalid mask %016llx for VDSO. Disabling VDSO support.\n" , cs->name, cs->mask); - cs->archdata.vclock_mode =VCLOCK_NONE ; + cs->vdso_clock_mode =VDSO_CLOCKMODE_NONE ; } }arch/x86/kernel/tsc.c +3
-3@@ -1110,7 +1110,7 @@ static void tsc_cs_tick_stable(struct clstatic int tsc_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_TSC ); +vclocks_set_used (VDSO_CLOCKMODE_TSC );return 0 ; }@@ -1124,7 +1124,7 @@ static struct clocksource clocksource_ts .mask = CLOCKSOURCE_MASK(64), .flags =CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,@@ -1145,7 +1145,7 @@ static struct clocksource clocksource_ts .flags = CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,arch/x86/kvm/trace.h +2
-2@@ -815,8 +815,8 @@ TRACE_EVENT(kvm_write_tsc_offset,#ifdef CONFIG_X86_64 #define host_clocks \ - {VCLOCK_NONE, "none"}, \ - {VCLOCK_TSC, "tsc"} \ + {VDSO_CLOCKMODE_NONE, "none"}, \ + {VDSO_CLOCKMODE_TSC, "tsc"} \ TRACE_EVENT(kvm_update_master_clock, TP_PROTO(bool use_master_clock, unsigned int host_clock, bool offset_matched),arch/x86/kvm/x86.c +11
-11@@ -1635,7 +1635,7 @@ static void update_pvclock_gtod(struct twrite_seqcount_begin (&vdata->seq);/* copy pvclock gtod data */ - vdata->clock.vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; + vdata->clock.vclock_mode = tk->tkr_mono.clock->vdso_clock_mode; vdata->clock.cycle_last = tk->tkr_mono.cycle_last; vdata->clock.mask = tk->tkr_mono.mask; vdata->clock.mult = tk->tkr_mono.mult;@@ -1643,7 +1643,7 @@ static void update_pvclock_gtod(struct t vdata->clock.base_cycles = tk->tkr_mono.xtime_nsec; vdata->clock.offset = tk->tkr_mono.base; - vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->archdata.vclock_mode; + vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->vdso_clock_mode; vdata->raw_clock.cycle_last = tk->tkr_raw.cycle_last; vdata->raw_clock.mask = tk->tkr_raw.mask; vdata->raw_clock.mult = tk->tkr_raw.mult;@@ -1844,7 +1844,7 @@ static u64 compute_guest_tsc(struct kvm_static inline int gtod_is_based_on_tsc (int mode) { -return mode ==VCLOCK_TSC || mode ==VCLOCK_HVCLOCK ; +return mode ==VDSO_CLOCKMODE_TSC || mode ==VDSO_CLOCKMODE_HVCLOCK ; }static void kvm_track_tsc_matching (struct kvm_vcpu *vcpu)@@ -1937,7 +1937,7 @@ static inline bool kvm_check_tsc_unstabl* TSC is marked unstable when we're running on Hyper-V, * 'TSC page' clocksource is good. */ -if (pvclock_gtod_data.clock.vclock_mode ==VCLOCK_HVCLOCK ) +if (pvclock_gtod_data.clock.vclock_mode ==VDSO_CLOCKMODE_HVCLOCK )return false;#endif return check_tsc_unstable ();@@ -2092,30 +2092,30 @@ static inline u64 vgettsc(struct pvclocku64 tsc_pg_val;switch (clock->vclock_mode) { -case VCLOCK_HVCLOCK : +case VDSO_CLOCKMODE_HVCLOCK : tsc_pg_val =hv_read_tsc_page_tsc (hv_get_tsc_page (), tsc_timestamp);if (tsc_pg_val !=U64_MAX ) {/* TSC page valid */ - *mode =VCLOCK_HVCLOCK ; + *mode =VDSO_CLOCKMODE_HVCLOCK ; v = (tsc_pg_val - clock->cycle_last) & clock->mask; }else {/* TSC page invalid */ - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; }break ; -case VCLOCK_TSC : - *mode =VCLOCK_TSC ; +case VDSO_CLOCKMODE_TSC : + *mode =VDSO_CLOCKMODE_TSC ; *tsc_timestamp =read_tsc (); v = (*tsc_timestamp - clock->cycle_last) & clock->mask;break ;default : - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; } -if (*mode ==VCLOCK_NONE ) +if (*mode ==VDSO_CLOCKMODE_NONE ) *tsc_timestamp = v =0 ;return v * clock->mult;arch/x86/xen/time.c +11
-10@@ -147,7 +147,7 @@ static struct notifier_block xen_pvclockstatic int xen_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }@@ -419,12 +419,13 @@ void xen_restore_time_memory_area(void) ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);/* -* We don't disable VCLOCK_PVCLOCK entirely if it fails to register the -* secondary time info with Xen or if we migrated to a host without the -* necessary flags. On both of these cases what happens is either -* process seeing a zeroed out pvti or seeing no PVCLOCK_TSC_STABLE_BIT -* bit set. Userspace checks the latter and if 0, it discards the data -* in pvti and fallbacks to a system call for a reliable timestamp. +* We don't disable VDSO_CLOCKMODE_PVCLOCK entirely if it fails to +* register the secondary time info with Xen or if we migrated to a +* host without the necessary flags. On both of these cases what +* happens is either process seeing a zeroed out pvti or seeing no +* PVCLOCK_TSC_STABLE_BIT bit set. Userspace checks the latter and +* if 0, it discards the data in pvti and fallbacks to a system +* call for a reliable timestamp. */ if (ret !=0 ) pr_notice("Cannot restore secondary vcpu_time_info (err %d)",@@ -450,7 +451,7 @@ static void xen_setup_vsyscall_time_info ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);if (ret) { -pr_notice ("xen: VCLOCK_PVCLOCK not supported (err %d)\n" , ret); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (err %d)\n" , ret);free_page ((unsigned long )ti);return ; }@@ -467,14 +468,14 @@ static void xen_setup_vsyscall_time_infoif (!ret)free_page ((unsigned long )ti); -pr_notice ("xen: VCLOCK_PVCLOCK not supported (tsc unstable)\n" ); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (tsc unstable)\n" );return ; } xen_clock = ti;pvclock_set_pvti_cpu0_va (xen_clock); - xen_clocksource.archdata.vclock_mode =VCLOCK_PVCLOCK ; + xen_clocksource.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ; }static void __init xen_time_init (void )
Re: [patch V2 09/17] x86/vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerQuoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Switch to the generic VDSO clock mode storage.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] x86/vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: b95a8a27c300d1a39a4e36f63a518ef36e4b966c Gitweb: https://git.kernel.org/tip/b95a8a27c300d1a39a4e36f63a518ef36e4b966c Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:56 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:23 +01:00
x86/vdso: Use generic VDSO clock mode storage
Switch to the generic VDSO clock mode storage.
--- arch/x86/Kconfig | 2 +- arch/x86/entry/vdso/vma.c | 6 +++--- arch/x86/include/asm/clocksource.h | 13 ++++--------- arch/x86/include/asm/mshyperv.h | 4 ++-- arch/x86/include/asm/vdso/gettimeofday.h | 6 +++--- arch/x86/include/asm/vdso/vsyscall.h | 7 ------- arch/x86/kernel/kvmclock.c | 4 ++-- arch/x86/kernel/pvclock.c | 2 +- arch/x86/kernel/time.c | 12 +++--------- arch/x86/kernel/tsc.c | 6 +++--- arch/x86/kvm/trace.h | 4 ++-- arch/x86/kvm/x86.c | 22 +++++++++++----------- arch/x86/xen/time.c | 21 +++++++++++---------- 13 files changed, 46 insertions(+), 63 deletions(-)
arch/x86/Kconfig +1 -1
@@ -57,7 +57,6 @@ config X86 select ACPI_LEGACY_TABLES_LOOKUP if ACPI select ACPI_SYSTEM_POWER_STATES_SUPPORT if ACPI select ARCH_32BIT_OFF_T if X86_32 - select ARCH_CLOCKSOURCE_DATA select ARCH_CLOCKSOURCE_INIT select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI select ARCH_HAS_DEBUG_VIRTUAL
@@ -126,6 +125,7 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
arch/x86/entry/vdso/vma.c +3 -3
@@ -221,7 +221,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm, }else if (sym_offset == image->sym_pvclock_page) {struct pvclock_vsyscall_time_info *pvti =pvclock_get_pvti_cpu0_va (); -if (pvti &&vclock_was_used (VCLOCK_PVCLOCK )) { +if (pvti &&vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )) {return vmf_insert_pfn_prot (vma, vmf->address,__pa (pvti) >>PAGE_SHIFT ,pgprot_decrypted (vma->vm_page_prot));
@@ -229,7 +229,7 @@ static vm_fault_t vvar_fault(const struct vm_special_mapping *sm, }else if (sym_offset == image->sym_hvclock_page) {struct ms_hyperv_tsc_page *tsc_pg =hv_get_tsc_page (); -if (tsc_pg &&vclock_was_used (VCLOCK_HVCLOCK )) +if (tsc_pg &&vclock_was_used (VDSO_CLOCKMODE_HVCLOCK ))return vmf_insert_pfn (vma, vmf->address,virt_to_phys (tsc_pg) >>PAGE_SHIFT ); }else if (sym_offset == image->sym_timens_page) {
@@ -447,7 +447,7 @@ __setup("vdso=", vdso_setup);static int __init init_vdso (void ) { -BUILD_BUG_ON (VCLOCK_MAX >=32 ); +BUILD_BUG_ON (VDSO_CLOCKMODE_MAX >=32 );init_vdso_image (&vdso_image_64);
arch/x86/include/asm/clocksource.h +4 -9
@@ -4,15 +4,10 @@#ifndef _ASM_X86_CLOCKSOURCE_H#define _ASM_X86_CLOCKSOURCE_H -#define VCLOCK_NONE 0/* No vDSO clock available. */ -#define VCLOCK_TSC 1/* vDSO should use vread_tsc. */ -#define VCLOCK_PVCLOCK 2/* vDSO should use vread_pvclock. */ -#define VCLOCK_HVCLOCK 3/* vDSO should use vread_hvclock. */ -#define VCLOCK_MAX 3 - -struct arch_clocksource_data { -int vclock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_TSC, \ + VDSO_CLOCKMODE_PVCLOCK, \ + VDSO_CLOCKMODE_HVCLOCKextern unsigned int vclocks_used;
arch/x86/include/asm/mshyperv.h +2 -2
@@ -46,9 +46,9 @@ typedef int (*hyperv_fill_flush_list_func)(#define hv_set_reference_tsc (val) \ wrmsrl(HV_X64_MSR_REFERENCE_TSC, val)#define hv_set_clocksource_vdso (val) \ - ((val).archdata.vclock_mode = VCLOCK_HVCLOCK) + ((val).vdso_clock_mode = VDSO_CLOCKMODE_HVCLOCK)#define hv_enable_vdso_clocksource () \ - vclocks_set_used(VCLOCK_HVCLOCK); + vclocks_set_used(VDSO_CLOCKMODE_HVCLOCK);#define hv_get_raw_timer () rdtsc_ordered()void hyperv_callback_vector (void );
arch/x86/include/asm/vdso/gettimeofday.h +3 -3
@@ -243,7 +243,7 @@ static u64 vread_hvclock(void)static inline u64 __arch_get_hw_counter (s32 clock_mode) { -if (likely (clock_mode ==VCLOCK_TSC )) +if (likely (clock_mode ==VDSO_CLOCKMODE_TSC ))return (u64 )rdtsc_ordered ();/* * For any memory-mapped vclock type, we need to make sure that gcc
@@ -252,13 +252,13 @@ static inline u64 __arch_get_hw_counter(s32 clock_mode)* question isn't enabled, which will segfault. Hence the barriers. */ #ifdef CONFIG_PARAVIRT_CLOCK -if (clock_mode ==VCLOCK_PVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_PVCLOCK ) {barrier ();return vread_pvclock (); }#endif #ifdef CONFIG_HYPERV_TIMER -if (clock_mode ==VCLOCK_HVCLOCK ) { +if (clock_mode ==VDSO_CLOCKMODE_HVCLOCK ) {barrier ();return vread_hvclock (); }
arch/x86/include/asm/vdso/vsyscall.h +0 -7
@@ -21,13 +21,6 @@ struct vdso_data *__x86_get_k_vdso_data(void) }#define __arch_get_k_vdso_data __x86_get_k_vdso_data -static __always_inline -int __x86_get_clock_mode(struct timekeeper *tk) -{ -return tk->tkr_mono.clock->archdata.vclock_mode; -} -#define __arch_get_clock_mode __x86_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h>
arch/x86/kernel/kvmclock.c +2 -2
@@ -161,7 +161,7 @@ bool kvm_check_and_clear_guest_paused(void)static int kvm_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }
@@ -279,7 +279,7 @@ static int __init kvm_setup_vsyscall_timeinfo(void)if (!(flags &PVCLOCK_TSC_STABLE_BIT ))return 0 ; - kvm_clock.archdata.vclock_mode =VCLOCK_PVCLOCK ; + kvm_clock.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ;#endif kvmclock_init_mem ();
arch/x86/kernel/pvclock.c +1 -1
@@ -145,7 +145,7 @@ void pvclock_read_wallclock(struct pvclock_wall_clock *wall_clock,void pvclock_set_pvti_cpu0_va (struct pvclock_vsyscall_time_info *pvti) { -WARN_ON (vclock_was_used (VCLOCK_PVCLOCK )); +WARN_ON (vclock_was_used (VDSO_CLOCKMODE_PVCLOCK )); pvti_cpu0_va = pvti; }
arch/x86/kernel/time.c +3 -9
@@ -122,18 +122,12 @@ void __init time_init(void)*/ void clocksource_arch_init (struct clocksource *cs) { -if (cs->archdata.vclock_mode ==VCLOCK_NONE ) +if (cs->vdso_clock_mode ==VDSO_CLOCKMODE_NONE )return ; -if (cs->archdata.vclock_mode >VCLOCK_MAX ) { -pr_warn ("clocksource %s registered with invalid vclock_mode %d. Disabling vclock.\n" , - cs->name, cs->archdata.vclock_mode); - cs->archdata.vclock_mode =VCLOCK_NONE ; - } -if (cs->mask !=CLOCKSOURCE_MASK (64 )) { -pr_warn ("clocksource %s registered with invalid mask %016llx. Disabling vclock.\n" , +pr_warn ("clocksource %s registered with invalid mask %016llx for VDSO. Disabling VDSO support.\n" , cs->name, cs->mask); - cs->archdata.vclock_mode =VCLOCK_NONE ; + cs->vdso_clock_mode =VDSO_CLOCKMODE_NONE ; } }
arch/x86/kernel/tsc.c +3 -3
@@ -1110,7 +1110,7 @@ static void tsc_cs_tick_stable(struct clocksource *cs)static int tsc_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_TSC ); +vclocks_set_used (VDSO_CLOCKMODE_TSC );return 0 ; }
@@ -1124,7 +1124,7 @@ static struct clocksource clocksource_tsc_early = { .mask = CLOCKSOURCE_MASK(64), .flags =CLOCK_SOURCE_IS_CONTINUOUS | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,
@@ -1145,7 +1145,7 @@ static struct clocksource clocksource_tsc = { .flags = CLOCK_SOURCE_IS_CONTINUOUS |CLOCK_SOURCE_VALID_FOR_HRES | CLOCK_SOURCE_MUST_VERIFY, - .archdata = { .vclock_mode = VCLOCK_TSC }, + .vdso_clock_mode = VDSO_CLOCKMODE_TSC, .enable = tsc_cs_enable, .resume = tsc_resume, .mark_unstable = tsc_cs_mark_unstable,
arch/x86/kvm/trace.h +2 -2
@@ -815,8 +815,8 @@ TRACE_EVENT(kvm_write_tsc_offset,#ifdef CONFIG_X86_64 #define host_clocks \ - {VCLOCK_NONE, "none"}, \ - {VCLOCK_TSC, "tsc"} \ + {VDSO_CLOCKMODE_NONE, "none"}, \ + {VDSO_CLOCKMODE_TSC, "tsc"} \ TRACE_EVENT(kvm_update_master_clock, TP_PROTO(bool use_master_clock, unsigned int host_clock, bool offset_matched),
arch/x86/kvm/x86.c +11 -11
@@ -1631,7 +1631,7 @@ static void update_pvclock_gtod(struct timekeeper *tk)write_seqcount_begin (&vdata->seq);/* copy pvclock gtod data */ - vdata->clock.vclock_mode = tk->tkr_mono.clock->archdata.vclock_mode; + vdata->clock.vclock_mode = tk->tkr_mono.clock->vdso_clock_mode; vdata->clock.cycle_last = tk->tkr_mono.cycle_last; vdata->clock.mask = tk->tkr_mono.mask; vdata->clock.mult = tk->tkr_mono.mult;
@@ -1639,7 +1639,7 @@ static void update_pvclock_gtod(struct timekeeper *tk) vdata->clock.base_cycles = tk->tkr_mono.xtime_nsec; vdata->clock.offset = tk->tkr_mono.base; - vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->archdata.vclock_mode; + vdata->raw_clock.vclock_mode = tk->tkr_raw.clock->vdso_clock_mode; vdata->raw_clock.cycle_last = tk->tkr_raw.cycle_last; vdata->raw_clock.mask = tk->tkr_raw.mask; vdata->raw_clock.mult = tk->tkr_raw.mult;
@@ -1840,7 +1840,7 @@ static u64 compute_guest_tsc(struct kvm_vcpu *vcpu, s64 kernel_ns)static inline int gtod_is_based_on_tsc (int mode) { -return mode ==VCLOCK_TSC || mode ==VCLOCK_HVCLOCK ; +return mode ==VDSO_CLOCKMODE_TSC || mode ==VDSO_CLOCKMODE_HVCLOCK ; }static void kvm_track_tsc_matching (struct kvm_vcpu *vcpu)
@@ -1933,7 +1933,7 @@ static inline bool kvm_check_tsc_unstable(void)* TSC is marked unstable when we're running on Hyper-V, * 'TSC page' clocksource is good. */ -if (pvclock_gtod_data.clock.vclock_mode ==VCLOCK_HVCLOCK ) +if (pvclock_gtod_data.clock.vclock_mode ==VDSO_CLOCKMODE_HVCLOCK )return false;#endif return check_tsc_unstable ();
@@ -2088,30 +2088,30 @@ static inline u64 vgettsc(struct pvclock_clock *clock, u64 *tsc_timestamp,u64 tsc_pg_val;switch (clock->vclock_mode) { -case VCLOCK_HVCLOCK : +case VDSO_CLOCKMODE_HVCLOCK : tsc_pg_val =hv_read_tsc_page_tsc (hv_get_tsc_page (), tsc_timestamp);if (tsc_pg_val !=U64_MAX ) {/* TSC page valid */ - *mode =VCLOCK_HVCLOCK ; + *mode =VDSO_CLOCKMODE_HVCLOCK ; v = (tsc_pg_val - clock->cycle_last) & clock->mask; }else {/* TSC page invalid */ - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; }break ; -case VCLOCK_TSC : - *mode =VCLOCK_TSC ; +case VDSO_CLOCKMODE_TSC : + *mode =VDSO_CLOCKMODE_TSC ; *tsc_timestamp =read_tsc (); v = (*tsc_timestamp - clock->cycle_last) & clock->mask;break ;default : - *mode =VCLOCK_NONE ; + *mode =VDSO_CLOCKMODE_NONE ; } -if (*mode ==VCLOCK_NONE ) +if (*mode ==VDSO_CLOCKMODE_NONE ) *tsc_timestamp = v =0 ;return v * clock->mult;
arch/x86/xen/time.c +11 -10
@@ -147,7 +147,7 @@ static struct notifier_block xen_pvclock_gtod_notifier = {static int xen_cs_enable (struct clocksource *cs) { -vclocks_set_used (VCLOCK_PVCLOCK ); +vclocks_set_used (VDSO_CLOCKMODE_PVCLOCK );return 0 ; }
@@ -419,12 +419,13 @@ void xen_restore_time_memory_area(void) ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);/* -* We don't disable VCLOCK_PVCLOCK entirely if it fails to register the -* secondary time info with Xen or if we migrated to a host without the -* necessary flags. On both of these cases what happens is either -* process seeing a zeroed out pvti or seeing no PVCLOCK_TSC_STABLE_BIT -* bit set. Userspace checks the latter and if 0, it discards the data -* in pvti and fallbacks to a system call for a reliable timestamp. +* We don't disable VDSO_CLOCKMODE_PVCLOCK entirely if it fails to +* register the secondary time info with Xen or if we migrated to a +* host without the necessary flags. On both of these cases what +* happens is either process seeing a zeroed out pvti or seeing no +* PVCLOCK_TSC_STABLE_BIT bit set. Userspace checks the latter and +* if 0, it discards the data in pvti and fallbacks to a system +* call for a reliable timestamp. */ if (ret !=0 ) pr_notice("Cannot restore secondary vcpu_time_info (err %d)",
@@ -450,7 +451,7 @@ static void xen_setup_vsyscall_time_info(void) ret =HYPERVISOR_vcpu_op (VCPUOP_register_vcpu_time_memory_area,0 , &t);if (ret) { -pr_notice ("xen: VCLOCK_PVCLOCK not supported (err %d)\n" , ret); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (err %d)\n" , ret);free_page ((unsigned long )ti);return ; }
@@ -467,14 +468,14 @@ static void xen_setup_vsyscall_time_info(void)if (!ret)free_page ((unsigned long )ti); -pr_notice ("xen: VCLOCK_PVCLOCK not supported (tsc unstable)\n" ); +pr_notice ("xen: VDSO_CLOCKMODE_PVCLOCK not supported (tsc unstable)\n" );return ; } xen_clock = ti;pvclock_set_pvti_cpu0_va (xen_clock); - xen_clocksource.archdata.vclock_mode =VCLOCK_PVCLOCK ; + xen_clocksource.vdso_clock_mode =VDSO_CLOCKMODE_PVCLOCK ; }static void __init xen_time_init (void )
PATCH v2 10/17 mips: vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Switch to the generic VDSO clock mode storage.
--- arch/mips/Kconfig | 2 +- arch/mips/include/asm/clocksource.h | 18 +++--------------- arch/mips/include/asm/vdso/gettimeofday.h | 24 ++++++------------------ arch/mips/include/asm/vdso/vsyscall.h | 9 --------- arch/mips/kernel/csrc-r4k.c | 2 +- drivers/clocksource/mips-gic-timer.c | 8 ++++---- 6 files changed, 15 insertions(+), 48 deletions(-)
arch/mips/Kconfig +1 -1
@@ -4,7 +4,6 @@ config MIPS default y select ARCH_32BIT_OFF_T if !64BIT select ARCH_BINFMT_ELF_STATE if MIPS_FP_SUPPORT - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV select ARCH_HAS_PTE_SPECIAL if !(32BIT && CPU_HAS_RIXI)
@@ -38,6 +37,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL + select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/mips/include/asm/clocksource.h +3 -15
@@ -3,23 +3,11 @@* Copyright (C) 2015 Imagination Technologies * Author: Alex Smith <[email protected]> */ -#ifndef __ASM_CLOCKSOURCE_H#define __ASM_CLOCKSOURCE_H -#include <linux/types.h> - -/* VDSO clocksources. */ -#define VDSO_CLOCK_NONE 0/* No suitable clocksource. */ -#define VDSO_CLOCK_R4K 1/* Use the coprocessor 0 count. */ -#define VDSO_CLOCK_GIC 2/* Use the GIC. */ - -/** -* struct arch_clocksource_data - Architecture-specific clocksource information. -* @vdso_clock_mode: Method the VDSO should use to access the clocksource. -*/ -struct arch_clocksource_data { -u8 vdso_clock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMDOE_R4K, \ + VDSO_CLOCKMODE_GIC#endif /* __ASM_CLOCKSOURCE_H */
arch/mips/include/asm/vdso/gettimeofday.h +6 -18
@@ -175,28 +175,16 @@ static __always_inline u64 read_gic_coun static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) { -#ifdef CONFIG_CLKSRC_MIPS_GIC -const struct vdso_data *data =get_vdso_data (); -#endif -u64 cycle_now; - -switch (clock_mode) {#ifdef CONFIG_CSRC_R4K -case VDSO_CLOCK_R4K : - cycle_now =read_r4k_count (); -break ; +if (clock_mode ==VDSO_CLOCKMODE_R4K ) +return read_r4k_count ();#endif #ifdef CONFIG_CLKSRC_MIPS_GIC -case VDSO_CLOCK_GIC : - cycle_now =read_gic_count (data); -break ; +if (clock_mode ==VDSO_CLOCKMODE_GIC ) +return read_gic_count (get_vdso_data ());#endif -default : - cycle_now = __VDSO_USE_SYSCALL; -break ; - } - -return cycle_now; +/* Keep GCC happy */ +return U64_MAX ; }static inline bool mips_vdso_hres_capable (void )
arch/mips/include/asm/vdso/vsyscall.h +0 -9
@@ -19,15 +19,6 @@ struct vdso_data *__mips_get_k_vdso_data }#define __arch_get_k_vdso_data __mips_get_k_vdso_data -static __always_inline -int __mips_get_clock_mode(struct timekeeper *tk) -{ -u32 clock_mode = tk->tkr_mono.clock->archdata.vdso_clock_mode; - -return clock_mode; -} -#define __arch_get_clock_mode __mips_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h>
arch/mips/kernel/csrc-r4k.c +1 -1
@@ -78,7 +78,7 @@ int __init init_r4k_clocksource(void)* by the VDSO (HWREna is configured by configure_hwrena()). */ if (cpu_has_mips_r2_r6 &&rdhwr_count_usable ()) - clocksource_mips.archdata.vdso_clock_mode =VDSO_CLOCK_R4K ; + clocksource_mips.vdso_clock_mode =VDSO_CLOCKMODE_R4K ;clocksource_register_hz (&clocksource_mips, mips_hpt_frequency);
drivers/clocksource/mips-gic-timer.c +4 -4
@@ -155,10 +155,10 @@ static u64 gic_hpt_read(struct clocksour }static struct clocksource gic_clocksource = { - .name ="GIC" , - .read = gic_hpt_read, - .flags =CLOCK_SOURCE_IS_CONTINUOUS , - .archdata = { .vdso_clock_mode =VDSO_CLOCK_GIC }, + .name ="GIC" , + .read = gic_hpt_read, + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .vdso_clock_mode =VDSO_CLOCKMODE_GIC , };static int __init __gic_clocksource_init (void )
Re: [patch V2 10/17] mips: vdso: Use generic VDSO clock mode storage
Quoted 6 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Switch to the generic VDSO clock mode storage.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] mips: vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 2ad415f5e62c4ad93cdd01e44bc935b991e6b03a Gitweb: https://git.kernel.org/tip/2ad415f5e62c4ad93cdd01e44bc935b991e6b03a Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:57 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:24 +01:00
mips: vdso: Use generic VDSO clock mode storage
Switch to the generic VDSO clock mode storage.
--- arch/mips/Kconfig | 2 +- arch/mips/include/asm/clocksource.h | 18 ++--------------- arch/mips/include/asm/vdso/gettimeofday.h | 24 +++++----------------- arch/mips/include/asm/vdso/vsyscall.h | 9 +-------- arch/mips/kernel/csrc-r4k.c | 2 +- drivers/clocksource/mips-gic-timer.c | 8 +++---- 6 files changed, 15 insertions(+), 48 deletions(-)
arch/mips/Kconfig +1 -1
@@ -4,7 +4,6 @@ config MIPS default y select ARCH_32BIT_OFF_T if !64BIT select ARCH_BINFMT_ELF_STATE if MIPS_FP_SUPPORT - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV select ARCH_HAS_PTE_SPECIAL if !(32BIT && CPU_HAS_RIXI)
@@ -38,6 +37,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL + select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/mips/include/asm/clocksource.h +3 -15
@@ -3,23 +3,11 @@* Copyright (C) 2015 Imagination Technologies * Author: Alex Smith <[email protected]> */ -#ifndef __ASM_CLOCKSOURCE_H#define __ASM_CLOCKSOURCE_H -#include <linux/types.h> - -/* VDSO clocksources. */ -#define VDSO_CLOCK_NONE 0/* No suitable clocksource. */ -#define VDSO_CLOCK_R4K 1/* Use the coprocessor 0 count. */ -#define VDSO_CLOCK_GIC 2/* Use the GIC. */ - -/** -* struct arch_clocksource_data - Architecture-specific clocksource information. -* @vdso_clock_mode: Method the VDSO should use to access the clocksource. -*/ -struct arch_clocksource_data { -u8 vdso_clock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMDOE_R4K, \ + VDSO_CLOCKMODE_GIC#endif /* __ASM_CLOCKSOURCE_H */
arch/mips/include/asm/vdso/gettimeofday.h +6 -18
@@ -175,28 +175,16 @@ static __always_inline u64 read_gic_count(const struct vdso_data *data) static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) { -#ifdef CONFIG_CLKSRC_MIPS_GIC -const struct vdso_data *data =get_vdso_data (); -#endif -u64 cycle_now; - -switch (clock_mode) {#ifdef CONFIG_CSRC_R4K -case VDSO_CLOCK_R4K : - cycle_now =read_r4k_count (); -break ; +if (clock_mode ==VDSO_CLOCKMODE_R4K ) +return read_r4k_count ();#endif #ifdef CONFIG_CLKSRC_MIPS_GIC -case VDSO_CLOCK_GIC : - cycle_now =read_gic_count (data); -break ; +if (clock_mode ==VDSO_CLOCKMODE_GIC ) +return read_gic_count (get_vdso_data ());#endif -default : - cycle_now = __VDSO_USE_SYSCALL; -break ; - } - -return cycle_now; +/* Keep GCC happy */ +return U64_MAX ; }static inline bool mips_vdso_hres_capable (void )
arch/mips/include/asm/vdso/vsyscall.h +0 -9
@@ -19,15 +19,6 @@ struct vdso_data *__mips_get_k_vdso_data(void) }#define __arch_get_k_vdso_data __mips_get_k_vdso_data -static __always_inline -int __mips_get_clock_mode(struct timekeeper *tk) -{ -u32 clock_mode = tk->tkr_mono.clock->archdata.vdso_clock_mode; - -return clock_mode; -} -#define __arch_get_clock_mode __mips_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h>
arch/mips/kernel/csrc-r4k.c +1 -1
@@ -78,7 +78,7 @@ int __init init_r4k_clocksource(void)* by the VDSO (HWREna is configured by configure_hwrena()). */ if (cpu_has_mips_r2_r6 &&rdhwr_count_usable ()) - clocksource_mips.archdata.vdso_clock_mode =VDSO_CLOCK_R4K ; + clocksource_mips.vdso_clock_mode =VDSO_CLOCKMODE_R4K ;clocksource_register_hz (&clocksource_mips, mips_hpt_frequency);
drivers/clocksource/mips-gic-timer.c +4 -4
@@ -155,10 +155,10 @@ static u64 gic_hpt_read(struct clocksource *cs) }static struct clocksource gic_clocksource = { - .name ="GIC" , - .read = gic_hpt_read, - .flags =CLOCK_SOURCE_IS_CONTINUOUS , - .archdata = { .vdso_clock_mode =VDSO_CLOCK_GIC }, + .name ="GIC" , + .read = gic_hpt_read, + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .vdso_clock_mode =VDSO_CLOCKMODE_GIC , };static int __init __gic_clocksource_init (void )
[tip: timers/core] mips: vdso: Use generic VDSO clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: e1bdb22ebe5363ed75ddedf836ca9f19e1195337 Gitweb: https://git.kernel.org/tip/e1bdb22ebe5363ed75ddedf836ca9f19e1195337 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:57 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:16 +01:00
mips: vdso: Use generic VDSO clock mode storage
Switch to the generic VDSO clock mode storage.
--- arch/mips/Kconfig | 2 +- arch/mips/include/asm/clocksource.h | 18 ++----------- arch/mips/include/asm/vdso/gettimeofday.h | 30 +++++++--------------- arch/mips/include/asm/vdso/vsyscall.h | 9 +------- arch/mips/kernel/csrc-r4k.c | 2 +- drivers/clocksource/mips-gic-timer.c | 8 +++--- 6 files changed, 19 insertions(+), 50 deletions(-)
arch/mips/Kconfig +1 -1
@@ -4,7 +4,6 @@ config MIPS default y select ARCH_32BIT_OFF_T if !64BIT select ARCH_BINFMT_ELF_STATE if MIPS_FP_SUPPORT - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV select ARCH_HAS_PTE_SPECIAL if !(32BIT && CPU_HAS_RIXI)
@@ -38,6 +37,7 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL + select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/mips/include/asm/clocksource.h +3 -15
@@ -3,23 +3,11 @@* Copyright (C) 2015 Imagination Technologies * Author: Alex Smith <[email protected]> */ -#ifndef __ASM_CLOCKSOURCE_H#define __ASM_CLOCKSOURCE_H -#include <linux/types.h> - -/* VDSO clocksources. */ -#define VDSO_CLOCK_NONE 0/* No suitable clocksource. */ -#define VDSO_CLOCK_R4K 1/* Use the coprocessor 0 count. */ -#define VDSO_CLOCK_GIC 2/* Use the GIC. */ - -/** -* struct arch_clocksource_data - Architecture-specific clocksource information. -* @vdso_clock_mode: Method the VDSO should use to access the clocksource. -*/ -struct arch_clocksource_data { -u8 vdso_clock_mode; -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_R4K, \ + VDSO_CLOCKMODE_GIC#endif /* __ASM_CLOCKSOURCE_H */
arch/mips/include/asm/vdso/gettimeofday.h +10 -20
@@ -24,8 +24,6 @@#define VDSO_HAS_CLOCK_GETRES 1 -#define __VDSO_USE_SYSCALL ULLONG_MAX -static __always_inline long gettimeofday_fallback (struct __kernel_old_timeval *_tv,struct timezone *_tz)
@@ -175,28 +173,20 @@ static __always_inline u64 read_gic_count(const struct vdso_data *data) static __always_inline u64 __arch_get_hw_counter(s32 clock_mode) { -#ifdef CONFIG_CLKSRC_MIPS_GIC -const struct vdso_data *data =get_vdso_data (); -#endif -u64 cycle_now; - -switch (clock_mode) {#ifdef CONFIG_CSRC_R4K -case VDSO_CLOCK_R4K : - cycle_now =read_r4k_count (); -break ; +if (clock_mode ==VDSO_CLOCKMODE_R4K ) +return read_r4k_count ();#endif #ifdef CONFIG_CLKSRC_MIPS_GIC -case VDSO_CLOCK_GIC : - cycle_now =read_gic_count (data); -break ; +if (clock_mode ==VDSO_CLOCKMODE_GIC ) +return read_gic_count (get_vdso_data ());#endif -default : - cycle_now = __VDSO_USE_SYSCALL; -break ; - } - -return cycle_now; +/* +* Core checks mode already. So this raced against a concurrent +* update. Return something. Core will do another round see the +* change and fallback to syscall. +*/ +return 0 ; }static inline bool mips_vdso_hres_capable (void )
arch/mips/include/asm/vdso/vsyscall.h +0 -9
@@ -19,15 +19,6 @@ struct vdso_data *__mips_get_k_vdso_data(void) }#define __arch_get_k_vdso_data __mips_get_k_vdso_data -static __always_inline -int __mips_get_clock_mode(struct timekeeper *tk) -{ -u32 clock_mode = tk->tkr_mono.clock->archdata.vdso_clock_mode; - -return clock_mode; -} -#define __arch_get_clock_mode __mips_get_clock_mode -/* The asm-generic header needs to be included after the definitions above */ #include <asm-generic/vdso/vsyscall.h>
arch/mips/kernel/csrc-r4k.c +1 -1
@@ -78,7 +78,7 @@ int __init init_r4k_clocksource(void)* by the VDSO (HWREna is configured by configure_hwrena()). */ if (cpu_has_mips_r2_r6 &&rdhwr_count_usable ()) - clocksource_mips.archdata.vdso_clock_mode =VDSO_CLOCK_R4K ; + clocksource_mips.vdso_clock_mode =VDSO_CLOCKMODE_R4K ;clocksource_register_hz (&clocksource_mips, mips_hpt_frequency);
drivers/clocksource/mips-gic-timer.c +4 -4
@@ -155,10 +155,10 @@ static u64 gic_hpt_read(struct clocksource *cs) }static struct clocksource gic_clocksource = { - .name ="GIC" , - .read = gic_hpt_read, - .flags =CLOCK_SOURCE_IS_CONTINUOUS , - .archdata = { .vdso_clock_mode =VDSO_CLOCK_GIC }, + .name ="GIC" , + .read = gic_hpt_read, + .flags =CLOCK_SOURCE_IS_CONTINUOUS , + .vdso_clock_mode =VDSO_CLOCKMODE_GIC , };static int __init __gic_clocksource_init (void )
PATCH v2 11/17 ARM/arm64: vdso: Use common vdso clock mode storage
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
--- arch/arm/Kconfig | 1 - arch/arm/include/asm/clocksource.h | 5 ++--- arch/arm/include/asm/vdso/vsyscall.h | 21 --------------------- arch/arm/mm/Kconfig | 1 + arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/clocksource.h | 5 ++--- arch/arm64/include/asm/vdso/vsyscall.h | 9 --------- drivers/clocksource/arm_arch_timer.c | 8 ++++---- 8 files changed, 10 insertions(+), 42 deletions(-)
arch/arm/Kconfig +0 -1
@@ -3,7 +3,6 @@ config ARM bool default y select ARCH_32BIT_OFF_T - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEVMEM_IS_ALLOWED
arch/arm/include/asm/clocksource.h +2 -3
@@ -1,8 +1,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm/include/asm/vdso/vsyscall.h +0 -21
@@ -11,18 +11,6 @@extern struct vdso_data *vdso_data;extern bool cntvct_ok; -static __always_inline -bool tk_is_cntvct(const struct timekeeper *tk) -{ -if (!IS_ENABLED (CONFIG_ARM_ARCH_TIMER )) -return false; - -if (!tk->tkr_mono.clock->archdata.vdso_direct) -return false; - -return true; -} -/* * Update the vDSO data page to keep in sync with kernel timekeeping. */
@@ -41,15 +29,6 @@ bool __arm_update_vdso_data(void)#define __arch_update_vdso_data __arm_update_vdso_datastatic __always_inline -int __arm_get_clock_mode(struct timekeeper *tk) -{ -u32 __tk_is_cntvct =tk_is_cntvct (tk); - -return __tk_is_cntvct; -} -#define __arch_get_clock_mode __arm_get_clock_mode - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
arch/arm/mm/Kconfig +1 -0
@@ -900,6 +900,7 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +1 -1
@@ -9,7 +9,6 @@ config ARM64 select ACPI_MCFG if (ACPI && PCI) select ACPI_SPCR_TABLE if ACPI select ACPI_PPTT if ACPI - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_PREP_COHERENT
@@ -110,6 +109,7 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/arm64/include/asm/clocksource.h +2 -3
@@ -2,8 +2,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm64/include/asm/vdso/vsyscall.h +0 -9
@@ -22,15 +22,6 @@ struct vdso_data *__arm64_get_k_vdso_dat#define __arch_get_k_vdso_data __arm64_get_k_vdso_datastatic __always_inline -int __arm64_get_clock_mode(struct timekeeper *tk) -{ -u32 use_syscall = !tk->tkr_mono.clock->archdata.vdso_direct; - -return use_syscall; -} -#define __arch_get_clock_mode __arm64_get_clock_mode - -static __always_inline void __arm64_update_vsyscall(struct vdso_data *vdata, struct timekeeper *tk) { vdata[CS_HRES_COARSE ].mask =VDSO_PRECISION_MASK ;
drivers/clocksource/arm_arch_timer.c +4 -4
@@ -69,7 +69,7 @@ static enum arch_timer_ppi_nr arch_timerstatic bool arch_timer_c3stop;static bool arch_timer_mem_use_virtual;static bool arch_counter_suspend_stop; -static bool vdso_default = true; +static enum vdso_clock_mode vdso_default =VDSO_CLOCKMODE_ARCHTIMER ;static cpumask_t evtstrm_available =CPU_MASK_NONE ;static bool evtstrm_enable =IS_ENABLED (CONFIG_ARM_ARCH_TIMER_EVTSTREAM );
@@ -560,8 +560,8 @@ void arch_timer_enable_workaround(const* change both the default value and the vdso itself. */ if (wa->read_cntvct_el0) { - clocksource_counter.archdata.vdso_direct = false; - vdso_default = false; + clocksource_counter.vdso_clock_mode =VDSO_CLOCKMODE_NONE ; + vdso_default =VDSO_CLOCKMODE_NONE ; } }
@@ -979,7 +979,7 @@ static void __init arch_counter_register } arch_timer_read_counter = rd; - clocksource_counter.archdata.vdso_direct = vdso_default; + clocksource_counter.vdso_clock_mode = vdso_default; }else { arch_timer_read_counter = arch_counter_get_cntvct_mem; }
Re: [patch V2 11/17] ARM/arm64: vdso: Use common vdso clock mode storage
Quoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] ARM/arm64: vdso: Use common vdso clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: db26b4e9ff74027e16ee47be2da7b86b7e070352 Gitweb: https://git.kernel.org/tip/db26b4e9ff74027e16ee47be2da7b86b7e070352 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:58 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:25 +01:00
ARM/arm64: vdso: Use common vdso clock mode storage
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
--- arch/arm/Kconfig | 1 - arch/arm/include/asm/clocksource.h | 5 ++--- arch/arm/include/asm/vdso/vsyscall.h | 21 --------------------- arch/arm/mm/Kconfig | 1 + arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/clocksource.h | 5 ++--- arch/arm64/include/asm/vdso/vsyscall.h | 9 --------- drivers/clocksource/arm_arch_timer.c | 8 ++++---- 8 files changed, 10 insertions(+), 42 deletions(-)
arch/arm/Kconfig +0 -1
@@ -3,7 +3,6 @@ config ARM bool default y select ARCH_32BIT_OFF_T - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEVMEM_IS_ALLOWED
arch/arm/include/asm/clocksource.h +2 -3
@@ -1,8 +1,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm/include/asm/vdso/vsyscall.h +0 -21
@@ -11,18 +11,6 @@extern struct vdso_data *vdso_data;extern bool cntvct_ok; -static __always_inline -bool tk_is_cntvct(const struct timekeeper *tk) -{ -if (!IS_ENABLED (CONFIG_ARM_ARCH_TIMER )) -return false; - -if (!tk->tkr_mono.clock->archdata.vdso_direct) -return false; - -return true; -} -/* * Update the vDSO data page to keep in sync with kernel timekeeping. */
@@ -41,15 +29,6 @@ bool __arm_update_vdso_data(void)#define __arch_update_vdso_data __arm_update_vdso_datastatic __always_inline -int __arm_get_clock_mode(struct timekeeper *tk) -{ -u32 __tk_is_cntvct =tk_is_cntvct (tk); - -return __tk_is_cntvct; -} -#define __arch_get_clock_mode __arm_get_clock_mode - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
arch/arm/mm/Kconfig +1 -0
@@ -900,6 +900,7 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +1 -1
@@ -9,7 +9,6 @@ config ARM64 select ACPI_MCFG if (ACPI && PCI) select ACPI_SPCR_TABLE if ACPI select ACPI_PPTT if ACPI - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_PREP_COHERENT
@@ -111,6 +110,7 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/arm64/include/asm/clocksource.h +2 -3
@@ -2,8 +2,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm64/include/asm/vdso/vsyscall.h +0 -9
@@ -22,15 +22,6 @@ struct vdso_data *__arm64_get_k_vdso_data(void)#define __arch_get_k_vdso_data __arm64_get_k_vdso_datastatic __always_inline -int __arm64_get_clock_mode(struct timekeeper *tk) -{ -u32 use_syscall = !tk->tkr_mono.clock->archdata.vdso_direct; - -return use_syscall; -} -#define __arch_get_clock_mode __arm64_get_clock_mode - -static __always_inline void __arm64_update_vsyscall(struct vdso_data *vdata, struct timekeeper *tk) { vdata[CS_HRES_COARSE ].mask =VDSO_PRECISION_MASK ;
drivers/clocksource/arm_arch_timer.c +4 -4
@@ -69,7 +69,7 @@ static enum arch_timer_ppi_nr arch_timer_uses_ppi = ARCH_TIMER_VIRT_PPI;static bool arch_timer_c3stop;static bool arch_timer_mem_use_virtual;static bool arch_counter_suspend_stop; -static bool vdso_default = true; +static enum vdso_clock_mode vdso_default =VDSO_CLOCKMODE_ARCHTIMER ;static cpumask_t evtstrm_available =CPU_MASK_NONE ;static bool evtstrm_enable =IS_ENABLED (CONFIG_ARM_ARCH_TIMER_EVTSTREAM );
@@ -560,8 +560,8 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa* change both the default value and the vdso itself. */ if (wa->read_cntvct_el0) { - clocksource_counter.archdata.vdso_direct = false; - vdso_default = false; + clocksource_counter.vdso_clock_mode =VDSO_CLOCKMODE_NONE ; + vdso_default =VDSO_CLOCKMODE_NONE ; } }
@@ -979,7 +979,7 @@ static void __init arch_counter_register(unsigned type) } arch_timer_read_counter = rd; - clocksource_counter.archdata.vdso_direct = vdso_default; + clocksource_counter.vdso_clock_mode = vdso_default; }else { arch_timer_read_counter = arch_counter_get_cntvct_mem; }
[tip: timers/core] ARM/arm64: vdso: Use common vdso clock mode storage
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 5e3c6a312a0946d2d83e32359612cbb925a8bed0 Gitweb: https://git.kernel.org/tip/5e3c6a312a0946d2d83e32359612cbb925a8bed0 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:58 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:16 +01:00
ARM/arm64: vdso: Use common vdso clock mode storage
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
--- arch/arm/Kconfig | 1 +- arch/arm/include/asm/clocksource.h | 5 +-- arch/arm/include/asm/vdso/gettimeofday.h | 12 ++++++-- arch/arm/include/asm/vdso/vsyscall.h | 21 +-------------- arch/arm/mm/Kconfig | 1 +- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/clocksource.h | 5 +-- arch/arm64/include/asm/vdso/compat_gettimeofday.h | 11 +++---- arch/arm64/include/asm/vdso/gettimeofday.h | 11 +++---- arch/arm64/include/asm/vdso/vsyscall.h | 9 +------ drivers/clocksource/arm_arch_timer.c | 8 ++--- 11 files changed, 29 insertions(+), 57 deletions(-)
arch/arm/Kconfig +0 -1
@@ -3,7 +3,6 @@ config ARM bool default y select ARCH_32BIT_OFF_T - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_BINFMT_FLAT select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEVMEM_IS_ALLOWED
arch/arm/include/asm/clocksource.h +2 -3
@@ -1,8 +1,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm/include/asm/vdso/gettimeofday.h +9 -3
@@ -117,15 +117,21 @@ static __always_inline u64 __arch_get_hw_counter(int clock_mode)#ifdef CONFIG_ARM_ARCH_TIMER u64 cycle_now; -if (!clock_mode) -return -EINVAL ; +/* +* Core checks for mode already, so this raced against a concurrent +* update. Return something. Core will do another round and then +* see the mode change and fallback to the syscall. +*/ +if (clock_mode ==VDSO_CLOCKMODE_NONE ) +return 0 ;isb (); cycle_now =read_sysreg (CNTVCT );return cycle_now;#else -return -EINVAL ;/* use fallback */ +/* Make GCC happy. This is compiled out anyway */ +return 0 ;#endif }
arch/arm/include/asm/vdso/vsyscall.h +0 -21
@@ -11,18 +11,6 @@extern struct vdso_data *vdso_data;extern bool cntvct_ok; -static __always_inline -bool tk_is_cntvct(const struct timekeeper *tk) -{ -if (!IS_ENABLED (CONFIG_ARM_ARCH_TIMER )) -return false; - -if (!tk->tkr_mono.clock->archdata.vdso_direct) -return false; - -return true; -} -/* * Update the vDSO data page to keep in sync with kernel timekeeping. */
@@ -41,15 +29,6 @@ bool __arm_update_vdso_data(void)#define __arch_update_vdso_data __arm_update_vdso_datastatic __always_inline -int __arm_get_clock_mode(struct timekeeper *tk) -{ -u32 __tk_is_cntvct =tk_is_cntvct (tk); - -return __tk_is_cntvct; -} -#define __arch_get_clock_mode __arm_get_clock_mode - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
arch/arm/mm/Kconfig +1 -0
@@ -900,6 +900,7 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +1 -1
@@ -9,7 +9,6 @@ config ARM64 select ACPI_MCFG if (ACPI && PCI) select ACPI_SPCR_TABLE if ACPI select ACPI_PPTT if ACPI - select ARCH_CLOCKSOURCE_DATA select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEVMEM_IS_ALLOWED select ARCH_HAS_DMA_PREP_COHERENT
@@ -111,6 +110,7 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY + select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/arm64/include/asm/clocksource.h +2 -3
@@ -2,8 +2,7 @@#ifndef _ASM_CLOCKSOURCE_H#define _ASM_CLOCKSOURCE_H -struct arch_clocksource_data { -bool vdso_direct;/* Usable for direct VDSO access? */ -}; +#define VDSO_ARCH_CLOCKMODES \ + VDSO_CLOCKMODE_ARCHTIMER#endif
arch/arm64/include/asm/vdso/compat_gettimeofday.h +5 -6
@@ -12,8 +12,6 @@#include <asm/vdso/compat_barrier.h> -#define __VDSO_USE_SYSCALL ULLONG_MAX -#define VDSO_HAS_CLOCK_GETRES 1#define BUILD_VDSO32 1
@@ -117,11 +115,12 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode)u64 res;/* -* clock_mode == 0 implies that vDSO are enabled otherwise -* fallback on syscall. +* Core checks for mode already, so this raced against a concurrent +* update. Return something. Core will do another round and then +* see the mode change and fallback to the syscall. */ -if (clock_mode) -return __VDSO_USE_SYSCALL; +if (clock_mode ==VDSO_CLOCKMODE_NONE ) +return 0 ;/* * This isb() is required to prevent that the counter value
arch/arm64/include/asm/vdso/gettimeofday.h +5 -6
@@ -10,8 +10,6 @@#include <asm/unistd.h> #include <uapi/linux/time.h> -#define __VDSO_USE_SYSCALL ULLONG_MAX -#define VDSO_HAS_CLOCK_GETRES 1 static __always_inline
@@ -71,11 +69,12 @@ static __always_inline u64 __arch_get_hw_counter(s32 clock_mode)u64 res;/* -* clock_mode == 0 implies that vDSO are enabled otherwise -* fallback on syscall. +* Core checks for mode already, so this raced against a concurrent +* update. Return something. Core will do another round and then +* see the mode change and fallback to the syscall. */ -if (clock_mode) -return __VDSO_USE_SYSCALL; +if (clock_mode ==VDSO_CLOCKMODE_NONE ) +return 0 ;/* * This isb() is required to prevent that the counter value
arch/arm64/include/asm/vdso/vsyscall.h +0 -9
@@ -22,15 +22,6 @@ struct vdso_data *__arm64_get_k_vdso_data(void)#define __arch_get_k_vdso_data __arm64_get_k_vdso_datastatic __always_inline -int __arm64_get_clock_mode(struct timekeeper *tk) -{ -u32 use_syscall = !tk->tkr_mono.clock->archdata.vdso_direct; - -return use_syscall; -} -#define __arch_get_clock_mode __arm64_get_clock_mode - -static __always_inline void __arm64_update_vsyscall(struct vdso_data *vdata, struct timekeeper *tk) { vdata[CS_HRES_COARSE ].mask =VDSO_PRECISION_MASK ;
drivers/clocksource/arm_arch_timer.c +4 -4
@@ -69,7 +69,7 @@ static enum arch_timer_ppi_nr arch_timer_uses_ppi = ARCH_TIMER_VIRT_PPI;static bool arch_timer_c3stop;static bool arch_timer_mem_use_virtual;static bool arch_counter_suspend_stop; -static bool vdso_default = true; +static enum vdso_clock_mode vdso_default =VDSO_CLOCKMODE_ARCHTIMER ;static cpumask_t evtstrm_available =CPU_MASK_NONE ;static bool evtstrm_enable =IS_ENABLED (CONFIG_ARM_ARCH_TIMER_EVTSTREAM );
@@ -560,8 +560,8 @@ void arch_timer_enable_workaround(const struct arch_timer_erratum_workaround *wa* change both the default value and the vdso itself. */ if (wa->read_cntvct_el0) { - clocksource_counter.archdata.vdso_direct = false; - vdso_default = false; + clocksource_counter.vdso_clock_mode =VDSO_CLOCKMODE_NONE ; + vdso_default =VDSO_CLOCKMODE_NONE ; } }
@@ -979,7 +979,7 @@ static void __init arch_counter_register(unsigned type) } arch_timer_read_counter = rd; - clocksource_counter.archdata.vdso_direct = vdso_default; + clocksource_counter.vdso_clock_mode = vdso_default; }else { arch_timer_read_counter = arch_counter_get_cntvct_mem; }
Re: [patch V2 11/17] ARM/arm64: vdso: Use common vdso clock mode storage
In reply to Thomas GleixnerHi Thomas,
Quoted 8 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
Signed-off-by Thomas Gleixner <[email protected]>Reviewed-by Vincenzo Frascino <[email protected]>Tested-by Vincenzo Frascino <[email protected]>
This patch landed in today's linux-next (next-20200221) as commit 5e3c6a312a09. It breaks ARM 32bit build without VDSO enabled in .config:
$ make ARCH=arm multi_v7_defconfig
$ ./scripts/config -e ARM_LPAE -e VIRTUALIZATION -e KVM -d VDSO
$ make ARCH=arm olddefconfig
$ make
...
drivers/clocksource/arm_arch_timer.c:73:44: error:
‘VDSO_CLOCKMODE_ARCHTIMER’ undeclared here (not in a function)
static enum vdso_clock_mode vdso_default = VDSO_CLOCKMODE_ARCHTIMER; ^scripts/Makefile.build:267: recipe for target 'drivers/clocksource/arm_arch_timer.o' failed
make[2]: *** [drivers/clocksource/arm_arch_timer.o] Error 1 make[2]: *** Waiting for unfinished jobs.... scripts/Makefile.build:505: recipe for target 'drivers/clocksource' failed make[1]: *** [drivers/clocksource] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:1683: recipe for target 'drivers' failed
make: *** [drivers] Error 2
Quoted 1 line
...
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Vincenzo Frascino
maintainerRe: [patch V2 11/17] ARM/arm64: vdso: Use common vdso clock mode storage
Hi Marek,
Quoted 39 lines · Marek Szyprowski
Hi Thomas,
Quoted 8 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Convert ARM/ARM64 to the generic VDSO clock mode storage. This needs to happen in one go as they share the clocksource driver.
Signed-off-by Thomas Gleixner <[email protected]>Reviewed-by Vincenzo Frascino <[email protected]>Tested-by Vincenzo Frascino <[email protected]>This patch landed in today's linux-next (next-20200221) as commit
5e3c6a312a09. It breaks ARM 32bit build without VDSO enabled in .config:$ make ARCH=arm multi_v7_defconfig$ ./scripts/config -e ARM_LPAE -e VIRTUALIZATION -e KVM -d VDSO$ make ARCH=arm olddefconfig$ make...
drivers/clocksource/arm_arch_timer.c:73:44: error:
‘VDSO_CLOCKMODE_ARCHTIMER’ undeclared here (not in a function)
static enumvdso_clock_modevdso_default=VDSO_CLOCKMODE_ARCHTIMER; ^scripts/Makefile.build:267: recipe for target 'drivers/clocksource/arm_arch_timer.o' failed
make[2]: *** [drivers/clocksource/arm_arch_timer.o] Error 1 make[2]: *** Waiting for unfinished jobs....scripts/Makefile.build:505: recipe for target 'drivers/clocksource' failed make[1]: *** [drivers/clocksource] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:1683: recipe for target 'drivers' failed
make: *** [drivers] Error 2
Quoted 4 lines
Quoted 1 line
...
Best regards
I think I have a fix for your problem, could you please try it [1]?
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Regards, Vincenzo
Vincenzo Frascino
maintainerRe: [patch V2 11/17] ARM/arm64: vdso: Use common vdso clock mode storage
In reply to Marek SzyprowskiHi Marek,
On 21/02/2020 11:56, Marek Szyprowski wrote:
[...]
Quoted 6 lines
This patch landed in today's linux-next (next-20200221) as commit
5e3c6a312a09. It breaks ARM 32bit build without VDSO enabled in .config:$ make ARCH=arm multi_v7_defconfig
Quoted 8 lines
$ ./scripts/config -e ARM_LPAE -e VIRTUALIZATION -e KVM -d VDSO$ make ARCH=arm olddefconfig$ make...
[...]
I have a new version [1]. Could you please test it?
[1] https://lore.kernel.org/lkml/[email protected]/
Signature · 2 lines
Regards, Vincenzo
PATCH v2 12/17 lib/vdso: Cleanup clock mode storage leftovers
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Now that all architectures are converted to use the generic storage the helpers and conditionals can be removed.
--- arch/arm/mm/Kconfig | 1 - arch/arm64/Kconfig | 1 - arch/mips/Kconfig | 1 - arch/x86/Kconfig | 1 - include/asm-generic/vdso/vsyscall.h | 7 ------- include/linux/clocksource.h | 4 ++-- kernel/time/vsyscall.c | 4 ---- lib/vdso/Kconfig | 3 --- lib/vdso/gettimeofday.c | 17 +++++------------ 9 files changed, 7 insertions(+), 32 deletions(-)
arch/arm/mm/Kconfig +0 -1
@@ -900,7 +900,6 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +0 -1
@@ -109,7 +109,6 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/mips/Kconfig +0 -1
@@ -35,7 +35,6 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL - select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/x86/Kconfig +0 -1
@@ -124,7 +124,6 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -18,13 +18,6 @@ static __always_inline bool __arch_updat }#endif /* __arch_update_vdso_data */ -#ifndef __arch_get_clock_mode -static __always_inline int __arch_get_clock_mode(struct timekeeper *tk) -{ -return 0 ; -} -#endif /* __arch_get_clock_mode */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
include/linux/clocksource.h +2 -2
@@ -24,13 +24,13 @@ struct clocksource;struct module ;#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ - defined(CONFIG_GENERIC_VDSO_CLOCK_MODE ) + defined(CONFIG_GENERIC_GETTIMEOFDAY )#include <asm/clocksource.h> #endif enum vdso_clock_mode {VDSO_CLOCKMODE_NONE , -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +#ifdef CONFIG_GENERIC_GETTIMEOFDAY VDSO_ARCH_CLOCKMODES ,#endif VDSO_CLOCKMODE_MAX ,
kernel/time/vsyscall.c +0 -4
@@ -77,11 +77,7 @@ void update_vsyscall(struct timekeeper */* copy vsyscall data */ vdso_write_begin (vdata); -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE clock_mode = tk->tkr_mono.clock->vdso_clock_mode; -#else - clock_mode =__arch_get_clock_mode (tk); -#endif vdata[CS_HRES_COARSE ].clock_mode = clock_mode; vdata[CS_RAW ].clock_mode = clock_mode;
lib/vdso/Kconfig +0 -3
@@ -30,7 +30,4 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO -config GENERIC_VDSO_CLOCK_MODE - bool - endif
lib/vdso/gettimeofday.c +5 -12
@@ -65,16 +65,13 @@ static int do_hres_timens(const struct v do { seq = vdso_read_begin(vd); - if (IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - vd->clock_mode == VDSO_CLOCKMODE_NONE) + + if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) return -1; + cycles = __arch_get_hw_counter(vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; - if (!IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - unlikely((s64)cycles < 0)) - return -1; - ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
@@ -137,16 +134,12 @@ static __always_inline int do_hres(const }smp_rmb (); -if (IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && - vd->clock_mode ==VDSO_CLOCKMODE_NONE ) +if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE ))return -1 ; + cycles =__arch_get_hw_counter (vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; -if (!IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && -unlikely ((s64 )cycles <0 )) -return -1 ; - ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
Re: [patch V2 12/17] lib/vdso: Cleanup clock mode storage leftovers
Quoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Now that all architectures are converted to use the generic storage the helpers and conditionals can be removed.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] lib/vdso: Cleanup clock mode storage leftovers
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: c2130090f32165f4303202c7839dcbaf6bf943b6 Gitweb: https://git.kernel.org/tip/c2130090f32165f4303202c7839dcbaf6bf943b6 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:59 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:25 +01:00
lib/vdso: Cleanup clock mode storage leftovers
Now that all architectures are converted to use the generic storage the helpers and conditionals can be removed.
--- arch/arm/mm/Kconfig | 1 - arch/arm64/Kconfig | 1 - arch/mips/Kconfig | 1 - arch/x86/Kconfig | 1 - include/asm-generic/vdso/vsyscall.h | 7 ------- include/linux/clocksource.h | 4 ++-- kernel/time/vsyscall.c | 4 ---- lib/vdso/Kconfig | 3 --- lib/vdso/gettimeofday.c | 17 +++++------------ 9 files changed, 7 insertions(+), 32 deletions(-)
arch/arm/mm/Kconfig +0 -1
@@ -900,7 +900,6 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +0 -1
@@ -110,7 +110,6 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/mips/Kconfig +0 -1
@@ -37,7 +37,6 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL - select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/x86/Kconfig +0 -1
@@ -125,7 +125,6 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -18,13 +18,6 @@ static __always_inline bool __arch_update_vdso_data(void) }#endif /* __arch_update_vdso_data */ -#ifndef __arch_get_clock_mode -static __always_inline int __arch_get_clock_mode(struct timekeeper *tk) -{ -return 0 ; -} -#endif /* __arch_get_clock_mode */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
include/linux/clocksource.h +2 -2
@@ -24,13 +24,13 @@ struct clocksource;struct module ;#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ - defined(CONFIG_GENERIC_VDSO_CLOCK_MODE ) + defined(CONFIG_GENERIC_GETTIMEOFDAY )#include <asm/clocksource.h> #endif enum vdso_clock_mode {VDSO_CLOCKMODE_NONE , -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +#ifdef CONFIG_GENERIC_GETTIMEOFDAY VDSO_ARCH_CLOCKMODES ,#endif VDSO_CLOCKMODE_MAX ,
kernel/time/vsyscall.c +0 -4
@@ -77,11 +77,7 @@ void update_vsyscall(struct timekeeper *tk)/* copy vsyscall data */ vdso_write_begin (vdata); -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE clock_mode = tk->tkr_mono.clock->vdso_clock_mode; -#else - clock_mode =__arch_get_clock_mode (tk); -#endif vdata[CS_HRES_COARSE ].clock_mode = clock_mode; vdata[CS_RAW ].clock_mode = clock_mode;
lib/vdso/Kconfig +0 -3
@@ -30,7 +30,4 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO -config GENERIC_VDSO_CLOCK_MODE - bool - endif
lib/vdso/gettimeofday.c +5 -12
@@ -65,16 +65,13 @@ static int do_hres_timens(const struct vdso_data *vdns, clockid_t clk, do { seq = vdso_read_begin(vd); - if (IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - vd->clock_mode == VDSO_CLOCKMODE_NONE) + + if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) return -1; + cycles = __arch_get_hw_counter(vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; - if (!IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - unlikely((s64)cycles < 0)) - return -1; - ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
@@ -137,16 +134,12 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, }smp_rmb (); -if (IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && - vd->clock_mode ==VDSO_CLOCKMODE_NONE ) +if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE ))return -1 ; + cycles =__arch_get_hw_counter (vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; -if (!IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && -unlikely ((s64 )cycles <0 )) -return -1 ; - ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
[tip: timers/core] lib/vdso: Cleanup clock mode storage leftovers
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: f86fd32db706613fe8d0104057efa6e83e0d7e8f Gitweb: https://git.kernel.org/tip/f86fd32db706613fe8d0104057efa6e83e0d7e8f Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:38:59 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:17 +01:00
lib/vdso: Cleanup clock mode storage leftovers
Now that all architectures are converted to use the generic storage the helpers and conditionals can be removed.
--- arch/arm/mm/Kconfig | 1 - arch/arm64/Kconfig | 1 - arch/mips/Kconfig | 1 - arch/x86/Kconfig | 1 - include/asm-generic/vdso/vsyscall.h | 7 ------- include/linux/clocksource.h | 4 ++-- kernel/time/vsyscall.c | 4 ---- lib/vdso/Kconfig | 3 --- lib/vdso/gettimeofday.c | 17 +++++------------ 9 files changed, 7 insertions(+), 32 deletions(-)
arch/arm/mm/Kconfig +0 -1
@@ -900,7 +900,6 @@ config VDSO select GENERIC_TIME_VSYSCALL select GENERIC_VDSO_32 select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE help Place in the process address space an ELF shared object providing fast implementations of gettimeofday and
arch/arm64/Kconfig +0 -1
@@ -110,7 +110,6 @@ config ARM64 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select HANDLE_DOMAIN_IRQ select HARDIRQS_SW_RESEND select HAVE_PCI
arch/mips/Kconfig +0 -1
@@ -37,7 +37,6 @@ config MIPS select GENERIC_SCHED_CLOCK if !CAVIUM_OCTEON_SOC select GENERIC_SMP_IDLE_THREAD select GENERIC_TIME_VSYSCALL - select GENERIC_VDSO_CLOCK_MODE select GUP_GET_PTE_LOW_HIGH if CPU_MIPS32 && PHYS_ADDR_T_64BIT select HANDLE_DOMAIN_IRQ select HAVE_ARCH_COMPILER_H
arch/x86/Kconfig +0 -1
@@ -125,7 +125,6 @@ config X86 select GENERIC_STRNLEN_USER select GENERIC_TIME_VSYSCALL select GENERIC_GETTIMEOFDAY - select GENERIC_VDSO_CLOCK_MODE select GENERIC_VDSO_TIME_NS select GUP_GET_PTE_LOW_HIGH if X86_PAE select HARDLOCKUP_CHECK_TIMESTAMP if X86_64
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -18,13 +18,6 @@ static __always_inline bool __arch_update_vdso_data(void) }#endif /* __arch_update_vdso_data */ -#ifndef __arch_get_clock_mode -static __always_inline int __arch_get_clock_mode(struct timekeeper *tk) -{ -return 0 ; -} -#endif /* __arch_get_clock_mode */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
include/linux/clocksource.h +2 -2
@@ -24,13 +24,13 @@ struct clocksource;struct module ;#if defined(CONFIG_ARCH_CLOCKSOURCE_DATA ) || \ - defined(CONFIG_GENERIC_VDSO_CLOCK_MODE ) + defined(CONFIG_GENERIC_GETTIMEOFDAY )#include <asm/clocksource.h> #endif enum vdso_clock_mode {VDSO_CLOCKMODE_NONE , -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE +#ifdef CONFIG_GENERIC_GETTIMEOFDAY VDSO_ARCH_CLOCKMODES ,#endif VDSO_CLOCKMODE_MAX ,
kernel/time/vsyscall.c +0 -4
@@ -77,11 +77,7 @@ void update_vsyscall(struct timekeeper *tk)/* copy vsyscall data */ vdso_write_begin (vdata); -#ifdef CONFIG_GENERIC_VDSO_CLOCK_MODE clock_mode = tk->tkr_mono.clock->vdso_clock_mode; -#else - clock_mode =__arch_get_clock_mode (tk); -#endif vdata[CS_HRES_COARSE ].clock_mode = clock_mode; vdata[CS_RAW ].clock_mode = clock_mode;
lib/vdso/Kconfig +0 -3
@@ -30,7 +30,4 @@ config GENERIC_VDSO_TIME_NS Selected by architectures which support time namespaces in the VDSO -config GENERIC_VDSO_CLOCK_MODE - bool - endif
lib/vdso/gettimeofday.c +5 -12
@@ -65,16 +65,13 @@ static int do_hres_timens(const struct vdso_data *vdns, clockid_t clk, do { seq = vdso_read_begin(vd); - if (IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - vd->clock_mode == VDSO_CLOCKMODE_NONE) + + if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) return -1; + cycles = __arch_get_hw_counter(vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; - if (!IS_ENABLED(CONFIG_GENERIC_VDSO_CLOCK_MODE) && - unlikely((s64)cycles < 0)) - return -1; - ns += vdso_calc_delta(cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
@@ -137,16 +134,12 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, }smp_rmb (); -if (IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && - vd->clock_mode ==VDSO_CLOCKMODE_NONE ) +if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE ))return -1 ; + cycles =__arch_get_hw_counter (vd->clock_mode); ns = vdso_ts->nsec; last = vd->cycle_last; -if (!IS_ENABLED (CONFIG_GENERIC_VDSO_CLOCK_MODE ) && -unlikely ((s64 )cycles <0 )) -return -1 ; - ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult); ns >>= vd->shift; sec = vdso_ts->sec;
PATCH v2 13/17 lib/vdso: Avoid highres update if clocksource is not VDSO capable
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
If the current clocksource is not VDSO capable there is no point in updating the high resolution parts of the VDSO data.
Replace the architecture specific check with a check for a VDSO capable clocksource and skip the update if there is none.
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- include/asm-generic/vdso/vsyscall.h | 7 ------- kernel/time/vsyscall.c | 6 +++--- 3 files changed, 3 insertions(+), 17 deletions(-)
arch/arm/include/asm/vdso/vsyscall.h +0 -7
@@ -22,13 +22,6 @@ struct vdso_data *__arm_get_k_vdso_data(#define __arch_get_k_vdso_data __arm_get_k_vdso_datastatic __always_inline -bool __arm_update_vdso_data(void) -{ -return cntvct_ok; -} -#define __arch_update_vdso_data __arm_update_vdso_data - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -11,13 +11,6 @@ static __always_inline struct vdso_data }#endif /* __arch_get_k_vdso_data */ -#ifndef __arch_update_vdso_data -static __always_inline bool __arch_update_vdso_data(void) -{ -return true; -} -#endif /* __arch_update_vdso_data */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
kernel/time/vsyscall.c +3 -3
@@ -105,10 +105,10 @@ void update_vsyscall(struct timekeeper *WRITE_ONCE (vdata[CS_HRES_COARSE ].hrtimer_res, hrtimer_resolution);/* -* Architectures can opt out of updating the high resolution part -* of the VDSO. +* If the current clocksource is not VDSO capable, then spare the +* update of the high reolution parts. */ -if (__arch_update_vdso_data ()) +if (clock_mode !=VDSO_CLOCKMODE_NONE )update_vdso_data (vdata, tk);__arch_update_vsyscall (vdata, tk);
Re: [patch V2 13/17] lib/vdso: Avoid highres update if clocksource is not VDSO capable
Quoted 9 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
If the current clocksource is not VDSO capable there is no point in updating the high resolution parts of the VDSO data.
Replace the architecture specific check with a check for a VDSO capable clocksource and skip the update if there is none.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] lib/vdso: Avoid highres update if clocksource is not VDSO capable
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: ae3003fdec352e0d474a98ebf1617d48a945aa5a Gitweb: https://git.kernel.org/tip/ae3003fdec352e0d474a98ebf1617d48a945aa5a Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:00 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:26 +01:00
lib/vdso: Avoid highres update if clocksource is not VDSO capable
If the current clocksource is not VDSO capable there is no point in updating the high resolution parts of the VDSO data.
Replace the architecture specific check with a check for a VDSO capable clocksource and skip the update if there is none.
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- include/asm-generic/vdso/vsyscall.h | 7 ------- kernel/time/vsyscall.c | 6 +++--- 3 files changed, 3 insertions(+), 17 deletions(-)
arch/arm/include/asm/vdso/vsyscall.h +0 -7
@@ -22,13 +22,6 @@ struct vdso_data *__arm_get_k_vdso_data(void)#define __arch_get_k_vdso_data __arm_get_k_vdso_datastatic __always_inline -bool __arm_update_vdso_data(void) -{ -return cntvct_ok; -} -#define __arch_update_vdso_data __arm_update_vdso_data - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -11,13 +11,6 @@ static __always_inline struct vdso_data *__arch_get_k_vdso_data(void) }#endif /* __arch_get_k_vdso_data */ -#ifndef __arch_update_vdso_data -static __always_inline bool __arch_update_vdso_data(void) -{ -return true; -} -#endif /* __arch_update_vdso_data */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
kernel/time/vsyscall.c +3 -3
@@ -105,10 +105,10 @@ void update_vsyscall(struct timekeeper *tk)WRITE_ONCE (vdata[CS_HRES_COARSE ].hrtimer_res, hrtimer_resolution);/* -* Architectures can opt out of updating the high resolution part -* of the VDSO. +* If the current clocksource is not VDSO capable, then spare the +* update of the high reolution parts. */ -if (__arch_update_vdso_data ()) +if (clock_mode !=VDSO_CLOCKMODE_NONE )update_vdso_data (vdata, tk);__arch_update_vsyscall (vdata, tk);
[tip: timers/core] lib/vdso: Avoid highres update if clocksource is not VDSO capable
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: c7a18100bdffdff440c7291db6e80863fab0461e Gitweb: https://git.kernel.org/tip/c7a18100bdffdff440c7291db6e80863fab0461e Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:00 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:17 +01:00
lib/vdso: Avoid highres update if clocksource is not VDSO capable
If the current clocksource is not VDSO capable there is no point in updating the high resolution parts of the VDSO data.
Replace the architecture specific check with a check for a VDSO capable clocksource and skip the update if there is none.
--- arch/arm/include/asm/vdso/vsyscall.h | 7 ------- include/asm-generic/vdso/vsyscall.h | 7 ------- kernel/time/vsyscall.c | 6 +++--- 3 files changed, 3 insertions(+), 17 deletions(-)
arch/arm/include/asm/vdso/vsyscall.h +0 -7
@@ -22,13 +22,6 @@ struct vdso_data *__arm_get_k_vdso_data(void)#define __arch_get_k_vdso_data __arm_get_k_vdso_datastatic __always_inline -bool __arm_update_vdso_data(void) -{ -return cntvct_ok; -} -#define __arch_update_vdso_data __arm_update_vdso_data - -static __always_inline void __arm_sync_vdso_data(struct vdso_data *vdata) {flush_dcache_page (virt_to_page (vdata));
include/asm-generic/vdso/vsyscall.h +0 -7
@@ -11,13 +11,6 @@ static __always_inline struct vdso_data *__arch_get_k_vdso_data(void) }#endif /* __arch_get_k_vdso_data */ -#ifndef __arch_update_vdso_data -static __always_inline bool __arch_update_vdso_data(void) -{ -return true; -} -#endif /* __arch_update_vdso_data */ -#ifndef __arch_update_vsyscall static __always_inline void __arch_update_vsyscall(struct vdso_data *vdata,struct timekeeper *tk)
kernel/time/vsyscall.c +3 -3
@@ -105,10 +105,10 @@ void update_vsyscall(struct timekeeper *tk)WRITE_ONCE (vdata[CS_HRES_COARSE ].hrtimer_res, hrtimer_resolution);/* -* Architectures can opt out of updating the high resolution part -* of the VDSO. +* If the current clocksource is not VDSO capable, then spare the +* update of the high reolution parts. */ -if (__arch_update_vdso_data ()) +if (clock_mode !=VDSO_CLOCKMODE_NONE )update_vdso_data (vdata, tk);__arch_update_vsyscall (vdata, tk);
PATCH v2 14/17 lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
In reply to Thomas GleixnerFrom: Thomas Gleixner <[email protected]>
Move the time namespace indicator clock mode to the other ones for consistency sake.
--- include/linux/clocksource.h | 3 +++ include/vdso/datapage.h | 2 -- kernel/time/namespace.c | 7 ++++--- lib/vdso/gettimeofday.c | 18 ++++++++++-------- 4 files changed, 17 insertions(+), 13 deletions(-)
include/linux/clocksource.h +3 -0
@@ -34,6 +34,9 @@ enum vdso_clock_mode {VDSO_ARCH_CLOCKMODES , #endifVDSO_CLOCKMODE_MAX , + +/* Indicator for time namespace VDSO */ +VDSO_CLOCKMODE_TIMENS =INT_MAX };/**
include/vdso/datapage.h +0 -2
@@ -21,8 +21,6 @@#define CS_RAW 1#define CS_BASES (CS_RAW + 1) -#define VCLOCK_TIMENS UINT_MAX -/** * struct vdso_timestamp - basetime per clock_id * @sec: seconds
kernel/time/namespace.c +4 -3
@@ -8,6 +8,7 @@#include <linux/user_namespace.h> #include <linux/sched/signal.h> #include <linux/sched/task.h> +#include <linux/clocksource.h> #include <linux/seq_file.h> #include <linux/proc_ns.h> #include <linux/export.h>
@@ -172,8 +173,8 @@ static struct timens_offset offset_from_* for vdso_data->clock_mode is a non-issue. The task is spin waiting for the * update to finish and for 'seq' to become even anyway. * -* Timens page has vdso_data->clock_mode set to VCLOCK_TIMENS which enforces -* the time namespace handling path. +* Timens page has vdso_data->clock_mode set to VDSO_CLOCKMODE_TIMENS which +* enforces the time namespace handling path. */ static void timens_setup_vdso_data (struct vdso_data *vdata,struct time_namespace *ns)
@@ -183,7 +184,7 @@ static void timens_setup_vdso_data(strucstruct timens_offset boottime =offset_from_ts (ns->offsets.boottime); vdata->seq =1 ; - vdata->clock_mode =VCLOCK_TIMENS ; + vdata->clock_mode =VDSO_CLOCKMODE_TIMENS ; offset[CLOCK_MONOTONIC ] = monotonic; offset[CLOCK_MONOTONIC_RAW ] = monotonic; offset[CLOCK_MONOTONIC_COARSE ] = monotonic;
lib/vdso/gettimeofday.c +10 -8
@@ -116,10 +116,10 @@ static __always_inline int do_hres(const do { /* - * Open coded to handle VCLOCK_TIMENS. Time namespace + * Open coded to handle VDSO_CLOCKMODE_TIMENS. Time namespace * enabled tasks have a special VVAR page installed which * has vd->seq set to 1 and vd->clock_mode set to - * VCLOCK_TIMENS. For non time namespace affected tasks + * VDSO_CLOCKMODE_TIMENS. For non time namespace affected tasks * this does not affect performance because if vd->seq is * odd, i.e. a concurrent update is in progress the extra * check for vd->clock_mode is just a few extra
@@ -128,7 +128,7 @@ static __always_inline int do_hres(const*/ while (unlikely ((seq =READ_ONCE (vd->seq)) &1 )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS )return do_hres_timens (vd, clk, ts);cpu_relax (); }
@@ -200,12 +200,12 @@ static __always_inline int do_coarse(con do { /* - * Open coded to handle VCLOCK_TIMENS. See comment in + * Open coded to handle VDSO_CLOCK_TIMENS. See comment in * do_hres(). */ while ((seq = READ_ONCE(vd->seq)) & 1) { if (IS_ENABLED(CONFIG_TIME_NS) && - vd->clock_mode == VCLOCK_TIMENS) + vd->clock_mode == VDSO_CLOCKMODE_TIMENS) return do_coarse_timens(vd, clk, ts); cpu_relax(); }
@@ -292,7 +292,7 @@ static __maybe_unused intif (unlikely (tz !=NULL )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); tz->tz_minuteswest = vd[CS_HRES_COARSE ].tz_minuteswest;
@@ -308,7 +308,8 @@ static __maybe_unused __kernel_old_time_const struct vdso_data *vd =__arch_get_vdso_data ();__kernel_old_time_t t; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); t =READ_ONCE (vd[CS_HRES_COARSE ].basetime[CLOCK_REALTIME ].sec);
@@ -332,7 +333,8 @@ int __cvdso_clock_getres_common(clockid_if (unlikely ((u32 ) clock >=MAX_CLOCKS ))return -1 ; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data ();/*
Re: [patch V2 14/17] lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
Quoted 7 lines · Thomas Gleixner
From: Thomas Gleixner <[email protected]>
Move the time namespace indicator clock mode to the other ones for consistency sake.
Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 80bebf447de9fe2d7a0fe4dd795aa057e2293205 Gitweb: https://git.kernel.org/tip/80bebf447de9fe2d7a0fe4dd795aa057e2293205 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:01 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:27 +01:00
lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
Move the time namespace indicator clock mode to the other ones for consistency sake.
--- include/linux/clocksource.h | 3 +++ include/vdso/datapage.h | 2 -- kernel/time/namespace.c | 7 ++++--- lib/vdso/gettimeofday.c | 18 ++++++++++-------- 4 files changed, 17 insertions(+), 13 deletions(-)
include/linux/clocksource.h +3 -0
@@ -34,6 +34,9 @@ enum vdso_clock_mode {VDSO_ARCH_CLOCKMODES , #endifVDSO_CLOCKMODE_MAX , + +/* Indicator for time namespace VDSO */ +VDSO_CLOCKMODE_TIMENS =INT_MAX };/**
include/vdso/datapage.h +0 -2
@@ -21,8 +21,6 @@#define CS_RAW 1#define CS_BASES (CS_RAW + 1) -#define VCLOCK_TIMENS UINT_MAX -/** * struct vdso_timestamp - basetime per clock_id * @sec: seconds
kernel/time/namespace.c +4 -3
@@ -8,6 +8,7 @@#include <linux/user_namespace.h> #include <linux/sched/signal.h> #include <linux/sched/task.h> +#include <linux/clocksource.h> #include <linux/seq_file.h> #include <linux/proc_ns.h> #include <linux/export.h>
@@ -172,8 +173,8 @@ static struct timens_offset offset_from_ts(struct timespec64 off)* for vdso_data->clock_mode is a non-issue. The task is spin waiting for the * update to finish and for 'seq' to become even anyway. * -* Timens page has vdso_data->clock_mode set to VCLOCK_TIMENS which enforces -* the time namespace handling path. +* Timens page has vdso_data->clock_mode set to VDSO_CLOCKMODE_TIMENS which +* enforces the time namespace handling path. */ static void timens_setup_vdso_data (struct vdso_data *vdata,struct time_namespace *ns)
@@ -183,7 +184,7 @@ static void timens_setup_vdso_data(struct vdso_data *vdata,struct timens_offset boottime =offset_from_ts (ns->offsets.boottime); vdata->seq =1 ; - vdata->clock_mode =VCLOCK_TIMENS ; + vdata->clock_mode =VDSO_CLOCKMODE_TIMENS ; offset[CLOCK_MONOTONIC ] = monotonic; offset[CLOCK_MONOTONIC_RAW ] = monotonic; offset[CLOCK_MONOTONIC_COARSE ] = monotonic;
lib/vdso/gettimeofday.c +10 -8
@@ -116,10 +116,10 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, do { /* - * Open coded to handle VCLOCK_TIMENS. Time namespace + * Open coded to handle VDSO_CLOCKMODE_TIMENS. Time namespace * enabled tasks have a special VVAR page installed which * has vd->seq set to 1 and vd->clock_mode set to - * VCLOCK_TIMENS. For non time namespace affected tasks + * VDSO_CLOCKMODE_TIMENS. For non time namespace affected tasks * this does not affect performance because if vd->seq is * odd, i.e. a concurrent update is in progress the extra * check for vd->clock_mode is just a few extra
@@ -128,7 +128,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk,*/ while (unlikely ((seq =READ_ONCE (vd->seq)) &1 )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS )return do_hres_timens (vd, clk, ts);cpu_relax (); }
@@ -200,12 +200,12 @@ static __always_inline int do_coarse(const struct vdso_data *vd, clockid_t clk, do { /* - * Open coded to handle VCLOCK_TIMENS. See comment in + * Open coded to handle VDSO_CLOCK_TIMENS. See comment in * do_hres(). */ while ((seq = READ_ONCE(vd->seq)) & 1) { if (IS_ENABLED(CONFIG_TIME_NS) && - vd->clock_mode == VCLOCK_TIMENS) + vd->clock_mode == VDSO_CLOCKMODE_TIMENS) return do_coarse_timens(vd, clk, ts); cpu_relax(); }
@@ -292,7 +292,7 @@ __cvdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz)if (unlikely (tz !=NULL )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); tz->tz_minuteswest = vd[CS_HRES_COARSE ].tz_minuteswest;
@@ -308,7 +308,8 @@ static __maybe_unused __kernel_old_time_t __cvdso_time(__kernel_old_time_t *timeconst struct vdso_data *vd =__arch_get_vdso_data ();__kernel_old_time_t t; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); t =READ_ONCE (vd[CS_HRES_COARSE ].basetime[CLOCK_REALTIME ].sec);
@@ -332,7 +333,8 @@ int __cvdso_clock_getres_common(clockid_t clock, struct __kernel_timespec *res)if (unlikely ((u32 ) clock >=MAX_CLOCKS ))return -1 ; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data ();/*
[tip: timers/core] lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 2d6b01bd88ccabba06d342ef80eaab6b39d12497 Gitweb: https://git.kernel.org/tip/2d6b01bd88ccabba06d342ef80eaab6b39d12497 Author: Thomas Gleixner <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:01 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:17 +01:00
lib/vdso: Move VCLOCK_TIMENS to vdso_clock_modes
Move the time namespace indicator clock mode to the other ones for consistency sake.
--- include/linux/clocksource.h | 3 +++ include/vdso/datapage.h | 2 -- kernel/time/namespace.c | 7 ++++--- lib/vdso/gettimeofday.c | 18 ++++++++++-------- 4 files changed, 17 insertions(+), 13 deletions(-)
include/linux/clocksource.h +3 -0
@@ -34,6 +34,9 @@ enum vdso_clock_mode {VDSO_ARCH_CLOCKMODES , #endifVDSO_CLOCKMODE_MAX , + +/* Indicator for time namespace VDSO */ +VDSO_CLOCKMODE_TIMENS =INT_MAX };/**
include/vdso/datapage.h +0 -2
@@ -21,8 +21,6 @@#define CS_RAW 1#define CS_BASES (CS_RAW + 1) -#define VCLOCK_TIMENS UINT_MAX -/** * struct vdso_timestamp - basetime per clock_id * @sec: seconds
kernel/time/namespace.c +4 -3
@@ -8,6 +8,7 @@#include <linux/user_namespace.h> #include <linux/sched/signal.h> #include <linux/sched/task.h> +#include <linux/clocksource.h> #include <linux/seq_file.h> #include <linux/proc_ns.h> #include <linux/export.h>
@@ -172,8 +173,8 @@ static struct timens_offset offset_from_ts(struct timespec64 off)* for vdso_data->clock_mode is a non-issue. The task is spin waiting for the * update to finish and for 'seq' to become even anyway. * -* Timens page has vdso_data->clock_mode set to VCLOCK_TIMENS which enforces -* the time namespace handling path. +* Timens page has vdso_data->clock_mode set to VDSO_CLOCKMODE_TIMENS which +* enforces the time namespace handling path. */ static void timens_setup_vdso_data (struct vdso_data *vdata,struct time_namespace *ns)
@@ -183,7 +184,7 @@ static void timens_setup_vdso_data(struct vdso_data *vdata,struct timens_offset boottime =offset_from_ts (ns->offsets.boottime); vdata->seq =1 ; - vdata->clock_mode =VCLOCK_TIMENS ; + vdata->clock_mode =VDSO_CLOCKMODE_TIMENS ; offset[CLOCK_MONOTONIC ] = monotonic; offset[CLOCK_MONOTONIC_RAW ] = monotonic; offset[CLOCK_MONOTONIC_COARSE ] = monotonic;
lib/vdso/gettimeofday.c +10 -8
@@ -116,10 +116,10 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, do { /* - * Open coded to handle VCLOCK_TIMENS. Time namespace + * Open coded to handle VDSO_CLOCKMODE_TIMENS. Time namespace * enabled tasks have a special VVAR page installed which * has vd->seq set to 1 and vd->clock_mode set to - * VCLOCK_TIMENS. For non time namespace affected tasks + * VDSO_CLOCKMODE_TIMENS. For non time namespace affected tasks * this does not affect performance because if vd->seq is * odd, i.e. a concurrent update is in progress the extra * check for vd->clock_mode is just a few extra
@@ -128,7 +128,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk,*/ while (unlikely ((seq =READ_ONCE (vd->seq)) &1 )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS )return do_hres_timens (vd, clk, ts);cpu_relax (); }
@@ -200,12 +200,12 @@ static __always_inline int do_coarse(const struct vdso_data *vd, clockid_t clk, do { /* - * Open coded to handle VCLOCK_TIMENS. See comment in + * Open coded to handle VDSO_CLOCK_TIMENS. See comment in * do_hres(). */ while ((seq = READ_ONCE(vd->seq)) & 1) { if (IS_ENABLED(CONFIG_TIME_NS) && - vd->clock_mode == VCLOCK_TIMENS) + vd->clock_mode == VDSO_CLOCKMODE_TIMENS) return do_coarse_timens(vd, clk, ts); cpu_relax(); }
@@ -292,7 +292,7 @@ __cvdso_gettimeofday(struct __kernel_old_timeval *tv, struct timezone *tz)if (unlikely (tz !=NULL )) {if (IS_ENABLED (CONFIG_TIME_NS ) && - vd->clock_mode ==VCLOCK_TIMENS ) + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); tz->tz_minuteswest = vd[CS_HRES_COARSE ].tz_minuteswest;
@@ -308,7 +308,8 @@ static __maybe_unused __kernel_old_time_t __cvdso_time(__kernel_old_time_t *timeconst struct vdso_data *vd =__arch_get_vdso_data ();__kernel_old_time_t t; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data (); t =READ_ONCE (vd[CS_HRES_COARSE ].basetime[CLOCK_REALTIME ].sec);
@@ -332,7 +333,8 @@ int __cvdso_clock_getres_common(clockid_t clock, struct __kernel_timespec *res)if (unlikely ((u32 ) clock >=MAX_CLOCKS ))return -1 ; -if (IS_ENABLED (CONFIG_TIME_NS ) && vd->clock_mode ==VCLOCK_TIMENS ) +if (IS_ENABLED (CONFIG_TIME_NS ) && + vd->clock_mode ==VDSO_CLOCKMODE_TIMENS ) vd =__arch_get_timens_vdso_data ();/*
PATCH v2 15/17 lib/vdso: Allow fixed clock mode
In reply to Thomas GleixnerFrom: Christophe Leroy <[email protected]>
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases the clock mode check in the VDSO code is pointless.
Move the check for a VDSO capable clocksource into an inline function and allow architectures to redefine it via a macro.
[ tglx: Removed the #ifdef mess ]
--- lib/vdso/gettimeofday.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/vdso/gettimeofday.c +9 -2
@@ -46,6 +46,13 @@ static inline bool __arch_vdso_hres_capa }#endif +#ifndef vdso_clocksource_ok +static inline bool vdso_clocksource_ok (const struct vdso_data *vd) +{ +return vd->clock_mode !=VDSO_CLOCKMODE_NONE ; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)
@@ -66,7 +73,7 @@ static int do_hres_timens(const struct v do { seq = vdso_read_begin(vd); - if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) + if (unlikely(!vdso_clocksource_ok(vd))) return -1; cycles = __arch_get_hw_counter(vd->clock_mode);
@@ -134,7 +141,7 @@ static __always_inline int do_hres(const }smp_rmb (); -if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE )) +if (unlikely (!vdso_clocksource_ok (vd)))return -1 ; cycles =__arch_get_hw_counter (vd->clock_mode);
Re: [patch V2 15/17] lib/vdso: Allow fixed clock mode
Quoted 6 lines · Thomas Gleixner
From: Christophe Leroy <[email protected]>
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases the clock mode check in the VDSO code is pointless.
Quoted 8 lines
Move the check for a VDSO capable clocksource into an inline function and allow architectures to redefine it via a macro.
[ tglx: Removed the #ifdef mess ]
Signed-off-by Christophe Leroy <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>
[...]
Signature · 2 lines
Regards, Vincenzo
[tip: timers/core] lib/vdso: Allow fixed clock mode
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: 997cf1781d0a694952048de2705f755d2c8c1bcd Gitweb: https://git.kernel.org/tip/997cf1781d0a694952048de2705f755d2c8c1bcd Author: Christophe Leroy <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:02 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 14:40:28 +01:00
lib/vdso: Allow fixed clock mode
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases the clock mode check in the VDSO code is pointless.
Move the check for a VDSO capable clocksource into an inline function and allow architectures to redefine it via a macro.
[ tglx: Removed the #ifdef mess ]
--- lib/vdso/gettimeofday.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/vdso/gettimeofday.c +9 -2
@@ -46,6 +46,13 @@ static inline bool __arch_vdso_hres_capable(void) }#endif +#ifndef vdso_clocksource_ok +static inline bool vdso_clocksource_ok (const struct vdso_data *vd) +{ +return vd->clock_mode !=VDSO_CLOCKMODE_NONE ; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)
@@ -66,7 +73,7 @@ static int do_hres_timens(const struct vdso_data *vdns, clockid_t clk, do { seq = vdso_read_begin(vd); - if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) + if (unlikely(!vdso_clocksource_ok(vd))) return -1; cycles = __arch_get_hw_counter(vd->clock_mode);
@@ -134,7 +141,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, }smp_rmb (); -if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE )) +if (unlikely (!vdso_clocksource_ok (vd)))return -1 ; cycles =__arch_get_hw_counter (vd->clock_mode);
[tip: timers/core] lib/vdso: Allow fixed clock mode
In reply to Thomas GleixnerThe following commit has been merged into the timers/core branch of tip:
The following commit has been merged into the timers/core branch of tip:
Commit-ID: ae12e08539de6717502c2f9f83bd60df939b5c08 Gitweb: https://git.kernel.org/tip/ae12e08539de6717502c2f9f83bd60df939b5c08 Author: Christophe Leroy <[email protected]> AuthorDate: Fri, 07 Feb 2020 13:39:02 +01:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 17 Feb 2020 20:12:18 +01:00
lib/vdso: Allow fixed clock mode
Some architectures have a fixed clocksource which is known at compile time and cannot be replaced or disabled at runtime, e.g. timebase on PowerPC. For such cases the clock mode check in the VDSO code is pointless.
Move the check for a VDSO capable clocksource into an inline function and allow architectures to redefine it via a macro.
[ tglx: Removed the #ifdef mess ]
--- lib/vdso/gettimeofday.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/vdso/gettimeofday.c +9 -2
@@ -46,6 +46,13 @@ static inline bool __arch_vdso_hres_capable(void) }#endif +#ifndef vdso_clocksource_ok +static inline bool vdso_clocksource_ok (const struct vdso_data *vd) +{ +return vd->clock_mode !=VDSO_CLOCKMODE_NONE ; +} +#endif +#ifdef CONFIG_TIME_NS static int do_hres_timens (const struct vdso_data *vdns,clockid_t clk,struct __kernel_timespec *ts)
@@ -66,7 +73,7 @@ static int do_hres_timens(const struct vdso_data *vdns, clockid_t clk, do { seq = vdso_read_begin(vd); - if (unlikely(vd->clock_mode == VDSO_CLOCKMODE_NONE)) + if (unlikely(!vdso_clocksource_ok(vd))) return -1; cycles = __arch_get_hw_counter(vd->clock_mode);
@@ -134,7 +141,7 @@ static __always_inline int do_hres(const struct vdso_data *vd, clockid_t clk, }smp_rmb (); -if (unlikely (vd->clock_mode ==VDSO_CLOCKMODE_NONE )) +if (unlikely (!vdso_clocksource_ok (vd)))return -1 ; cycles =__arch_get_hw_counter (vd->clock_mode);
PATCH v2 16/17 lib/vdso: Allow architectures to override the ns shift operation
In reply to Thomas GleixnerFrom: Christophe Leroy <[email protected]>
On powerpc/32, GCC (8.1) generates pretty bad code for the ns >>= vd->shift operation taking into account that the shift is always <= 32 and the upper part of the result is likely to be zero. GCC makes reversed assumptions considering the shift to be likely >= 32 and the upper part to be like not zero.
unsigned long long shift(unsigned long long x, unsigned char s)
{
return x >> s;
}results in:
00000018 <shift>:
18: 35 25 ff e0 addic. r9,r5,-32 1c: 41 80 00 10 blt 2c <shift+0x14> 20: 7c 64 4c 30 srw r4,r3,r9 24: 38 60 00 00 li r3,0 28: 4e 80 00 20 blr 2c: 54 69 08 3c rlwinm r9,r3,1,0,30 30: 21 45 00 1f subfic r10,r5,31 34: 7c 84 2c 30 srw r4,r4,r5 38: 7d 29 50 30 slw r9,r9,r10 3c: 7c 63 2c 30 srw r3,r3,r5 40: 7d 24 23 78 or r4,r9,r4
44: 4e 80 00 20 blr
Even when forcing the shift to be smaller than 32 with an &= 31, it still considers the shift as likely >= 32.
Move the default shift implementation into an inline which can be redefined in architecture code via a macro.
[ tglx: Made the shift argument u32 and removed the __arch prefix ]
--- lib/vdso/gettimeofday.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
lib/vdso/gettimeofday.c +9 -2
@@ -39,6 +39,13 @@ u64 vdso_calc_delta(u64 cycles, u64 last }#endif +#ifndef vdso_shift_ns +static __always_inline u64 vdso_shift_ns(u64 ns, u32 shift) +{ +return ns >> shift; +} +#endif +#ifndef __arch_vdso_hres_capablestatic inline bool __arch_vdso_hres_capable (void ) {
@@ -80,7 +87,7 @@ static int do_hres_timens(const struct v ns = vdso_ts->nsec; last = vd->cycle_last; ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult); - ns >>= vd->shift; + ns =vdso_shift_ns (ns, vd->shift); sec = vdso_ts->sec; }while (unlikely (vdso_read_retry (vd, seq)));
@@ -148,7 +155,7 @@ static __always_inline int do_hres(const ns = vdso_ts->nsec; last = vd->cycle_last; ns +=vdso_calc_delta (cycles, last, vd->mask, vd->mult); - ns >>= vd->shift; + ns =vdso_shift_ns (ns, vd->shift); sec = vdso_ts->sec; }while (unlikely (vdso_read_retry (vd, seq)));
Re: [patch V2 16/17] lib/vdso: Allow architectures to override the ns shift operation
Quoted 32 lines · Thomas Gleixner
From: Christophe Leroy <[email protected]>
On powerpc/32, GCC (8.1) generates pretty bad code for the ns >>=
vd->shiftoperation taking into account that the shift is always <= 32 and the upper part of the result is likely to be zero. GCC makes reversed assumptions considering the shift to be likely >= 32 and the upper part to be like not zero.unsigned long long
shift(unsigned long long x, unsigned char s){ return x >> s; }results in:
00000018 <shift>:
18: 35 25 ff e0 addic. r9,r5,-32 1c: 41 80 00 10 blt 2c <shift+0x14> 20: 7c 64 4c 30 srw r4,r3,r9 24: 38 60 00 00 li r3,0 28: 4e 80 00 20 blr 2c: 54 69 08 3c rlwinm r9,r3,1,0,30 30: 21 45 00 1f subfic r10,r5,31 34: 7c 84 2c 30 srw r4,r4,r5 38: 7d 29 50 30 slw r9,r9,r10 3c: 7c 63 2c 30 srw r3,r3,r5 40: 7d 24 23 78 or r4,r9,r444: 4e 80 00 20 blr
Even when forcing the shift to be smaller than 32 with an &= 31, it still considers the shift as likely >= 32.
Quoted 9 lines
Move the default shift implementation into an inline which can be redefined in architecture code via a macro.
[ tglx: Made the shift argument u32 and removed the
__archprefix ]
[...]
Signature · 2 lines
Regards, Vincenzo
PATCH v2 17/17 lib/vdso: Allow architectures to provide the vdso data pointer
In reply to Thomas GleixnerFrom: Christophe Leroy <[email protected]>
On powerpc, __arch_get_vdso_data() clobbers the link register, requiring the caller to save it.
As the parent function already has to set a stack frame and saves the link register before calling the C vdso function, retrieving the vdso data pointer there is less overhead.
Split out the functional code from the __cvdso.*() interfaces into new static functions which can either be called from the existing interfaces with the vdso data pointer supplied via __arch_get_vdso_data() or directly from ASM code.
--- lib/vdso/gettimeofday.c | 72 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 56 insertions(+), 16 deletions(-)
lib/vdso/gettimeofday.c +56 -16
@@ -233,9 +233,9 @@ static __always_inline int do_coarse(con }static __maybe_unused int -__cvdso_clock_gettime_common (clockid_t clock,struct __kernel_timespec *ts) +__cvdso_clock_gettime_common (const struct vdso_data *vd,clockid_t clock, +struct __kernel_timespec *ts) { -const struct vdso_data *vd =__arch_get_vdso_data ();u32 msk;/* Check for negative values or invalid clocks */
@@ -260,23 +260,31 @@ static __maybe_unused int }static __maybe_unused int -__cvdso_clock_gettime (clockid_t clock,struct __kernel_timespec *ts) +__cvdso_clock_gettime_data (const struct vdso_data *vd,clockid_t clock, +struct __kernel_timespec *ts) { -int ret =__cvdso_clock_gettime_common (clock, ts); +int ret =__cvdso_clock_gettime_common (vd, clock, ts);if (unlikely (ret))return clock_gettime_fallback (clock, ts);return 0 ; } +static __maybe_unused int +__cvdso_clock_gettime (clockid_t clock,struct __kernel_timespec *ts) +{ +return __cvdso_clock_gettime_data (__arch_get_vdso_data (), clock, ts); +} +#ifdef BUILD_VDSO32 static __maybe_unused int -__cvdso_clock_gettime32 (clockid_t clock,struct old_timespec32 *res) +__cvdso_clock_gettime32_data (const struct vdso_data *vd,clockid_t clock, +struct old_timespec32 *res) {struct __kernel_timespec ts;int ret; - ret =__cvdso_clock_gettime_common (clock, &ts); + ret =__cvdso_clock_gettime_common (vd, clock, &ts);if (unlikely (ret))return clock_gettime32_fallback (clock, res);
@@ -287,12 +295,18 @@ static __maybe_unused int return ret; } + +static __maybe_unused int +__cvdso_clock_gettime32 (clockid_t clock,struct old_timespec32 *res) +{ +return __cvdso_clock_gettime32_data (__arch_get_vdso_data (), clock, res); +}#endif /* BUILD_VDSO32 */ static __maybe_unused int -__cvdso_gettimeofday (struct __kernel_old_timeval *tv,struct timezone *tz) +__cvdso_gettimeofday_data (const struct vdso_data *vd, +struct __kernel_old_timeval *tv,struct timezone *tz) { -const struct vdso_data *vd =__arch_get_vdso_data ();if (likely (tv !=NULL )) {struct __kernel_timespec ts;
@@ -316,10 +330,16 @@ static __maybe_unused intreturn 0 ; } +static __maybe_unused int +__cvdso_gettimeofday (struct __kernel_old_timeval *tv,struct timezone *tz) +{ +return __cvdso_gettimeofday_data (__arch_get_vdso_data (), tv, tz); +} +#ifdef VDSO_HAS_TIME -static __maybe_unused __kernel_old_time_t __cvdso_time (__kernel_old_time_t *time) +static __maybe_unused __kernel_old_time_t +__cvdso_time_data (const struct vdso_data *vd,__kernel_old_time_t *time) { -const struct vdso_data *vd =__arch_get_vdso_data ();__kernel_old_time_t t; if (IS_ENABLED(CONFIG_TIME_NS) &&
@@ -333,13 +353,18 @@ static __maybe_unused __kernel_old_time_return t; } + +static __maybe_unused __kernel_old_time_t __cvdso_time (__kernel_old_time_t *time) +{ +return __cvdso_time_data (__arch_get_vdso_data (), time); +}#endif /* VDSO_HAS_TIME */ #ifdef VDSO_HAS_CLOCK_GETRES static __maybe_unused -int __cvdso_clock_getres_common (clockid_t clock,struct __kernel_timespec *res) +int __cvdso_clock_getres_common (const struct vdso_data *vd,clockid_t clock, +struct __kernel_timespec *res) { -const struct vdso_data *vd =__arch_get_vdso_data ();u32 msk;u64 ns;
@@ -378,23 +403,31 @@ int __cvdso_clock_getres_common(clockid_ }static __maybe_unused -int __cvdso_clock_getres (clockid_t clock,struct __kernel_timespec *res) +int __cvdso_clock_getres_data (const struct vdso_data *vd,clockid_t clock, +struct __kernel_timespec *res) { -int ret =__cvdso_clock_getres_common (clock, res); +int ret =__cvdso_clock_getres_common (vd, clock, res);if (unlikely (ret))return clock_getres_fallback (clock, res);return 0 ; } +static __maybe_unused +int __cvdso_clock_getres (clockid_t clock,struct __kernel_timespec *res) +{ +return __cvdso_clock_getres_data (__arch_get_vdso_data (), clock, res); +} +#ifdef BUILD_VDSO32 static __maybe_unused int -__cvdso_clock_getres_time32 (clockid_t clock,struct old_timespec32 *res) +__cvdso_clock_getres_time32_data (const struct vdso_data *vd,clockid_t clock, +struct old_timespec32 *res) {struct __kernel_timespec ts;int ret; - ret =__cvdso_clock_getres_common (clock, &ts); + ret =__cvdso_clock_getres_common (vd, clock, &ts);if (unlikely (ret))return clock_getres32_fallback (clock, res);
@@ -405,5 +438,12 @@ static __maybe_unused int }return ret; } + +static __maybe_unused int +__cvdso_clock_getres_time32 (clockid_t clock,struct old_timespec32 *res) +{ +return __cvdso_clock_getres_time32_data (__arch_get_vdso_data (), + clock, res); +}#endif /* BUILD_VDSO32 */ #endif /* VDSO_HAS_CLOCK_GETRES */
Re: [patch V2 17/17] lib/vdso: Allow architectures to provide the vdso data pointer
Quoted 9 lines · Thomas Gleixner
From: Christophe Leroy <[email protected]>
On powerpc,
__arch_get_vdso_data()clobbers the link register, requiring the caller to save it.As the parent function already has to set a stack frame and saves the link register before calling the C vdso function, retrieving the vdso data pointer there is less overhead.
Quoted 9 lines
Split out the functional code from the __cvdso.*() interfaces into new static functions which can either be called from the existing interfaces with the vdso data pointer supplied via
__arch_get_vdso_data()or directly from ASM code.
[...]
Signature · 2 lines
Regards, Vincenzo