kvm: Use host timekeeping in guest.
Subsystems (6)
cover kvm: Use host timekeeping in guest.
This RFC is to try to solve the following problem:
We have some applications that are currently running in their own namespace, that still talk to other processes on the machine, using IPC, and expect to run on the same machine.
We want to move them into a virtual machine, for the usual benefits of virtualization.
However, some of these programs use CLOCK_MONOTONIC and CLOCK_BOOTTIME timestamps, as part of their protocol, when talking to the host.
Generally speaking, we have multiple event sources, for example sensors, input devices, display controller vsync, etc and we would like to rely on them in the guest for various scenarios.
As a specific example, we are trying to run some wayland clients (in the guest) who talk to the server (in the host), and the server gives input events based on host time. Additionally, there are also vsync events that the clients use for timing their rendering.
Another use case we have are timestamps from IIO sensors and cameras. There are applications that need to determine how the timestamps relate to the current time and the only way to get current time is clock_gettime(), which would return a value from a different time domain than the timestamps.
In this case, it is not feasible to change these programs, due to the number of the places we would have to change.
We spent some time thinking about this, and the best solution we could come up with was the following:
Make the guest kernel return the same CLOCK_MONOTONIC and CLOCK_GETTIME timestamps as the host.
To do that, I am changing kvmclock to request to the host to copy its timekeeping parameters (mult, base, cycle_last, etc), so that the guest timekeeper can use the same values, so that time can be synchronized between the guest and the host.
Any suggestions or feedback would be highly appreciated.
Suleiman Souhlal (2):
kvm: Mechanism to copy host timekeeping parameters into guest. x86/kvmclock: Use host timekeeping.
arch/x86/Kconfig | 9 ++ arch/x86/include/asm/kvm_host.h | 3 + arch/x86/include/asm/kvmclock.h | 2 + arch/x86/include/asm/pvclock-abi.h | 27 ++++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kernel/kvmclock.c | 127 ++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 121 +++++++++++++++++++++++++ kernel/time/timekeeping.c | 21 +++++ 8 files changed, 307 insertions(+), 4 deletions(-)
Signature · 1 line
2.23.0.237.gc6a4ce50a0-goog
RFC 1/2 kvm: Mechanism to copy host timekeeping parameters into guest.
This is used to synchronize time between host and guest. The guest can request the (guest) physical address it wants the data in through the MSR_KVM_TIMEKEEPER_EN MSR.
We maintain a shadow copy of the timekeeper that gets updated whenever the timekeeper gets updated, and then copied into the guest.
It currently assumes the host timekeeper is "tsc".
--- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/include/asm/pvclock-abi.h | 27 ++++++ arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/x86.c | 121 +++++++++++++++++++++++++++ 4 files changed, 152 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -666,7 +666,10 @@ struct kvm_vcpu_arch {struct pvclock_vcpu_time_info hv_clock;unsigned int hw_tsc_khz;struct gfn_to_hva_cache pv_time; +struct gfn_to_hva_cache pv_timekeeper_g2h; +struct pvclock_timekeeper pv_timekeeper;bool pv_time_enabled; +bool pv_timekeeper_enabled;/* set guest stopped flag in pvclock flags field */ bool pvclock_set_guest_stopped_request;
arch/x86/include/asm/pvclock-abi.h +27 -0
@@ -40,6 +40,33 @@ struct pvclock_wall_clock {u32 nsec; } __attribute__((__packed__)); +struct pvclock_read_base { +u64 mask; +u64 cycle_last; +u32 mult; +u32 shift; +u64 xtime_nsec; +u64 base; +} __attribute__((__packed__ )); + +struct pvclock_timekeeper { +u64 gen; +u64 flags; +struct pvclock_read_base tkr_mono; +struct pvclock_read_base tkr_raw; +u64 xtime_sec; +u64 ktime_sec; +u64 wall_to_monotonic_sec; +u64 wall_to_monotonic_nsec; +u64 offs_real; +u64 offs_boot; +u64 offs_tai; +u64 raw_sec; +u64 tsc_offset; +} __attribute__((__packed__ )); + +#define PVCLOCK_TIMEKEEPER_ENABLED (1 << 0) +#define PVCLOCK_TSC_STABLE_BIT (1 << 0)#define PVCLOCK_GUEST_STOPPED (1 << 1)/* PVCLOCK_COUNTS_FROM_ZERO broke ABI and can't be used anymore. */
arch/x86/include/uapi/asm/kvm_para.h +1 -0
@@ -50,6 +50,7 @@#define MSR_KVM_STEAL_TIME 0x4b564d03#define MSR_KVM_PV_EOI_EN 0x4b564d04#define MSR_KVM_POLL_CONTROL 0x4b564d05 +#define MSR_KVM_TIMEKEEPER_EN 0x4b564d06struct kvm_steal_time {__u64 steal;
arch/x86/kvm/x86.c +121 -0
157 lines not on this page · raw message
Signature · 1 line
2.23.0.237.gc6a4ce50a0-goog
RFC 2/2 x86/kvmclock: Use host timekeeping.
In reply to Suleiman SouhlalWhen CONFIG_KVMCLOCK_HOST_TIMEKEEPING is enabled, and the host supports it, update our timekeeping parameters to be the same as the host. This lets us have our time synchronized with the host's, even in the presence of host NTP or suspend.
When enabled, kvmclock uses raw tsc instead of pvclock.
When enabled, syscalls that can change time, such as settimeofday(2) or adj_timex(2) are disabled in the guest.
--- arch/x86/Kconfig | 9 +++ arch/x86/include/asm/kvmclock.h | 2 + arch/x86/kernel/kvmclock.c | 127 +++++++++++++++++++++++++++++++- kernel/time/timekeeping.c | 21 ++++++ 4 files changed, 155 insertions(+), 4 deletions(-)
arch/x86/Kconfig +9 -0
@@ -837,6 +837,15 @@ config PARAVIRT_TIME_ACCOUNTING config PARAVIRT_CLOCK bool +config KVMCLOCK_HOST_TIMEKEEPING + bool "kvmclock uses host timekeeping" + depends on KVM_GUEST + ---help--- + Select this option to make the guest use the same timekeeping + parameters as the host. This means that time will be almost + exactly the same between the two. Only works if the host uses "tsc" + clocksource. + config JAILHOUSE_GUEST bool "Jailhouse non-root cell support" depends on X86_64 && PCI
arch/x86/include/asm/kvmclock.h +2 -0
@@ -4,4 +4,6 @@extern struct clocksource kvm_clock; +bool kvm_clock_copy_into_tk (struct timekeeper *tk); +#endif /* _ASM_X86_KVM_CLOCK_H */
arch/x86/kernel/kvmclock.c +123 -4
188 lines not on this page · raw message
kernel/time/timekeeping.c +21 -0
@@ -23,6 +23,10 @@#include <linux/compiler.h> #include <linux/audit.h> +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING +#include <asm/kvmclock.h> +#endif +#include "tick-internal.h" #include "ntp_internal.h" #include "timekeeping_internal.h"
@@ -1223,6 +1227,7 @@ EXPORT_SYMBOL_GPL(get_device_system_crosststamp);*/ int do_settimeofday64 (const struct timespec64 *ts) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING struct timekeeper *tk = &tk_core.timekeeper;struct timespec64 ts_delta, xt;unsigned long flags;
@@ -1261,9 +1266,13 @@ int do_settimeofday64(const struct timespec64 *ts)audit_tk_injoffset (ts_delta);return ret; +#else /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ +return -EINVAL ; +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ }EXPORT_SYMBOL (do_settimeofday64); +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING /** * timekeeping_inject_offset - Adds or subtracts from the current time. * @tv: pointer to the timespec variable containing the offset
@@ -1307,6 +1316,7 @@ static int timekeeping_inject_offset(const struct timespec64 *ts) return ret; } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ /* * Indicates if there is an offset between the system clock and the hardware
@@ -1332,6 +1342,7 @@ int persistent_clock_is_local;*/ void timekeeping_warp_clock (void ) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING if (sys_tz.tz_minuteswest !=0 ) {struct timespec64 adjust;
@@ -1340,6 +1351,7 @@ void timekeeping_warp_clock(void) adjust.tv_nsec =0 ; timekeeping_inject_offset(&adjust); } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ }/**
@@ -2107,6 +2119,9 @@ static void timekeeping_advance(enum timekeeping_adv_mode mode) clock_set |=accumulate_nsecs_to_secs (tk);write_seqcount_begin (&tk_core.seq); +#ifdef CONFIG_KVMCLOCK_HOST_TIMEKEEPING +kvm_clock_copy_into_tk (tk); +#endif /* * Update the real timekeeper. *
@@ -2241,6 +2256,7 @@ ktime_t ktime_get_update_offsets_now(unsigned int *cwsseq, ktime_t *offs_real,return base; } +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING /** * timekeeping_validate_timex - Ensures the timex is ok for use in do_adjtimex */
@@ -2305,6 +2321,7 @@ static int timekeeping_validate_timex(const struct __kernel_timex *txc) return 0; } +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ /**
@@ -2312,6 +2329,7 @@ static int timekeeping_validate_timex(const struct __kernel_timex *txc)*/ int do_adjtimex (struct __kernel_timex *txc) { +#ifndef CONFIG_KVMCLOCK_HOST_TIMEKEEPING struct timekeeper *tk = &tk_core.timekeeper;struct audit_ntp_data ad;unsigned long flags;
@@ -2368,6 +2386,9 @@ int do_adjtimex(struct __kernel_timex *txc)ntp_notify_cmos_timer ();return ret; +#else /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ +return -EINVAL ; +#endif /* CONFIG_KVMCLOCK_HOST_TIMEKEEPING */ }#ifdef CONFIG_NTP_PPS
Signature · 1 line
2.23.0.237.gc6a4ce50a0-goog
Vitaly Kuznetsov
reviewerRe: [RFC 2/2] x86/kvmclock: Use host timekeeping.
Quoted 25 lines · Suleiman Souhlal
When
CONFIG_KVMCLOCK_HOST_TIMEKEEPINGis enabled, and the host supports it, update our timekeeping parameters to be the same as the host. This lets us have our time synchronized with the host's, even in the presence of host NTP or suspend.When enabled, kvmclock uses raw tsc instead of pvclock.
When enabled, syscalls that can change time, such as settimeofday(2) or
adj_timex(2)are disabled in the guest.Signed-off-by Suleiman Souhlal <[email protected]>--- arch/x86/Kconfig | 9 +++ arch/x86/include/asm/kvmclock.h | 2 + arch/x86/kernel/kvmclock.c | 127 +++++++++++++++++++++++++++++++- kernel/time/timekeeping.c | 21 ++++++ 4 files changed, 155 insertions(+), 4 deletions(-)arch/x86/Kconfig +0
-0@@ -837,6 +837,15 @@ config PARAVIRT_TIME_ACCOUNTING config PARAVIRT_CLOCK bool
arch/x86/Kconfig config PARAVIRT_TIME_ACCOUNTING · line 848
Quoted 10 lines
arch/x86/Kconfig +9
-0+config KVMCLOCK_HOST_TIMEKEEPING + bool "kvmclock uses host timekeeping" + depends on KVM_GUEST + ---help--- + Select this option to make the guest use the same timekeeping + parameters as the host. This means that time will be almost + exactly the same between the two. Only works if the host uses "tsc" + clocksource. +
I'd also like to speak up against this config, it is confusing. In case the goal is to come up with a TSC-based clock for guests which will return the same as clock_gettime() on the host (or, is the goal to just have the same reading for all guests on the host?) I'd suggest we create a separate (from KVMCLOCK) clocksource (mirroring host timekeeper) and guests will be free to pick the one they like.
Signature · 1 line
Vitaly
Re: [RFC 2/2] x86/kvmclock: Use host timekeeping.
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Suleiman Souhlal
When
CONFIG_KVMCLOCK_HOST_TIMEKEEPINGis enabled, and the host supports it, update our timekeeping parameters to be the same as the host. This lets us have our time synchronized with the host's, even in the presence of host NTP or suspend.When enabled, kvmclock uses raw tsc instead of pvclock.
When enabled, syscalls that can change time, such as settimeofday(2) or
adj_timex(2)are disabled in the guest.Signed-off-by Suleiman Souhlal <[email protected]>--- arch/x86/Kconfig | 9 +++ arch/x86/include/asm/kvmclock.h | 2 + arch/x86/kernel/kvmclock.c | 127 +++++++++++++++++++++++++++++++- kernel/time/timekeeping.c | 21 ++++++ 4 files changed, 155 insertions(+), 4 deletions(-)arch/x86/Kconfig +0
-0@@ -837,6 +837,15 @@ config PARAVIRT_TIME_ACCOUNTING config PARAVIRT_CLOCK bool
Quoted 16 lines
arch/x86/Kconfig config PARAVIRT_TIME_ACCOUNTING · line 848
Quoted 9 lines
arch/x86/Kconfig +9
-0+config KVMCLOCK_HOST_TIMEKEEPING + bool "kvmclock uses host timekeeping" + depends on KVM_GUEST + ---help--- + Select this option to make the guest use the same timekeeping + parameters as the host. This means that time will be almost + exactly the same between the two. Only works if the host uses "tsc" + clocksource. +I'd also like to speak up against this config, it is confusing. In case the goal is to come up with a TSC-based clock for guests which will return the same as
clock_gettime()on the host (or, is the goal to just have the same reading for all guests on the host?) I'd suggest we create a separate (from KVMCLOCK) clocksource (mirroring host timekeeper) and guests will be free to pick the one they like.
Fair enough. I'll do that in the next version of the patch.
The goal is to have a guest clock that gives the same clock_gettime(CLOCK_MONOTONIC) as the host.
Thanks,
-- Suleiman
Vitaly Kuznetsov
reviewerRe: [RFC 2/2] x86/kvmclock: Use host timekeeping.
Quoted 47 lines · Suleiman Souhlal
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Suleiman Souhlal
When
CONFIG_KVMCLOCK_HOST_TIMEKEEPINGis enabled, and the host supports it, update our timekeeping parameters to be the same as the host. This lets us have our time synchronized with the host's, even in the presence of host NTP or suspend.When enabled, kvmclock uses raw tsc instead of pvclock.
When enabled, syscalls that can change time, such as settimeofday(2) or
adj_timex(2)are disabled in the guest.Signed-off-by Suleiman Souhlal <[email protected]>--- arch/x86/Kconfig | 9 +++ arch/x86/include/asm/kvmclock.h | 2 + arch/x86/kernel/kvmclock.c | 127 +++++++++++++++++++++++++++++++- kernel/time/timekeeping.c | 21 ++++++ 4 files changed, 155 insertions(+), 4 deletions(-)arch/x86/Kconfig +0
-0@@ -837,6 +837,15 @@ config PARAVIRT_TIME_ACCOUNTING config PARAVIRT_CLOCK boolQuoted 16 lines
arch/x86/Kconfig config PARAVIRT_TIME_ACCOUNTING · line 848
Quoted 9 lines
arch/x86/Kconfig +9
-0+config KVMCLOCK_HOST_TIMEKEEPING + bool "kvmclock uses host timekeeping" + depends on KVM_GUEST + ---help--- + Select this option to make the guest use the same timekeeping + parameters as the host. This means that time will be almost + exactly the same between the two. Only works if the host uses "tsc" + clocksource. +I'd also like to speak up against this config, it is confusing. In case the goal is to come up with a TSC-based clock for guests which will return the same as
clock_gettime()on the host (or, is the goal to just have the same reading for all guests on the host?) I'd suggest we create a separate (from KVMCLOCK) clocksource (mirroring host timekeeper) and guests will be free to pick the one they like.
Quoted 5 lines
Fair enough. I'll do that in the next version of the patch.
The goal is to have a guest clock that gives the same
clock_gettime(CLOCK_MONOTONIC)as the host.
KVMCLOCK has lots of legacy derived from times when TSC synchronization was not a given (I heard that this is still sometimes problematic with multi-socket systems but oh well). If I was to design a new clock I'd probably mirror Hyper-V's TSC page clocksource invalidating the page when host timekeeper values are updated (and making guest spin).
The tricky part with this approach is probably tsc scaling/tsc offsetting which is still going to be controlled by kvmclock (so the guest has no option to read 'pure TSC'). As an alternative, you can make kvmclock un-pluggable so when it's not enabled TSC frequency/offset can remain intact. You can, of course, try to update timekeeper values to match the new frequency/offset every time they change but rounding errors may bite.
Signature · 1 line
Vitaly
Paolo Bonzini
maintainerQuoted 6 lines · Suleiman Souhlal
To do that, I am changing kvmclock to request to the host to copy its timekeeping parameters (mult, base,
cycle_last, etc), so that the guest timekeeper can use the same values, so that time can be synchronized between the guest and the host.Any suggestions or feedback would be highly appreciated.
I'm not a timekeeping maintainer, but I don't think the kernel/time/timekeeping.c changes are acceptable.
Is the PTP driver not enough?
Paolo
Quoted 10 lines · Paolo Bonzini
Quoted 6 lines · Suleiman Souhlal
To do that, I am changing kvmclock to request to the host to copy its timekeeping parameters (mult, base,
cycle_last, etc), so that the guest timekeeper can use the same values, so that time can be synchronized between the guest and the host.Any suggestions or feedback would be highly appreciated.
I'm not a timekeeping maintainer, but I don't think the
kernel/time/timekeeping.cchanges are acceptable.
Indeed. #ifdef WHATEVERTHEHECK does not go anywhere. If at all this needs to be a runtime switch, but I have yet to understand the whole picture of this.
Thanks,
tglx
Quoted 9 lines · Thomas Gleixner
Quoted 6 lines · Paolo Bonzini
Quoted 5 lines · Suleiman Souhlal
To do that, I am changing kvmclock to request to the host to copy its timekeeping parameters (mult, base,
cycle_last, etc), so that the guest timekeeper can use the same values, so that time can be synchronized between the guest and the host.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Any suggestions or feedback would be highly appreciated.
I'm not a timekeeping maintainer, but I don't think the
kernel/time/timekeeping.cchanges are acceptable.Indeed. #ifdef WHATEVERTHEHECK does not go anywhere. If at all this needs to be a runtime switch, but I have yet to understand the whole picture of this.
Yeah, I will try to make this a runtime switch.
As for the PTP driver, I don't think it will work for us because we need CLOCK_MONOTONIC and CLOCK_BOOTTIME to match the host, and from my understanding, PTP doesn't solve that.
Thanks,
-- Suleiman