Patch "x86, vdso, pvclock: Simplify and speed up the vdso pvclock reader" has been added to the 4.4-stable tree
Subsystems (4)
This is a note to let you know that I've just added the patch titled
x86, vdso, pvclock: Simplify and speed up the vdso pvclock reader
to the 4.4-stable tree which can be found at:
http://www.kernel.org/git/?p=linux/kernel/git/stable/stable-queue.git;a=summary
The filename of the patch is:
x86-vdso-pvclock-simplify-and-speed-up-the-vdso-pvclock-reader.patch
and it can be found in the queue-4.4 subdirectory.
If you, or anyone else, feels it should not be added to the stable tree, please let <[email protected]> know about it.
Quoted 1 line
From
6b078f5de7fc0851af4102493c7b5bb07e49c4cbMon Sep 17 00:00:00 2001
From: Andy Lutomirski <[email protected]>
Date: Thu, 10 Dec 2015 19:20:19 -0800
Subject: x86, vdso, pvclock: Simplify and speed up the vdso pvclock reader
From: Andy Lutomirski <[email protected]>
commit 6b078f5de7fc0851af4102493c7b5bb07e49c4cb upstream.
The pvclock vdso code was too abstracted to understand easily and excessively paranoid. Simplify it for a huge speedup.
This opens the door for additional simplifications, as the vdso no longer accesses the pvti for any vcpu other than vcpu 0.
Before, vclock_gettime using kvm-clock took about 45ns on my machine. With this change, it takes 29ns, which is almost as fast as the pure TSC implementation.
--- arch/x86/entry/vdso/vclock_gettime.c | 79 +++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 34 deletions(-)
arch/x86/entry/vdso/vclock_gettime.c +45 -34
@@ -78,47 +78,58 @@ static notrace const struct pvclock_vsys static notrace cycle_t vread_pvclock(int *mode) { -const struct pvclock_vsyscall_time_info *pvti; +const struct pvclock_vcpu_time_info *pvti = &get_pvti (0 )->pvti;cycle_t ret; -u64 last; -u32 version; -u8 flags; -unsigned cpu, cpu1; - +u64 tsc, pvti_tsc; +u64 last, delta, pvti_system_time; +u32 version, pvti_tsc_to_system_mul, pvti_tsc_shift;/* -* Note: hypervisor must guarantee that: -* 1. cpu ID number maps 1:1 to per-CPU pvclock time info. -* 2. that per-CPU pvclock time info is updated if the -* underlying CPU changes. -* 3. that version is increased whenever underlying CPU -* changes. +* Note: The kernel and hypervisor must guarantee that cpu ID +* number maps 1:1 to per-CPU pvclock time info. +* +* Because the hypervisor is entirely unaware of guest userspace +* preemption, it cannot guarantee that per-CPU pvclock time +* info is updated if the underlying CPU changes or that that +* version is increased whenever underlying CPU changes. +* +* On KVM, we are guaranteed that pvti updates for any vCPU are +* atomic as seen by *all* vCPUs. This is an even stronger +* guarantee than we get with a normal seqlock. * +* On Xen, we don't appear to have that guarantee, but Xen still +* supplies a valid seqlock using the version field. + +* We only do pvclock vdso timing at all if +* PVCLOCK_TSC_STABLE_BIT is set, and we interpret that bit to +* mean that all vCPUs have matching pvti and that the TSC is +* synced, so we can just look at vCPU 0's pvti. */ -do { - cpu =__getcpu () &VGETCPU_CPU_MASK ; -/* TODO: We can put vcpu id into higher bits of pvti.version. -* This will save a couple of cycles by getting rid of -* __getcpu() calls (Gleb). -*/ - - pvti =get_pvti (cpu); - - version =__pvclock_read_cycles (&pvti->pvti, &ret, &flags); - -/* -* Test we're still on the cpu as well as the version. -* We could have been migrated just after the first -* vgetcpu but before fetching the version, so we -* wouldn't notice a version change. -*/ - cpu1 =__getcpu () &VGETCPU_CPU_MASK ; - }while (unlikely (cpu != cpu1 || - (pvti->pvti.version &1 ) || - pvti->pvti.version != version)); -if (unlikely (!(flags &PVCLOCK_TSC_STABLE_BIT ))) +if (unlikely (!(pvti->flags &PVCLOCK_TSC_STABLE_BIT ))) { *mode =VCLOCK_NONE ; +return 0 ; + } + +do { + version = pvti->version; + +/* This is also a read barrier, so we'll read version first. */ + tsc =rdtsc_ordered (); + + pvti_tsc_to_system_mul = pvti->tsc_to_system_mul; + pvti_tsc_shift = pvti->tsc_shift; + pvti_system_time = pvti->system_time; + pvti_tsc = pvti->tsc_timestamp; + +/* Make sure that the version double-check is last. */ +smp_rmb (); + }while (unlikely ((version &1 ) || version != pvti->version)); + + delta = tsc - pvti_tsc; + ret = pvti_system_time + +pvclock_scale_delta (delta, pvti_tsc_to_system_mul, + pvti_tsc_shift);/* refer to tsc.c read_tsc() comment for rationale */ last = gtod->cycle_last;
Patches currently in stable-queue which might be from [email protected] are
queue-4.4/x86-vdso-pvclock-simplify-and-speed-up-the-vdso-pvclock-reader.patch queue-4.4/x86-vdso-get-pvclock-data-from-the-vvar-vma-instead-of-the-fixmap.patch