KVM_{GET,SET}_TSC_OFFSET ioctls

To date, VMMs have typically restored the guest's TSCs by value using the KVM_SET_MSRS ioctl for each vCPU. However, restoring the TSCs by value introduces some challenges with synchronization as the TSCs continue to tick throughout the restoration process. As such, KVM has some heuristics around TSC writes to infer whether or not the guest or host is attempting to synchronize the TSCs.

Instead of guessing at the intentions of a VMM, it'd be better to provide an interface that allows for explicit synchronization of the guest's TSCs. To that end, this series introduces the KVM_{GET,SET}_TSC_OFFSET ioctls, yielding control of the TSC offset to userspace.

v2 => v3:
- Mark kvm_write_tsc_offset() as static (whoops)

v1 => v2:
- Added clarification to the documentation of KVM_SET_TSC_OFFSET to indicate that it can be used instead of an IA32_TSC MSR restore through KVM_SET_MSRS
- Fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics, thereby enabling the KVM masterclock when all vCPUs are in phase.

Oliver Upton (4):
kvm: x86: refactor masterclock sync heuristics out of kvm_write_tsc
kvm: vmx: check tsc offsetting with nested_cpu_has()
selftests: kvm: use a helper function for reading cpuid
selftests: kvm: introduce tsc_offset_test

Peter Hornyack (1):
kvm: x86: add KVM_{GET,SET}_TSC_OFFSET ioctls

 Documentation/virt/kvm/api.rst                |  31 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/vmx/vmx.c                        |   2 +-
 arch/x86/kvm/x86.c                            | 147 ++++---
 include/uapi/linux/kvm.h                      |   5 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/include/test_util.h |   3 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 +
 .../selftests/kvm/include/x86_64/svm_util.h   |  10 +-
 .../selftests/kvm/include/x86_64/vmx.h        |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   1 +
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |  11 +
 .../selftests/kvm/x86_64/tsc_offset_test.c    | 362 ++++++++++++++++++
 14 files changed, 550 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/tsc_offset_test.c

--
2.28.0.rc0.142.g3c755180ce-goog

Quoted 51 lines · Oliver Upton

To date, VMMs have typically restored the guest's TSCs by value using the KVM_SET_MSRS ioctl for each vCPU. However, restoring the TSCs by value introduces some challenges with synchronization as the TSCs continue to tick throughout the restoration process. As such, KVM has some heuristics around TSC writes to infer whether or not the guest or host is attempting to synchronize the TSCs.

Instead of guessing at the intentions of a VMM, it'd be better to provide an interface that allows for explicit synchronization of the guest's TSCs. To that end, this series introduces the KVM_{GET,SET}_TSC_OFFSET ioctls, yielding control of the TSC offset to userspace.

v2 => v3:
- Mark kvm_write_tsc_offset() as static (whoops)

v1 => v2:
- Added clarification to the documentation of KVM_SET_TSC_OFFSET to indicate that it can be used instead of an IA32_TSC MSR restore through KVM_SET_MSRS
- Fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics, thereby enabling the KVM masterclock when all vCPUs are in phase.

Oliver Upton (4):
kvm: x86: refactor masterclock sync heuristics out of kvm_write_tsc
kvm: vmx: check tsc offsetting with nested_cpu_has()
selftests: kvm: use a helper function for reading cpuid
selftests: kvm: introduce tsc_offset_test

Peter Hornyack (1):
kvm: x86: add KVM_{GET,SET}_TSC_OFFSET ioctls

 Documentation/virt/kvm/api.rst                |  31 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/vmx/vmx.c                        |   2 +-
 arch/x86/kvm/x86.c                            | 147 ++++---
 include/uapi/linux/kvm.h                      |   5 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/include/test_util.h |   3 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 +
 .../selftests/kvm/include/x86_64/svm_util.h   |  10 +-
 .../selftests/kvm/include/x86_64/vmx.h        |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   1 +
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |  11 +
 .../selftests/kvm/x86_64/tsc_offset_test.c    | 362 ++++++++++++++++++
 14 files changed, 550 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/tsc_offset_test.c

Quoted 3 lines

--
2.28.0.rc0.142.g3c755180ce-goog

Ping :)

Quoted 53 lines · Oliver Upton

Quoted 51 lines · Oliver Upton

To date, VMMs have typically restored the guest's TSCs by value using the KVM_SET_MSRS ioctl for each vCPU. However, restoring the TSCs by value introduces some challenges with synchronization as the TSCs continue to tick throughout the restoration process. As such, KVM has some heuristics around TSC writes to infer whether or not the guest or host is attempting to synchronize the TSCs.

Instead of guessing at the intentions of a VMM, it'd be better to provide an interface that allows for explicit synchronization of the guest's TSCs. To that end, this series introduces the KVM_{GET,SET}_TSC_OFFSET ioctls, yielding control of the TSC offset to userspace.

v2 => v3:
- Mark kvm_write_tsc_offset() as static (whoops)

v1 => v2:
- Added clarification to the documentation of KVM_SET_TSC_OFFSET to indicate that it can be used instead of an IA32_TSC MSR restore through KVM_SET_MSRS
- Fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics, thereby enabling the KVM masterclock when all vCPUs are in phase.

Oliver Upton (4):
kvm: x86: refactor masterclock sync heuristics out of kvm_write_tsc
kvm: vmx: check tsc offsetting with nested_cpu_has()
selftests: kvm: use a helper function for reading cpuid
selftests: kvm: introduce tsc_offset_test

Peter Hornyack (1):
kvm: x86: add KVM_{GET,SET}_TSC_OFFSET ioctls

 Documentation/virt/kvm/api.rst                |  31 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/vmx/vmx.c                        |   2 +-
 arch/x86/kvm/x86.c                            | 147 ++++---
 include/uapi/linux/kvm.h                      |   5 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/include/test_util.h |   3 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 +
 .../selftests/kvm/include/x86_64/svm_util.h   |  10 +-
 .../selftests/kvm/include/x86_64/vmx.h        |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   1 +
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |  11 +
 .../selftests/kvm/x86_64/tsc_offset_test.c    | 362 ++++++++++++++++++
 14 files changed, 550 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/tsc_offset_test.c

Quoted 5 lines

Quoted 3 lines

--
2.28.0.rc0.142.g3c755180ce-goog

Ping :)

Ping

Quoted 58 lines · Oliver Upton

Quoted 57 lines · Oliver Upton

Quoted 54 lines · Oliver Upton

To date, VMMs have typically restored the guest's TSCs by value using the KVM_SET_MSRS ioctl for each vCPU. However, restoring the TSCs by value introduces some challenges with synchronization as the TSCs continue to tick throughout the restoration process. As such, KVM has some heuristics around TSC writes to infer whether or not the guest or host is attempting to synchronize the TSCs.

Instead of guessing at the intentions of a VMM, it'd be better to provide an interface that allows for explicit synchronization of the guest's TSCs. To that end, this series introduces the KVM_{GET,SET}_TSC_OFFSET ioctls, yielding control of the TSC offset to userspace.

v2 => v3:
- Mark kvm_write_tsc_offset() as static (whoops)

v1 => v2:
- Added clarification to the documentation of KVM_SET_TSC_OFFSET to indicate that it can be used instead of an IA32_TSC MSR restore through KVM_SET_MSRS
- Fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics, thereby enabling the KVM masterclock when all vCPUs are in phase.

Oliver Upton (4):
kvm: x86: refactor masterclock sync heuristics out of kvm_write_tsc
kvm: vmx: check tsc offsetting with nested_cpu_has()
selftests: kvm: use a helper function for reading cpuid
selftests: kvm: introduce tsc_offset_test

Peter Hornyack (1):
kvm: x86: add KVM_{GET,SET}_TSC_OFFSET ioctls

 Documentation/virt/kvm/api.rst                |  31 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/vmx/vmx.c                        |   2 +-
 arch/x86/kvm/x86.c                            | 147 ++++---
 include/uapi/linux/kvm.h                      |   5 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/include/test_util.h |   3 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 +
 .../selftests/kvm/include/x86_64/svm_util.h   |  10 +-
 .../selftests/kvm/include/x86_64/vmx.h        |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   1 +
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |  11 +
 .../selftests/kvm/x86_64/tsc_offset_test.c    | 362 ++++++++++++++++++
 14 files changed, 550 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/tsc_offset_test.c

--
2.28.0.rc0.142.g3c755180ce-goog

Quoted 3 lines

Quoted 1 line

Ping :)

Ping

Hi Oliver,

I saw these on vacation and decided I would delay them to 5.10. However they are definitely on my list.

I have one possibly very stupid question just by looking at the cover
letter: now that you've "fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics" what makes it still not "guessing the intentions of a VMM"? (No snark intended, just quoting the parts that puzzled me a bit).

My immediate reaction was that we should just migrate the heuristics state somehow

but perhaps I'm missing something obvious.

Paolo

Quoted 60 lines · Paolo Bonzini

Quoted 58 lines · Oliver Upton

Quoted 57 lines · Oliver Upton

Quoted 54 lines · Oliver Upton

To date, VMMs have typically restored the guest's TSCs by value using the KVM_SET_MSRS ioctl for each vCPU. However, restoring the TSCs by value introduces some challenges with synchronization as the TSCs continue to tick throughout the restoration process. As such, KVM has some heuristics around TSC writes to infer whether or not the guest or host is attempting to synchronize the TSCs.

Instead of guessing at the intentions of a VMM, it'd be better to provide an interface that allows for explicit synchronization of the guest's TSCs. To that end, this series introduces the KVM_{GET,SET}_TSC_OFFSET ioctls, yielding control of the TSC offset to userspace.

v2 => v3:
- Mark kvm_write_tsc_offset() as static (whoops)

v1 => v2:
- Added clarification to the documentation of KVM_SET_TSC_OFFSET to indicate that it can be used instead of an IA32_TSC MSR restore through KVM_SET_MSRS
- Fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics, thereby enabling the KVM masterclock when all vCPUs are in phase.

Oliver Upton (4):
kvm: x86: refactor masterclock sync heuristics out of kvm_write_tsc
kvm: vmx: check tsc offsetting with nested_cpu_has()
selftests: kvm: use a helper function for reading cpuid
selftests: kvm: introduce tsc_offset_test

Peter Hornyack (1):
kvm: x86: add KVM_{GET,SET}_TSC_OFFSET ioctls

 Documentation/virt/kvm/api.rst                |  31 ++
 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/vmx/vmx.c                        |   2 +-
 arch/x86/kvm/x86.c                            | 147 ++++---
 include/uapi/linux/kvm.h                      |   5 +
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../testing/selftests/kvm/include/test_util.h |   3 +
 .../selftests/kvm/include/x86_64/processor.h  |  15 +
 .../selftests/kvm/include/x86_64/svm_util.h   |  10 +-
 .../selftests/kvm/include/x86_64/vmx.h        |   9 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |   1 +
 tools/testing/selftests/kvm/lib/x86_64/vmx.c  |  11 +
 .../selftests/kvm/x86_64/tsc_offset_test.c    | 362 ++++++++++++++++++
 14 files changed, 550 insertions(+), 49 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86_64/tsc_offset_test.c

--
2.28.0.rc0.142.g3c755180ce-goog

Quoted 9 lines

Quoted 3 lines

Quoted 1 line

Ping :)

Ping

Hi Oliver,

I saw these on vacation and decided I would delay them to 5.10. However they are definitely on my list.

Hope you enjoyed vacation!

Quoted 5 lines

I have one possibly very stupid question just by looking at the cover
letter: now that you've "fixed KVM_SET_TSC_OFFSET to participate in the existing TSC synchronization heuristics" what makes it still not "guessing the intentions of a VMM"? (No snark intended, just quoting the parts that puzzled me a bit).

Great point.

I'd still posit that this series disambiguates userspace control/synchronization of the TSCs. If a VMM wants the TSCs to be in sync, it can write identical offsets to all vCPUs

That said, participation in TSC synchronization is presently necessary due to issues migrating a guest that was in the middle of a TSC sync. In doing so, we still accomplish synchronization on the other end of migration with a well-timed mix of host and guest writes.

Quoted 3 lines

My immediate reaction was that we should just migrate the heuristics state somehow

Yeah, I completely agree. I believe this series fixes the userspace-facing issues and your suggestion would address the guest-facing issues.

Quoted 1 line

but perhaps I'm missing something obvious.

Not necessarily obvious, but I can think of a rather contrived example where the sync heuristics break down. If we're running nested and get migrated in the middle of a VMM setting up TSCs, it's possible that enough time will pass that we believe subsequent writes to not be of the same TSC generation.

Quoted 6 lines

Quoted 1 line

but perhaps I'm missing something obvious.

Not necessarily obvious, but I can think of a rather contrived example where the sync heuristics break down. If we're running nested and get migrated in the middle of a VMM setting up TSCs, it's possible that enough time will pass that we believe subsequent writes to not be of the same TSC generation.

An example that has been biting us frequently in self-tests: migrate a VM with less than a second accumulated in its TSC. At the destination, the TSCs are zeroed.

Quoted 2 lines

My immediate reaction was that we should just migrate the heuristics state somehow

Yeah, I completely agree. I believe this series fixes the userspace-facing issues and your suggestion would address the guest-facing issues.

Quoted 7 lines · Jim Mattson

Quoted 6 lines

Quoted 1 line

but perhaps I'm missing something obvious.

Not necessarily obvious, but I can think of a rather contrived example where the sync heuristics break down. If we're running nested and get migrated in the middle of a VMM setting up TSCs, it's possible that enough time will pass that we believe subsequent writes to not be of the same TSC generation.

Quoted 3 lines

An example that has been biting us frequently in self-tests: migrate a VM with less than a second accumulated in its TSC. At the destination, the TSCs are zeroed.

Yeah, good point about selftests. But this would be about the sync heuristics messing up, and I don't understand how these ioctls improve things.

Quoted 7 lines

Quoted 3 lines

My immediate reaction was that we should just migrate the heuristics state somehow

Yeah, I completely agree. I believe this series fixes the userspace-facing issues and your suggestion would address the guest-facing issues.

I still don't understand how these ioctls are any better for userspace than migrating MSR_IA32_TSC. The host TSC is different between source and destination, so the TSC offset will be different.

I am all for improving migration of TSC state, but I think we should do it right, so we should migrate a host clock / TSC pair: then the destination host can use TSC frequency and host clock to compute the new TSC value. In fact, such a pair is exactly the data that the syncing heuristics track for each "generation" of syncing.

To migrate the synchronization state, instead, we only need to migrate the "already_matched" (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation) state.

Putting all of this together, it would be something like this:

- a VM-wide KVM_CLOCK/KVM_SET_CLOCK needs to migrate vcpu->arch.cur_tsc_{nsec,write} in addition to the current kvmclock value (it's unrelated, but I don't think it's worth creating a new ioctl). A new flag is set if these fields are set in the struct. If the flag is set, KVM_SET_CLOCK copies the fields back, bumps the generation and clears kvm->arch.nr_vcpus_matched_tsc.

- a VCPU-wide KVM_GET_TSC_INFO returns a host clock / guest TSC pair plus the "already matched" state. KVM_SET_TSC_INFO will only use the host clock / TSC pair if "already matched" is false, to compute the destination-side TSC offset but not otherwise doing anything with it; or if "already matched" is true, it will ignore the pair, compute the TSC offset from the data in kvm->arch, and update kvm->arch.nr_vcpus_matched_tsc.

Paolo

We allow our userspace to decide the host TSC / wall clock pair at which the vCPUs were paused. From that host TSC value we reconstruct the guest TSCs using the offsets and migrate that info. On the destination we grab another host TSC / clock pair and recalculate guest TSC offsets, which we then pass to KVM_SET_TSC_OFFSET. This is desirable over a per-vCPU read of MSR_IA32_TSC because we've effectively recorded all TSCs at the exact same moment in time.

Otherwise, we inadvertently add skew between guest TSCs by reading each vCPU at different times. It seems that the sync heuristics address this issue along with any guest TSC coordination.

Not only that, depending on the heuristics to detect a sync from userspace gets a bit tricky if we (KVM) are running nested. Couldn't more than a second of time elapse between successive KVM_SET_MSRS when running in L1 if L0 decides to pause our vCPUs (suspend/resume, migration)? It seems to me that in this case we will fail to detect a sync condition and configure the L2 TSCs to be out-of-phase.

Setting the guest TSCs by offset doesn't have these complications. Even if L0 were to pause L1 for some inordinate amount of time, the relation of L1 -> L2 TSC is never disturbed.

Quoted 20 lines

I am all for improving migration of TSC state, but I think we should do it right, so we should migrate a host clock / TSC pair: then the destination host can use TSC frequency and host clock to compute the new TSC value. In fact, such a pair is exactly the data that the syncing heuristics track for each "generation" of syncing.

To migrate the synchronization state, instead, we only need to migrate the "already_matched" (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation) state.

Putting all of this together, it would be something like this:

- a VM-wide KVM_CLOCK/KVM_SET_CLOCK needs to migrate vcpu->arch.cur_tsc_{nsec,write} in addition to the current kvmclock value (it's unrelated, but I don't think it's worth creating a new ioctl). A new flag is set if these fields are set in the struct. If the flag is set, KVM_SET_CLOCK copies the fields back, bumps the generation and clears kvm->arch.nr_vcpus_matched_tsc.

Quoted 8 lines

- a VCPU-wide KVM_GET_TSC_INFO returns a host clock / guest TSC pair plus the "already matched" state. KVM_SET_TSC_INFO will only use the host clock / TSC pair if "already matched" is false, to compute the destination-side TSC offset but not otherwise doing anything with it; or if "already matched" is true, it will ignore the pair, compute the TSC offset from the data in kvm->arch, and update kvm->arch.nr_vcpus_matched_tsc.

It seems to me that a per-vCPU ioctl (like you've suggested above) is necessary to uphold the guest-facing side of our sync scheme, regardless of what we do on the userspace-facing side.

Quoted 2 lines

Paolo

Quoted 7 lines · Oliver Upton

We allow our userspace to decide the host TSC / wall clock pair at which the vCPUs were paused. From that host TSC value we reconstruct the guest TSCs using the offsets and migrate that info. On the destination we grab another host TSC / clock pair and recalculate guest TSC offsets, which we then pass to KVM_SET_TSC_OFFSET. This is desirable over a per-vCPU read of MSR_IA32_TSC because we've effectively recorded all TSCs at the exact same moment in time.

Yes, this should work very well. But in the end KVM ends up with the vcpu->arch.cur_tsc_{nsec,write} of the source (only shifted in time), while losing the notion that the pair is per-VM rather than per-VCPU for the "already matched" vCPUs. So that is why I'm thinking of retrieving that pair from the kernel directly.

If you don't have time to work on it I can try to find some for 5.10, but I'm not sure exactly when.

Paolo

Quoted 51 lines

Otherwise, we inadvertently add skew between guest TSCs by reading each vCPU at different times. It seems that the sync heuristics address this issue along with any guest TSC coordination.

Not only that, depending on the heuristics to detect a sync from userspace gets a bit tricky if we (KVM) are running nested. Couldn't more than a second of time elapse between successive KVM_SET_MSRS when running in L1 if L0 decides to pause our vCPUs (suspend/resume, migration)? It seems to me that in this case we will fail to detect a sync condition and configure the L2 TSCs to be out-of-phase.

Setting the guest TSCs by offset doesn't have these complications. Even if L0 were to pause L1 for some inordinate amount of time, the relation of L1 -> L2 TSC is never disturbed.

Quoted 20 lines

I am all for improving migration of TSC state, but I think we should do it right, so we should migrate a host clock / TSC pair: then the destination host can use TSC frequency and host clock to compute the new TSC value. In fact, such a pair is exactly the data that the syncing heuristics track for each "generation" of syncing.

To migrate the synchronization state, instead, we only need to migrate the "already_matched" (vcpu->arch.this_tsc_generation == kvm->arch.cur_tsc_generation) state.

Putting all of this together, it would be something like this:

- a VM-wide KVM_CLOCK/KVM_SET_CLOCK needs to migrate vcpu->arch.cur_tsc_{nsec,write} in addition to the current kvmclock value (it's unrelated, but I don't think it's worth creating a new ioctl). A new flag is set if these fields are set in the struct. If the flag is set, KVM_SET_CLOCK copies the fields back, bumps the generation and clears kvm->arch.nr_vcpus_matched_tsc.

Quoted 8 lines

- a VCPU-wide KVM_GET_TSC_INFO returns a host clock / guest TSC pair plus the "already matched" state. KVM_SET_TSC_INFO will only use the host clock / TSC pair if "already matched" is false, to compute the destination-side TSC offset but not otherwise doing anything with it; or if "already matched" is true, it will ignore the pair, compute the TSC offset from the data in kvm->arch, and update kvm->arch.nr_vcpus_matched_tsc.

It seems to me that a per-vCPU ioctl (like you've suggested above) is necessary to uphold the guest-facing side of our sync scheme, regardless of what we do on the userspace-facing side.

Quoted 2 lines

Paolo

Quoted 2 lines

If you don't have time to work on it I can try to find some for 5.10, but I'm not sure exactly when.

Shouldn't be an issue, I'll futz around with some changes to the series and send them out in the coming weeks.

Quoted 5 lines · Oliver Upton

Quoted 2 lines

If you don't have time to work on it I can try to find some for 5.10, but I'm not sure exactly when.

Shouldn't be an issue, I'll futz around with some changes to the series and send them out in the coming weeks.

Ok, after looking more at the code with Maxim I can confidently say that it's a total mess. And a lot of the synchronization code is dead because 1) as far as we could see no guest synchronizes the TSC using MSR_IA32_TSC; and 2) writing to MSR_IA32_TSC_ADJUST does not trigger the synchronization code in kvm_write_tsc.

Your patch works not by some sort of miracle, but rather because it bypasses the mess and that's the smart thing to do.

The plan is now as follows:

1) guest-initiated MSR_IA32_TSC write never goes through the sync heuristics. I'll shortly send a patch for this, and it will fix the testcase issue

2) to have a new KVM_X86_DISABLE_QUIRKS value, that will toggle between "magic" and "vanilla" semantics for host-initiated TSC and TSC_ADJUST writes

3) if the quirk is present we still want existing userspace to work so:

- host-initiated MSR_IA32_TSC write always returns the L1 TSC as in Maxim's recent patch. They will also always go through the sync heuristics.

- host-initiated MSR_IA32_TSC_ADJUST write don't make the TSC jump, they only write to vcpu->arch.ia32_tsc_adjust_msr (as in the current kernel)

4) if the quirk is disabled however:

- the sync heuristics are never used except in kvm_arch_vcpu_postcreate

- host-initiated MSR_IA32_TSC and MSR_IA32_TSC_ADJUST accesses work like in the guest: reads of MSR_IA32_TSC return the "right" TSC, writes of MSR_IA32_TSC_ADJUST writes make the TSC jump.

- for live migration, userspace is expected to use the new KVM_GET/SET_TSC_PRECISE (or whatever the name will be) to get/set a (nanosecond, TSC, TSC_ADJUST) tuple. The sync heuristics will be bypassed and it will just set the right value for the MSRs. Setting MSR_IA32_TSC_ADJUST is optional and controlled by a flag in the struct, and the flag will be set by KVM_GET_TSC_PRECISE based on the guest CPUID.

Paolo