KVM: Dirty ring interface
Subsystems (4)
cover KVM: Dirty ring interface
Branch is here: https://github.com/xzpeter/linux/tree/kvm-dirty-ring
Overview
============
This is a continued work from Lei Cao <[email protected]> and Paolo on the KVM dirty ring interface. To make it simple, I'll still start with version 1 as RFC.
The new dirty ring interface is another way to collect dirty pages for the virtual machine, but it is different from the existing dirty logging interface in a few ways, majorly:
- Data format: The dirty data was in a ring format rather than a bitmap format, so the size of data to sync for dirty logging does not depend on the size of guest memory any more, but speed of dirtying. Also, the dirty ring is per-vcpu (currently plus another per-vm ring, so total ring number is N+1), while the dirty bitmap is per-vm.
- Data copy: The sync of dirty pages does not need data copy any more, but instead the ring is shared between the userspace and kernel by page sharings (mmap() on either the vm fd or vcpu fd)
- Interface: Instead of using the old KVM_GET_DIRTY_LOG, KVM_CLEAR_DIRTY_LOG interfaces, the new ring uses a new interface called KVM_RESET_DIRTY_RINGS when we want to reset the collected dirty pages to protected mode again (works like KVM_CLEAR_DIRTY_LOG, but ring based)
And more.
I would appreciate if the reviewers can start with patch "KVM:
Implement ring-based dirty memory tracking", especially the document update part for the big picture. Then I'll avoid copying into most of them into cover letter again.
I marked this series as RFC because I'm at least uncertain on this change of vcpu_enter_guest():
if (kvm_check_request(KVM_REQ_DIRTY_RING_FULL, vcpu)) {
vcpu->run->exit_reason = KVM_EXIT_DIRTY_RING_FULL;
/*
* If this is requested, it means that we've
* marked the dirty bit in the dirty ring BUT
* we've not written the date. Do it now.
*/
r = kvm_emulate_instruction(vcpu, 0);
r = r >= 0 ? 0 : r;
goto out;
}I did a kvm_emulate_instruction() when dirty ring reaches softlimit and want to exit to userspace, however I'm not really sure whether there could have any side effect. I'd appreciate any comment of above, or anything else.
Tests
===========
I wanted to continue work on the QEMU part, but after I noticed that the interface might still prone to change, I posted this series first. However to make sure it's at least working, I've provided unit tests together with the series. The unit tests should be able to test the series in at least three major paths:
(1) ./dirty_log_test -M dirty-ring
This tests async ring operations: this should be the major work mode for the dirty ring interface, say, when the kernel is queuing more data, the userspace is collecting too. Ring can hardly reaches full when working like this, because in most cases the collection could be fast.
(2) ./dirty_log_test -M dirty-ring -c 1024
This set the ring size to be very small so that ring soft-full always triggers (soft-full is a soft limit of the ring state, when the dirty ring reaches the soft limit it'll do a userspace exit and let the userspace to collect the data).
(3) ./dirty_log_test -M dirty-ring-wait-queue
This sololy test the extreme case where ring is full. When the ring is completely full, the thread (no matter vcpu or not) will be put onto a per-vm waitqueue, and KVM_RESET_DIRTY_RINGS will wake the threads up (assuming until which the ring will not be full any more).
Thanks,
Cao, Lei (2):
KVM: Add kvm/vcpu argument to mark_dirty_page_in_slot
KVM: X86: Implement ring-based dirty memory tracking
Paolo Bonzini (1):
KVM: Move running VCPU from ARM to common code
Peter Xu (12):
KVM: Add build-time error check on kvm_run size
KVM: Implement ring-based dirty memory tracking
KVM: Make dirty ring exclusive to dirty bitmap log
KVM: Introduce dirty ring wait queue
KVM: selftests: Always clear dirty bitmap after iteration
KVM: selftests: Sync uapi/linux/kvm.h to tools/
KVM: selftests: Use a single binary for dirty/clear log test
KVM: selftests: Introduce after_vcpu_run hook for dirty log test
KVM: selftests: Add dirty ring buffer test
KVM: selftests: Let dirty_log_test async for dirty ring test
KVM: selftests: Add "-c" parameter to dirty log test
KVM: selftests: Test dirty ring waitqueue
Documentation/virt/kvm/api.txt | 116 +++++ arch/arm/include/asm/kvm_host.h | 2 - arch/arm64/include/asm/kvm_host.h | 2 - arch/x86/include/asm/kvm_host.h | 5 + arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/Makefile | 3 +- arch/x86/kvm/mmu/mmu.c | 6 + arch/x86/kvm/vmx/vmx.c | 7 + arch/x86/kvm/x86.c | 12 + include/linux/kvm_dirty_ring.h | 67 +++ include/linux/kvm_host.h | 37 ++ include/linux/kvm_types.h | 1 + include/uapi/linux/kvm.h | 36 ++ tools/include/uapi/linux/kvm.h | 47 ++ tools/testing/selftests/kvm/Makefile | 2 - .../selftests/kvm/clear_dirty_log_test.c | 2 - tools/testing/selftests/kvm/dirty_log_test.c | 452 ++++++++++++++++-- .../testing/selftests/kvm/include/kvm_util.h | 6 + tools/testing/selftests/kvm/lib/kvm_util.c | 103 ++++ .../selftests/kvm/lib/kvm_util_internal.h | 5 + virt/kvm/arm/arm.c | 29 -- virt/kvm/arm/perf.c | 6 +- virt/kvm/arm/vgic/vgic-mmio.c | 15 +- virt/kvm/dirty_ring.c | 156 ++++++ virt/kvm/kvm_main.c | 315 +++++++++++- 25 files changed, 1329 insertions(+), 104 deletions(-) create mode 100644 include/linux/kvm_dirty_ring.h delete mode 100644 tools/testing/selftests/kvm/clear_dirty_log_test.c create mode 100644 virt/kvm/dirty_ring.c
Signature · 1 line
2.21.0
RFC PATCH 1/15 KVM: Move running VCPU from ARM to common code
From: Paolo Bonzini <[email protected]>
For ring-based dirty log tracking, it will be more efficient to account writes during schedule-out or schedule-in to the currently running VCPU. We would like to do it even if the write doesn't use the current VCPU's address space, as is the case for cached writes (see commit 4e335d9e7ddb, "Revert "KVM: Support vCPU-based gfn->hva cache"", 2017-05-02).
Therefore, add a mechanism to track the currently-loaded kvm_vcpu struct. There is already something similar in KVM/ARM; one important difference is that kvm_arch_vcpu_{load,put} have two callers in virt/kvm/kvm_main.c:
we have to update both the architecture-independent vcpu_{load,put} and the preempt notifiers.
Another change made in the process is to allow using kvm_get_running_vcpu() in preemptible code. This is allowed because preempt notifiers ensure that the value does not change even after the VCPU thread is migrated.
--- arch/arm/include/asm/kvm_host.h | 2 -- arch/arm64/include/asm/kvm_host.h | 2 -- include/linux/kvm_host.h | 3 +++ virt/kvm/arm/arm.c | 29 ----------------------------- virt/kvm/arm/perf.c | 6 +++--- virt/kvm/arm/vgic/vgic-mmio.c | 15 +++------------ virt/kvm/kvm_main.c | 25 ++++++++++++++++++++++++- 7 files changed, 33 insertions(+), 49 deletions(-)
arch/arm/include/asm/kvm_host.h +0 -2
@@ -284,8 +284,6 @@ int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *indices);int kvm_age_hva (struct kvm *kvm,unsigned long start,unsigned long end);int kvm_test_age_hva (struct kvm *kvm,unsigned long hva); -struct kvm_vcpu *kvm_arm_get_running_vcpu (void ); -struct kvm_vcpu __percpu **kvm_get_running_vcpus (void );void kvm_arm_halt_guest (struct kvm *kvm);void kvm_arm_resume_guest (struct kvm *kvm);
arch/arm64/include/asm/kvm_host.h +0 -2
@@ -446,8 +446,6 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);int kvm_age_hva (struct kvm *kvm,unsigned long start,unsigned long end);int kvm_test_age_hva (struct kvm *kvm,unsigned long hva); -struct kvm_vcpu *kvm_arm_get_running_vcpu (void ); -struct kvm_vcpu *__percpu *kvm_get_running_vcpus (void );void kvm_arm_halt_guest (struct kvm *kvm);void kvm_arm_resume_guest (struct kvm *kvm);
include/linux/kvm_host.h +3 -0
@@ -1342,6 +1342,9 @@ static inline void kvm_vcpu_set_dy_eligible(struct kvm_vcpu *vcpu, bool val) }#endif /* CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT */ +struct kvm_vcpu *kvm_get_running_vcpu (void ); +struct kvm_vcpu __percpu **kvm_get_running_vcpus (void ); +#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS bool kvm_arch_has_irq_bypass (void ); int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *,
virt/kvm/arm/arm.c +0 -29
@@ -51,9 +51,6 @@ __asm__(".arch_extension virt");DEFINE_PER_CPU (kvm_host_data_t, kvm_host_data); static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); -/* Per-CPU variable containing the currently running vcpu. */ -static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_arm_running_vcpu); -/* The VMID used in the VTTBR */ static atomic64_t kvm_vmid_gen =ATOMIC64_INIT (1 );static u32 kvm_next_vmid;
@@ -62,31 +59,8 @@ static DEFINE_SPINLOCK(kvm_vmid_lock);static bool vgic_present; static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled); - -static void kvm_arm_set_running_vcpu (struct kvm_vcpu *vcpu) -{ -__this_cpu_write (kvm_arm_running_vcpu, vcpu); -} -DEFINE_STATIC_KEY_FALSE (userspace_irqchip_in_use); -/** -* kvm_arm_get_running_vcpu - get the vcpu running on the current CPU. -* Must be called from non-preemptible context -*/ -struct kvm_vcpu *kvm_arm_get_running_vcpu (void ) -{ -return __this_cpu_read (kvm_arm_running_vcpu); -} - -/** -* kvm_arm_get_running_vcpus - get the per-CPU array of currently running vcpus. -*/ -struct kvm_vcpu *__percpu *kvm_get_running_vcpus (void ) -{ -return &kvm_arm_running_vcpu; -} -int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu) {return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ;
@@ -406,7 +380,6 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu->cpu = cpu; vcpu->arch.host_cpu_context = &cpu_data->host_ctxt; -kvm_arm_set_running_vcpu (vcpu);kvm_vgic_load (vcpu);kvm_timer_vcpu_load (vcpu);kvm_vcpu_load_sysregs (vcpu);
@@ -432,8 +405,6 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)kvm_vcpu_pmu_restore_host (vcpu); vcpu->cpu =-1 ; - -kvm_arm_set_running_vcpu (NULL ); }static void vcpu_power_off (struct kvm_vcpu *vcpu)
virt/kvm/arm/perf.c +3 -3
@@ -13,14 +13,14 @@static int kvm_is_in_guest (void ) { -return kvm_arm_get_running_vcpu () !=NULL ; +return kvm_get_running_vcpu () !=NULL ; }static int kvm_is_user_mode (void ) {struct kvm_vcpu *vcpu; - vcpu =kvm_arm_get_running_vcpu (); + vcpu =kvm_get_running_vcpu ();if (vcpu)return !vcpu_mode_priv (vcpu);
@@ -32,7 +32,7 @@ static unsigned long kvm_get_guest_ip(void) {struct kvm_vcpu *vcpu; - vcpu =kvm_arm_get_running_vcpu (); + vcpu =kvm_get_running_vcpu ();if (vcpu)return *vcpu_pc (vcpu);
virt/kvm/arm/vgic/vgic-mmio.c +3 -12
@@ -190,15 +190,6 @@ unsigned long vgic_mmio_read_pending(struct kvm_vcpu *vcpu,* value later will give us the same value as we update the per-CPU variable * in the preempt notifier handlers. */ -static struct kvm_vcpu *vgic_get_mmio_requester_vcpu (void ) -{ -struct kvm_vcpu *vcpu; - -preempt_disable (); - vcpu =kvm_arm_get_running_vcpu (); -preempt_enable (); -return vcpu; -}/* Must be called with irq->irq_lock held */ static void vgic_hw_irq_spending(struct kvm_vcpu *vcpu, struct vgic_irq *irq,
@@ -221,7 +212,7 @@ void vgic_mmio_write_spending(struct kvm_vcpu *vcpu, gpa_t addr, unsigned int len, unsigned long val) { - bool is_uaccess = !vgic_get_mmio_requester_vcpu(); + bool is_uaccess = !kvm_get_running_vcpu();u32 intid =VGIC_ADDR_TO_INTID (addr,1 );int i;unsigned long flags;
@@ -274,7 +265,7 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu, gpa_t addr, unsigned int len, unsigned long val) { - bool is_uaccess = !vgic_get_mmio_requester_vcpu(); + bool is_uaccess = !kvm_get_running_vcpu();u32 intid =VGIC_ADDR_TO_INTID (addr,1 );int i;unsigned long flags;
@@ -335,7 +326,7 @@ static void vgic_mmio_change_active(struct kvm_vcpu *vcpu, struct vgic_irq *irq, bool active) {unsigned long flags; -struct kvm_vcpu *requester_vcpu =vgic_get_mmio_requester_vcpu (); +struct kvm_vcpu *requester_vcpu =kvm_get_running_vcpu ();raw_spin_lock_irqsave (&irq->irq_lock, flags);
virt/kvm/kvm_main.c +24 -1
@@ -108,6 +108,7 @@ struct kmem_cache *kvm_vcpu_cache;EXPORT_SYMBOL_GPL (kvm_vcpu_cache);static __read_mostly struct preempt_ops kvm_preempt_ops; +static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_running_vcpu);struct dentry *kvm_debugfs_dir;EXPORT_SYMBOL_GPL (kvm_debugfs_dir);
@@ -197,6 +198,8 @@ bool kvm_is_reserved_pfn(kvm_pfn_t pfn)void vcpu_load (struct kvm_vcpu *vcpu) {int cpu =get_cpu (); + +__this_cpu_write (kvm_running_vcpu, vcpu);preempt_notifier_register (&vcpu->preempt_notifier);kvm_arch_vcpu_load (vcpu, cpu);put_cpu ();
@@ -208,6 +211,7 @@ void vcpu_put(struct kvm_vcpu *vcpu)preempt_disable ();kvm_arch_vcpu_put (vcpu);preempt_notifier_unregister (&vcpu->preempt_notifier); +__this_cpu_write (kvm_running_vcpu,NULL );preempt_enable (); }EXPORT_SYMBOL_GPL (vcpu_put);
@@ -4304,8 +4308,8 @@ static void kvm_sched_in(struct preempt_notifier *pn, int cpu)WRITE_ONCE (vcpu->preempted, false);WRITE_ONCE (vcpu->ready, false); +__this_cpu_write (kvm_running_vcpu, vcpu);kvm_arch_sched_in (vcpu, cpu); -kvm_arch_vcpu_load (vcpu, cpu); }
@@ -4319,6 +4323,25 @@ static void kvm_sched_out(struct preempt_notifier *pn,WRITE_ONCE (vcpu->ready, true); }kvm_arch_vcpu_put (vcpu); +__this_cpu_write (kvm_running_vcpu,NULL ); +} + +/** +* kvm_get_running_vcpu - get the vcpu running on the current CPU. +* Thanks to preempt notifiers, this can also be called from +* preemptible context. +*/ +struct kvm_vcpu *kvm_get_running_vcpu (void ) +{ +return __this_cpu_read (kvm_running_vcpu); +} + +/** +* kvm_get_running_vcpus - get the per-CPU array of currently running vcpus. +*/ +struct kvm_vcpu *__percpu *kvm_get_running_vcpus (void ) +{ +return &kvm_running_vcpu; }static void check_processor_compat (void *rtn)
Signature · 1 line
2.21.0