KVM: Mark a vCPU as preempted/ready iff it's scheduled out while running
PATCH v2 1/1 KVM: Mark a vCPU as preempted/ready iff it's scheduled out while running
Mark a vCPU as preempted/ready if-and-only-if it's scheduled out while running. i.e. Do not mark a vCPU preempted/ready if it's scheduled out during a non-KVM_RUN ioctl() or when userspace is doing KVM_RUN with immediate_exit.
Commit 54aa83c90198 ("KVM: x86: do not set st->preempted when going back to user space") stopped marking a vCPU as preempted when returning to userspace, but if userspace then invokes a KVM vCPU ioctl() that gets preempted, the vCPU will be marked preempted/ready. This is arguably incorrect behavior since the vCPU was not actually preempted while the guest was running, it was preempted while doing something on behalf of userspace.
This commit also avoids KVM dirtying guest memory after userspace has paused vCPUs, e.g. for Live Migration, which allows userspace to collect the final dirty bitmap before or in parallel with saving vCPU state without having to worry about saving vCPU state triggering writes to guest memory.
---
v2:
- Drop Google-specific "PRODKERNEL: " shortlog prefix
v1: https://lore.kernel.org/kvm/[email protected]/
include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +1 -0
@@ -378,6 +378,7 @@ struct kvm_vcpu { bool dy_eligible; } spin_loop;#endif +bool wants_to_run;bool preempted;bool ready;struct kvm_vcpu_arch arch;
virt/kvm/kvm_main.c +4 -1
@@ -4438,7 +4438,10 @@ static long kvm_vcpu_ioctl(struct file *filp,synchronize_rcu ();put_pid (oldpid); } + vcpu->wants_to_run = !vcpu->run->immediate_exit; r =kvm_arch_vcpu_ioctl_run (vcpu); + vcpu->wants_to_run = false; +trace_kvm_userspace_exit (vcpu->run->exit_reason, r);break ; }
@@ -6312,7 +6315,7 @@ static void kvm_sched_out(struct preempt_notifier *pn, {struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); -if (current->on_rq) { +if (current->on_rq && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
base-commit: 687d8f4c3dea0758afd748968d91288220bbe7e3
Signature · 1 line
2.44.0.278.ge034bb2e1d-goog
Quoted 14 lines · David Matlack
Mark a vCPU as preempted/ready if-and-only-if it's scheduled out while running. i.e. Do not mark a vCPU preempted/ready if it's scheduled out during a non-KVM_RUN
ioctl()or when userspace is doingKVM_RUNwithimmediate_exit.Commit
54aa83c90198("KVM: x86: do not setst->preemptedwhen going back to user space") stopped marking a vCPU as preempted when returning to userspace, but if userspace then invokes a KVM vCPUioctl()that gets preempted, the vCPU will be marked preempted/ready. This is arguably incorrect behavior since the vCPU was not actually preempted while the guest was running, it was preempted while doing something on behalf of userspace.
Quoted 8 lines
This commit also avoids KVM dirtying guest memory after userspace has paused vCPUs, e.g. for Live Migration, which allows userspace to collect the final dirty bitmap before or in parallel with saving vCPU state without having to worry about saving vCPU state triggering writes to guest memory.
Suggested-by Sean Christopherson <[email protected]>Signed-off-by David Matlack <[email protected]>
Gentle ping.
Sean Christopherson
maintainerQuoted 69 lines · David Matlack
Mark a vCPU as preempted/ready if-and-only-if it's scheduled out while running. i.e. Do not mark a vCPU preempted/ready if it's scheduled out during a non-KVM_RUN
ioctl()or when userspace is doingKVM_RUNwithimmediate_exit.Commit
54aa83c90198("KVM: x86: do not setst->preemptedwhen going back to user space") stopped marking a vCPU as preempted when returning to userspace, but if userspace then invokes a KVM vCPUioctl()that gets preempted, the vCPU will be marked preempted/ready. This is arguably incorrect behavior since the vCPU was not actually preempted while the guest was running, it was preempted while doing something on behalf of userspace.This commit also avoids KVM dirtying guest memory after userspace has paused vCPUs, e.g. for Live Migration, which allows userspace to collect the final dirty bitmap before or in parallel with saving vCPU state without having to worry about saving vCPU state triggering writes to guest memory.
Suggested-by Sean Christopherson <[email protected]>Signed-off-by David Matlack <[email protected]>---
v2:
- Drop Google-specific "PRODKERNEL: " shortlog prefixv1: https://lore.kernel.org/kvm/[email protected]/
include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 5 ++++- 2 files changed, 5 insertions(+), 1 deletion(-)include/linux/kvm_host.h +1
-0@@ -378,6 +378,7 @@ struct kvm_vcpu { bool dy_eligible; } spin_loop;#endif +bool wants_to_run;bool preempted;bool ready;struct kvm_vcpu_arch arch;virt/kvm/kvm_main.c +3
-0@@ -4438,7 +4438,10 @@ static long kvm_vcpu_ioctl(struct file *filp,synchronize_rcu ();put_pid (oldpid); } + vcpu->wants_to_run = !vcpu->run->immediate_exit; r =kvm_arch_vcpu_ioctl_run (vcpu); + vcpu->wants_to_run = false; +trace_kvm_userspace_exit (vcpu->run->exit_reason, r);break ;virt/kvm/kvm_main.c +1
-1}@@ -6312,7 +6315,7 @@ static void kvm_sched_out(struct preempt_notifier *pn, {struct kvm_vcpu *vcpu =preempt_notifier_to_vcpu (pn); -if (current->on_rq) { +if (current->on_rq && vcpu->wants_to_run) {WRITE_ONCE (vcpu->preempted, true);WRITE_ONCE (vcpu->ready, true); }
Quoted 1 line
base-commit:
687d8f4c3dea0758afd748968d91288220bbe7e3
Long story short, I was playing around with wants_to_run for a few hairbrained ideas, and realized that there's a TOCTOU bug here. Userspace can toggle run->immediate_exit at will, e.g. can clear it after the kernel loads it to compute vcpu->wants_to_run.
That's not fatal for this use case, since userspace would only be shooting itself in the foot, but it leaves a very dangerous landmine, e.g. if something else in KVM keys off of vcpu->wants_to_run to detect that KVM is in its run loop, i.e. relies on wants_to_run being set if KVM is in its core run loop.
To address that, I think we should have all architectures check wants_to_run, not immediate_exit. And loading immediate_exit needs to be done with READ_ONCE().
E.g. for x86 (every other arch has similar code)
arch/x86/kvm/x86.c +2 -2
@@ -11396,7 +11396,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)kvm_vcpu_srcu_read_lock (vcpu);if (unlikely (vcpu->arch.mp_state ==KVM_MP_STATE_UNINITIALIZED )) { -if (kvm_run->immediate_exit) { +if (!vcpu->wants_to_run) { r = -EINTR ; goto out; }
@@ -11474,7 +11474,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)WARN_ON_ONCE (vcpu->mmio_needed); } -if (kvm_run->immediate_exit) { +if (!vcpu->wants_to_run) { r = -EINTR ; goto out; }
include/linux/kvm_host.h +1 -3
@@ -1497,9 +1497,7 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, struct kvm_guest_debug *dbg);int kvm_arch_vcpu_ioctl_run (struct kvm_vcpu *vcpu); -void kvm_arch_sched_in (struct kvm_vcpu *vcpu,int cpu); - -void kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +void kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu,bool sched_in);void kvm_arch_vcpu_put (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_precreate (struct kvm *kvm,unsigned int id);int kvm_arch_vcpu_create (struct kvm_vcpu *vcpu);
virt/kvm/kvm_main.c +1 -1
@@ -4410,7 +4410,7 @@ static long kvm_vcpu_ioctl(struct file *filp,synchronize_rcu ();put_pid (oldpid); } - vcpu->wants_to_run = !vcpu->run->immediate_exit; + vcpu->wants_to_run = !READ_ONCE (vcpu->run->immediate_exit); r =kvm_arch_vcpu_ioctl_run (vcpu); vcpu->wants_to_run = false;
---
Hmm, and we should probably go a step further and actively prevent using immediate_exit from the kernel, e.g. rename it to something scary like:
include/uapi/linux/kvm.h +4 -0
@@ -196,7 +196,11 @@ struct kvm_xen_exit {struct kvm_run {/* in */ __u8 request_interrupt_window; +#ifndef __KERNEL____u8 immediate_exit; +#else +__u8 hidden_do_not_touch; +#endif __u8 padding1[6 ];/* out */
Quoted 13 lines · Sean Christopherson
Review
Quoted 6 lines · David Matlack
(quoted diff) +1
-1- if (current->on_rq) { + if (current->on_rq && vcpu->wants_to_run) { WRITE_ONCE(vcpu->preempted, true); WRITE_ONCE(vcpu->ready, true); }Long story short, I was playing around with
wants_to_runfor a few hairbrained ideas, and realized that there's a TOCTOU bug here. Userspace can togglerun->immediate_exitat will, e.g. can clear it after the kernel loads it to computevcpu->wants_to_run.
Quoted 7 lines
That's not fatal for this use case, since userspace would only be shooting itself in the foot, but it leaves a very dangerous landmine, e.g. if something else in KVM keys off of
vcpu->wants_to_runto detect that KVM is in its run loop, i.e. relies onwants_to_runbeing set if KVM is in its core run loop.To address that, I think we should have all architectures check
wants_to_run, notimmediate_exit.
Rephrasing to make sure I understand you correctly: It's possible for KVM to see conflicting values of wants_to_run and immediate_exit, since userspace can change immediate_exit at any point. e.g. It's possible for KVM to see wants_to_run=true and immediate_exit=true, which wouldn't make sense. This wouldn't cause any bugs today, but could result in buggy behavior down the road so we might as well clean it up now.
Quoted 1 line
And loading
immediate_exitneeds to be done withREAD_ONCE().
+1, good point
Quoted 64 lines
E.g. for x86 (every other arch has similar code)
arch/x86/kvm/x86.c +2
-2@@ -11396,7 +11396,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)kvm_vcpu_srcu_read_lock (vcpu);if (unlikely (vcpu->arch.mp_state ==KVM_MP_STATE_UNINITIALIZED )) { -if (kvm_run->immediate_exit) { +if (!vcpu->wants_to_run) { r = -EINTR ; goto out; }@@ -11474,7 +11474,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)WARN_ON_ONCE (vcpu->mmio_needed); } -if (kvm_run->immediate_exit) { +if (!vcpu->wants_to_run) { r = -EINTR ; goto out; }include/linux/kvm_host.h +1
-3@@ -1497,9 +1497,7 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, struct kvm_guest_debug *dbg);int kvm_arch_vcpu_ioctl_run (struct kvm_vcpu *vcpu); -void kvm_arch_sched_in (struct kvm_vcpu *vcpu,int cpu); - -void kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu); +void kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu,bool sched_in);void kvm_arch_vcpu_put (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_precreate (struct kvm *kvm,unsigned int id);int kvm_arch_vcpu_create (struct kvm_vcpu *vcpu);virt/kvm/kvm_main.c +1
-2@@ -4410,7 +4410,7 @@ static long kvm_vcpu_ioctl(struct file *filp,synchronize_rcu ();put_pid (oldpid); } - vcpu->wants_to_run = !vcpu->run->immediate_exit; + vcpu->wants_to_run = !READ_ONCE (vcpu->run->immediate_exit); r =kvm_arch_vcpu_ioctl_run (vcpu); vcpu->wants_to_run = false; ---Hmm, and we should probably go a step further and actively prevent using
immediate_exitfrom the kernel, e.g. rename it to something scary like:include/uapi/linux/kvm.h +0
-0
include/uapi/linux/kvm.h struct kvm_xen_exit { · line 203
Quoted 8 lines
include/uapi/linux/kvm.h +4
-0@@ -196,7 +196,11 @@ struct kvm_xen_exit {struct kvm_run {/* in */ __u8 request_interrupt_window; +#ifndef __KERNEL____u8 immediate_exit; +#else +__u8 hidden_do_not_touch; +#endif
This would result in:
vcpu->wants_to_run = !READ_ONCE(vcpu->run->hidden_do_not_touch);
:)
Of course we could pick a better name... but isn't every field in kvm_run open to TOCTOU issues? (Is immediate_exit really special enough to need this protection?)
Sean Christopherson
maintainerQuoted 22 lines · David Matlack
Quoted 13 lines · Sean Christopherson
Review
Quoted 6 lines · David Matlack
(quoted diff) +1
-1- if (current->on_rq) { + if (current->on_rq && vcpu->wants_to_run) { WRITE_ONCE(vcpu->preempted, true); WRITE_ONCE(vcpu->ready, true); }Long story short, I was playing around with
wants_to_runfor a few hairbrained ideas, and realized that there's a TOCTOU bug here. Userspace can togglerun->immediate_exitat will, e.g. can clear it after the kernel loads it to computevcpu->wants_to_run.Quoted 7 lines
That's not fatal for this use case, since userspace would only be shooting itself in the foot, but it leaves a very dangerous landmine, e.g. if something else in KVM keys off of
vcpu->wants_to_runto detect that KVM is in its run loop, i.e. relies onwants_to_runbeing set if KVM is in its core run loop.To address that, I think we should have all architectures check
wants_to_run, notimmediate_exit.
Quoted 7 lines
Rephrasing to make sure I understand you correctly: It's possible for KVM to see conflicting values of
wants_to_runandimmediate_exit, since userspace can changeimmediate_exitat any point. e.g. It's possible for KVM to see wants_to_run=true and immediate_exit=true, which wouldn't make sense. This wouldn't cause any bugs today, but could result in buggy behavior down the road so we might as well clean it up now.
Yep.
Quoted 19 lines
Quoted 8 lines
Hmm, and we should probably go a step further and actively prevent using
immediate_exitfrom the kernel, e.g. rename it to something scary like:include/uapi/linux/kvm.h +0
-0include/uapi/linux/kvm.h struct kvm_xen_exit { · line 203
Quoted 8 lines
include/uapi/linux/kvm.h +4
-0@@ -196,7 +196,11 @@ struct kvm_xen_exit {struct kvm_run {/* in */ __u8 request_interrupt_window; +#ifndef __KERNEL____u8 immediate_exit; +#else +__u8 hidden_do_not_touch; +#endif This would result in:
Quoted 5 lines
vcpu->wants_to_run = !READ_ONCE(vcpu->run->hidden_do_not_touch);:)
Of course we could pick a better name...
Heh, yeah, for demonstration purposes only.
Quoted 1 line
but isn't every field in
kvm_runopen to TOCTOU issues?
Yep, and we've had bugs, e.g. see commit 0d033770d43a ("KVM: x86: Fix KVM_CAP_SYNC_REGS's sync_regs() TOCTOU issues").
Quoted 1 line
(Is
immediate_exitreally special enough to need this protection?)
I think so.
It's not that immediate_exit is more prone to TOCTOU bugs than other fields in kvm_run (though I do think immediate_exit does have higher potential for future bugs), or even that the severity of bugs that could occur with immediate_exit is high (which I again think is the case), it's that it's actually feasible to effectively prevent TOCTOU bugs with minimal cost (including ongoing maintenance cost). At the cost of a small-ish one-time change, we can protect *all* KVM code against improer usage of immediate_exit.
Doing the same for other kvm_run fields is less feasiable, as the relevant logic is much more architecture specific. E.g. x86 now does a full copy of sregs and events in kvm_sync_regs, but not regs because the input for regs is never checked. And blindly creating an in-kernel copy of all state would be extremely wasteful for s390, which IIUC uses kvm_run.s.regs as _the_ buffer for guest register state.